From nobody Mon Sep 28 05:45:29 2026 Received: from CY3PR05CU001.outbound.protection.outlook.com (mail-westcentralusazon11013062.outbound.protection.outlook.com [40.93.201.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 642DD3A785D for ; Wed, 26 Aug 2026 06:29:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.201.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725794; cv=fail; b=kzBihiuH90mEzG0vRwT8//ifL64mfugSSJzJvltT8GeN2vGEQOkzHH+u/v8pOodbuEmjTsliwNFgBIy2tVRju5QsIhdhU7YaRYpKx29l8BzJi6d1bWk2rK5XPBQibpxy0rJKLaLoL2VeyGbqI4D0DCG41l0nGWrAzxIn9f2Ry2A= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725794; c=relaxed/simple; bh=p5DfeYT6N220aJOSJi28pSvgzqDlWsSpMUa8/CCbEh4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HYLM14FwDFWJTwI6qqtqqnsXud87bWO79P2DuFMplEkP5GwAaSugYm069og0kvNBOvKZDwKZNg9JOnQSKFAm418ev/lplkF3WIZjpposeU60i3heOHuM0QXtNgN83720bWumBZ1yUGSUKkMNVFj6LL0Dsr/gq+yzNYi1EpXP9uU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=1ZwwP4y2; arc=fail smtp.client-ip=40.93.201.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="1ZwwP4y2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Vkku2VJusPorRoJWCkI2unBh6ev4qCtoHdGXlRVRWic12GGBJ9qEnczlo3vYxOguVN++8TX91T1F1E49iwSXQPSvl6kG7yKjzgsckfEAMRnCQr5VttMQCelQQyd7i+kR9GKDqX/WQKkteQmGyv+Hf8BnsAV8HAVDUxudMFVrlMzC+nuxcHm2nRPEmHxz1KD2R46gwIqPBUZ81LgQ6ccAQrPbfcGBRCHzJuQAv53/ox5ZNkLjrmyUtoB07YrWPyqvPeMWR/LF/MqI0ly73qEOtPnn7eJvQet93oGUMe+yvEOY/un4+rcFRSV+OqInEMNaKLVz6gDeTHB67XNQ9Qwc8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=URZsp2EEDxLMjKUYkfLoGC9J3ASF/MO2C1qlMunuEew=; b=FavneDpsHFORgI3juaH8YPYQla9WPlH+RtK8rR3u4iGnnUE/HROlolIkPNyc2BrxFPy4YjdmYPiTL6DYD6qXaXG5XlN4Wo9o83bUBx/7HnB70rhOFK17uApgLgA5vCr48RhVXT5b/5A4onjW48vGUOJcxHgCFrFb/RtVOYOoGbnZUlDPvT9JzOqBeVwUgdQbxC6HO9857cmfZ+5czKXIJjQImJorTlMhxJMjGmoo4wu0bu+uGEqrIDA8H2WhKgjO4Apol/X27Jc0inlGVu/m/uxN/QiwZbsQYfJ3G11ES/VekFYKxXZ9qLNGG8t4l7nrz/IXJeY5WlIRTxTM+TmXsA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=URZsp2EEDxLMjKUYkfLoGC9J3ASF/MO2C1qlMunuEew=; b=1ZwwP4y26097QrGCUWt7ib3wgYXcMqB2ehmANCq6zh1gewdvdvnKW2TjNe7PjlPL5/pOlh8aDVpKQzM7CnCK1qJxWUkXh/WgA2DJaFHx+pRrz9Rh3S0EIEQ3UM/QwMjGYSxeWq3xF4eDGhGq2S9dDkFCoTDvioIQD/Jswo5DIhY= Received: from CH5PR05CA0020.namprd05.prod.outlook.com (2603:10b6:610:1f0::13) by IA0PR12MB8326.namprd12.prod.outlook.com (2603:10b6:208:40d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:29:45 +0000 Received: from CH2PEPF00000144.namprd02.prod.outlook.com (2603:10b6:610:1f0:cafe::58) by CH5PR05CA0020.outlook.office365.com (2603:10b6:610:1f0::13) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.5 via Frontend Transport; Wed, 26 Aug 2026 06:29:45 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000144.mail.protection.outlook.com (10.167.244.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:29:44 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:29:39 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 01/16] sched/core: Break activation of blocked task into a separate helper Date: Wed, 26 Aug 2026 06:28:45 +0000 Message-ID: <20260826062901.2137-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000144:EE_|IA0PR12MB8326:EE_ X-MS-Office365-Filtering-Correlation-Id: 3f534cd1-f730-4c54-092c-08df033b6b72 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|23010399003|7416014|376014|36860700016|921020|11063799006|10067099003|56012099006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: UxGY0i6DIacVxNj2ZxLpX1AwKd8LtYCfRvUgeCVxYU8ydzmYL1J0Y3JGJoMFhzMN9D6mmATBk8/ooygUEC39xA3beqxCelc5SC3mwG5ULC+QIfrVjY8PAFIJpa3aN6SnkDXcZ/02nBSiL1zP96UUjOijdROjIgz5IwEUW5S+3EF4vzJVF5d+vPVDks+shRlKYf4uTCXaea+VNmneyt14LntC6lBo8mK0AwCLzZlTalFldk856Tp0C2iYZ/VA1TpqPnhtdjGlPf7JRqKrnu7C8El7+xIo22983Uex3SdndrG+o+f9DfsVX93cCIrSar8hbYthn4Ayl3/3zIEKMGf5kOP3OniirfsCEoN5ZEWxBhgGCTX/zTVKLKbJ3gmCfAwpZw3efMrjapHiS1Dxd8Db89CziuSAi9RpaUkmlBa+X7fazi86gxdrFtEUzM4aCh/1Kh7Mu8mihELzrW6rFYaYPjvELSHA2MK3SzWTzWvmDhlb8eTFE9ZIHY/AWjFvSbt/OiCAjrG3k+aTxWKg8Gli92zPQU04qAdVN0doPS3nzhB0s5geHmeKaepDpmARQ+V9ntnUbFHtgUfxXaSOKM7GUeGr8kdaYvc1uEp7Oe1hzTHkhY53dFHD5IQUXpYStmI3e8y4weY75ZW3qXNMvtRPdX0TCLG6Mlij/Cd8cSc+6yEWPmVLztkOK6tYdICZHJx0ol56iqMcRGWz0XlGhKnQMnlfcdpqFsvOhviPHLM8DjClirOOp18hbGPq+WEj8sMD X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(23010399003)(7416014)(376014)(36860700016)(921020)(11063799006)(10067099003)(56012099006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: OknO3Hwnp4k0DwlEtTg5rTZXBmS2+PgJv6KL/r+vFxgm8wr2fzu9IwbRyiWhJKQA+/Y73yu/nawH2e78efloaXm9sYfd9evXpgkCUVgKD1WGW3bM68InmfTnZPqNTl4W9J76eYbaiG6rTkaRsjI++KXZDGjxWNFqHIvZ/H/z1Z+/PqbGF/uQaulIdq9xDGCKeGxFVP59xaxrGVKAr9c5wD4PPlTN1MAIhsPHYjuC/c+iSqtQXec5kDrH1RyiAtJF9xQgVvGi77XxY8b6djD336m9lHZZM8xa8rMuqodDG6X7iZnIUCsLIEkrFO+nr9B2JsmVRXc//4Lgldo0k72Qy5vndIVVZ7cjwbKJJslL6PVkqUC16R1I3HKPQoJ/gopfGw4Fsb/t7cIoPt91jVPvfwqxQSesHpyE5ckjymrJ8B3LgiEpKl0hig/LcUhNzsxs X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:29:44.9074 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3f534cd1-f730-4c54-092c-08df033b6b72 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000144.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8326 Content-Type: text/plain; charset="utf-8" Add activate_blocked_task() helper to activate a blocked task (Blocked task is !p->on_rq || p->se.sched_delayed). This is preparatory patch to clean up the task_on_rq_migrating() usage in on the enqueue path in the sched_class callbacks to make way for proxy activation on the enqueue path. No functional changes intended. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1bf60d78c920..7d1ffcc33fed 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2216,7 +2216,7 @@ inline bool dequeue_task(struct rq *rq, struct task_s= truct *p, int flags) return p->sched_class->dequeue_task(rq, p, flags); } =20 -void activate_task(struct rq *rq, struct task_struct *p, int flags) +static void __activate_task(struct rq *rq, struct task_struct *p, int flag= s) { if (task_on_rq_migrating(p)) flags |=3D ENQUEUE_MIGRATED; @@ -2227,6 +2227,16 @@ void activate_task(struct rq *rq, struct task_struct= *p, int flags) ASSERT_EXCLUSIVE_WRITER(p->on_rq); } =20 +void activate_task(struct rq *rq, struct task_struct *p, int en_flags) +{ + __activate_task(rq, p, en_flags); +} + +static void activate_blocked_task(struct rq *rq, struct task_struct *p, in= t en_flags) +{ + __activate_task(rq, p, en_flags); +} + void deactivate_task(struct rq *rq, struct task_struct *p, int flags) { WARN_ON_ONCE(flags & DEQUEUE_SLEEP); @@ -3822,7 +3832,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p= , int wake_flags, atomic_dec(&task_rq(p)->nr_iowait); } =20 - activate_task(rq, p, en_flags); + activate_blocked_task(rq, p, en_flags); wakeup_preempt(rq, p, wake_flags); =20 ttwu_do_wakeup(p); @@ -4961,7 +4971,7 @@ void wake_up_new_task(struct task_struct *p) update_rq_clock(rq); post_init_entity_util_avg(p); =20 - activate_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_INITIAL); + activate_blocked_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_INITIAL); trace_sched_wakeup_new(p); wakeup_preempt(rq, p, wake_flags); if (p->sched_class->task_woken) { --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012000.outbound.protection.outlook.com [52.101.48.0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54329347505 for ; Wed, 26 Aug 2026 06:30:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.0 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725813; cv=fail; b=rVyrxf9gX8wMNFFavnO89VhSj9GFGSj1aCuc8OxMmkFZ0SHhn3wGdUOkaHkiXtG+rDWIi/V40ajjdsmbGsEEwmjUnOBetm0sYnRDwV/ycWH0YN2ZVGmPMURohMVgBuLsGkCPHG5mQO4RpwbXD+oD1rAezEWxllWy9OBuHPEqss4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725813; c=relaxed/simple; bh=s7TSYtAJhb2VSQO5Eur/aLMyZcXcRIaTUpCaHciIR2o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=IKRssRabA30XpYdajzoNkmJgfn6hbx/4dNBTTFrCk2Q+eFy5tXiilAg3fcSYQag5kjQDn6sRCzFUrb8C5+aISDQYWFEDTqzTPVFYYsKhUVgcIbrUXvx/CuzjpEnhhtD00UKk/qhmg/27zd9ddpu4a/Uh4jjstntsKwcfLfJxTXk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=bhniMp/V; arc=fail smtp.client-ip=52.101.48.0 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="bhniMp/V" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Eu8ozRcMc49M0kLjR+mF61SZZY6Pv8O4DIjmqcURexEQiremXfm7RfdxnJ4BH+jxp3JFlR8oDP+NQFSHPpfXy5hkrdmgNaSnBrQZQYHWnNgokB7PqraCo7VKIfqZtzZ0QoUipxnAByaLnhzzBFJeH95ZvR43h+wn81F1aOLu4nE1Ch2vyl8XpL7ND4KSFSsOdpGGl/+qqi8qaSVaenXqOm6NtO/cuVKoEO/k6aFK8rjISPJNq5zjxohevUs5A9KidxGMefkI/Xw2+3M7Cr8l/CP3yMM09soSUZjPr1+wCGSJeJCHru3sTgacffC2kwrjFCP9jLK5847KpPF0hYQZxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JOP+uTLhdvSd6jbQOP8MepEIh5XOZv9JlpX/YU3Czxc=; b=tW3C2Y1Eej1SKq4CPB5iIuTX1hjBvKkHjMjpIUblSs537lK5oB9gpGooXV3Ic2ByXQqNG+dczKabidvTWWWwLlDsSnsV8PeJRoPUqHHKT4FS8V3HdnrodhI0rB+upA5oFTAgtrwjmbSTkmsb4+c8vq2kxDxSlbDOqI5pCJL9tZdLmYwvNQtG+YF0y/cSZ7KcA1jsM4Z5KhpBmJyNxahl8kQYUjO7zVLv3NkO3tyrXYHLbLsIJ0ZFLZWwLJYv+r39YjHAO7puFnP6b/D2drvPFNFapbyFCf6MiWbLV1WXLA+S0Obesrr/t9aUeSXyvFZCI8r3iIWqdj9G9/7c3y80Kw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JOP+uTLhdvSd6jbQOP8MepEIh5XOZv9JlpX/YU3Czxc=; b=bhniMp/VsX8ffbLn3W5mxWZBX/DaW0ChY/m+ovMuDSiYQStcWIsD4SqhNcAoPsWtZbfWMSAEBsaO9t8salukwvQ1fZQSP34R2yK7vxCQYgEkcn8H7U3C1YINk3gzI72T4wleS49eY1vWaDcqDHXPFVroE6eyqBf6F7sNhURAjDs= Received: from CH5PR05CA0011.namprd05.prod.outlook.com (2603:10b6:610:1f0::23) by SA3PR12MB7976.namprd12.prod.outlook.com (2603:10b6:806:312::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:30:04 +0000 Received: from CH2PEPF00000144.namprd02.prod.outlook.com (2603:10b6:610:1f0:cafe::38) by CH5PR05CA0011.outlook.office365.com (2603:10b6:610:1f0::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.5 via Frontend Transport; Wed, 26 Aug 2026 06:30:04 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000144.mail.protection.outlook.com (10.167.244.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:30:04 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:29:57 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 02/16] sched/core: Use enqueue/dequeue flags instead of task_on_rq_migrating() Date: Wed, 26 Aug 2026 06:28:46 +0000 Message-ID: <20260826062901.2137-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000144:EE_|SA3PR12MB7976:EE_ X-MS-Office365-Filtering-Correlation-Id: db106da4-a699-4d88-cf47-08df033b7710 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|36860700016|82310400026|7416014|23010399003|22082099003|18002099003|11063799006|56012099006|10067099003|921020; X-Microsoft-Antispam-Message-Info: RT2oAABjKgBgJmoqSoybWlILmJQtoSSi0lquScLL0xSQuUcL6hIbf7GsJmnJ+IlCiiP1ir617tlohYXnWs4uP9wWZBzxOUp2GeE5ixT1TTD47ny7dUPiHjBks80ZGUkBxfFjnq4bUti9qbw8k7Ehzu4JaD+IPCwYwv7CZUQAw9jUF6mIKqUGcUoxHZ4ciAmGizEXNC1qAVp5GLsRH+7i1+/M+6K0CEBtZPBKauy62DVewE3OJuzAAMbCftM1cxead/U4m8BF4gAlw3KoVudEy9p336EfXJ3pxvmcUuDHkgcnP2VQa06I5WosenIvAP8immXcsIMuzk5DXNZh91r3UZHL+bTh+aM+g1vcURKNsPZ54NsrtITU+1mAIPlZWU1cq8SwhqWMTD/9bch06ZMfwPakWu6dh+OAfeRFcPeaTgYt9URZa1KNfC7MQsXhgqj6id1SsirHd11sLs7ueKfpSTOh/R7/+6O4+sByacUCxHQgim8j4uIf4PQ8QHguwL5yd3rg3f1EJ57Og6hx3uh12/KERor0Daeae6clR097kyYdkbHUiYUM63dvDi2A0JEmxCdTNWyEl6tICr1G85KljtnDTv7QVox78o9R3xEfj2UlGraAjATbc6aCxtiYzUMpoeRJX8vQAyZ2CrAu5knjnCqIll6BdTHUQRiJy0AGFBLcSqguiTA5f3JEXOJ4xxX/lzhnddSGx0806pGwX0oYn/KWIQQRfOZG4drGond6om8= X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(36860700016)(82310400026)(7416014)(23010399003)(22082099003)(18002099003)(11063799006)(56012099006)(10067099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gF/okKYeUk+F5jO7DcSYtFzlnRo4964glvVm+7BG7wHp7VaOsa3LGKFKvlHKF08ouD44hKD0ZgMaZpt53EMKqpW91ki73rDEbFLu/adl3ZFtrCRo4fZtL6d+aOrc6a/8FA0ju32x95+oT4WeB8PVsEki0FPHBgIuatx22JwGw2ZTPc6fju/c2Wi5+M4xtKOFFvF8/97gvdLMlEsCrwQcA+V76LKGffHjXJrxJ8mVGjkFtt8R8wGHp8FWJIglIvHLazfXkaRRo0AjXykDTp0ksuw6a/yryVSiSd0l2JFWF8XywC8DS24is89JFosAz9K3JxBVuZUEKZMP+1CCuZqKIvYiGzqSdypVbt+m128ZSzKJBACV/Umq59CghOMXaJroHOOll8xgEcn/t2t0dcc6Kj5bAQ2sV2h9nTg8S1o74B/GQip5ez7IPkUWHO0BeVrv X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:30:04.4050 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: db106da4-a699-4d88-cf47-08df033b7710 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000144.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7976 Content-Type: text/plain; charset="utf-8" With the blocked task activation path converted to use a separate helper, add the DEQUEUE_MIGRATING and ENQUEUE_MIGRATING flag in deactivate_task() and activate_task(). Avoid checking task_on_rq_migrating() in the sched class callbacks and instead depend on flags to know if the enqueue/dequeue is a part of a task migration. No functional changes intended. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 7 ++----- kernel/sched/deadline.c | 6 ------ kernel/sched/fair.c | 6 +++--- kernel/sched/stats.h | 2 +- 4 files changed, 6 insertions(+), 15 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7d1ffcc33fed..4cd69b08b415 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2218,9 +2218,6 @@ inline bool dequeue_task(struct rq *rq, struct task_s= truct *p, int flags) =20 static void __activate_task(struct rq *rq, struct task_struct *p, int flag= s) { - if (task_on_rq_migrating(p)) - flags |=3D ENQUEUE_MIGRATED; - enqueue_task(rq, p, flags); =20 WRITE_ONCE(p->on_rq, TASK_ON_RQ_QUEUED); @@ -2229,7 +2226,7 @@ static void __activate_task(struct rq *rq, struct tas= k_struct *p, int flags) =20 void activate_task(struct rq *rq, struct task_struct *p, int en_flags) { - __activate_task(rq, p, en_flags); + __activate_task(rq, p, en_flags | ENQUEUE_MIGRATING); } =20 static void activate_blocked_task(struct rq *rq, struct task_struct *p, in= t en_flags) @@ -2249,7 +2246,7 @@ void deactivate_task(struct rq *rq, struct task_struc= t *p, int flags) * dequeue_task() and cleared *after* enqueue_task(). */ =20 - dequeue_task(rq, p, flags); + dequeue_task(rq, p, flags | DEQUEUE_MIGRATING); } =20 static void block_task(struct rq *rq, struct task_struct *p, unsigned long= task_state) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 3e89b3abeb27..9abda560c633 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2530,9 +2530,6 @@ static void enqueue_task_dl(struct rq *rq, struct tas= k_struct *p, int flags) check_schedstat_required(); update_stats_wait_start_dl(dl_rq, dl_se); =20 - if (task_on_rq_migrating(p)) - flags |=3D ENQUEUE_MIGRATING; - enqueue_dl_entity(dl_se, flags); =20 if (dl_server(dl_se)) @@ -2552,9 +2549,6 @@ static bool dequeue_task_dl(struct rq *rq, struct tas= k_struct *p, int flags) { update_curr_dl(rq); =20 - if (task_on_rq_migrating(p)) - flags |=3D DEQUEUE_MIGRATING; - dequeue_dl_entity(&p->dl, flags); if (!p->dl.dl_throttled && !dl_server(&p->dl)) dequeue_pushable_dl_task(rq, p); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6d2fa0cd7777..244f1c68c041 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6265,7 +6265,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) =20 action =3D UPDATE_TG; if (entity_is_task(se)) { - if (task_on_rq_migrating(task_of(se))) + if (flags & DEQUEUE_MIGRATING) action |=3D DO_DETACH; =20 if (sleep && !(flags & DEQUEUE_DELAYED)) @@ -6663,7 +6663,7 @@ static void dequeue_throttled_task(struct task_struct= *p, int flags) * task is migrating off its old cfs_rq, detach * the task's load from its old cfs_rq. */ - if (task_on_rq_migrating(p)) + if (flags & DEQUEUE_MIGRATING) detach_task_cfs_rq(p); } =20 @@ -7981,7 +7981,7 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) break; } flags |=3D DEQUEUE_SLEEP; - flags &=3D ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL); + flags &=3D ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL | DEQUEUE_MIGRATING); } =20 for_each_sched_entity(se) { diff --git a/kernel/sched/stats.h b/kernel/sched/stats.h index ebe0a7765f98..d0d9cff168a2 100644 --- a/kernel/sched/stats.h +++ b/kernel/sched/stats.h @@ -137,7 +137,7 @@ static inline void psi_enqueue(struct task_struct *p, i= nt flags) =20 if (p->se.sched_delayed) { /* CPU migration of "sleeping" task */ - WARN_ON_ONCE(!(flags & ENQUEUE_MIGRATED)); + WARN_ON_ONCE(!(flags & ENQUEUE_MIGRATING)); if (p->in_memstall) set |=3D TSK_MEMSTALL; if (p->in_iowait) --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010037.outbound.protection.outlook.com [52.101.201.37]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04F42395ADA for ; Wed, 26 Aug 2026 06:30:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.201.37 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725832; cv=fail; b=D7Yb2eMED+4dSJ7OvNjcaJ7kdQk5SMe4MVKBHulnYuf2/6LcM9gRGAY4UXWaIVFt212G3arz6kSbKxEtjZadwXmX2NGtK+r0uemMk1UrwaN97LDlDPxR7tskPl2yaGD2I0tcFglbpGVNZj2uMjtPDiQQ/G0JING5OHDzc8IBW0w= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725832; c=relaxed/simple; bh=Av2Q/b0gTvyQBxAe4LtBbGIJwh0GslAue7Zm1tHYoIo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iwvwHZxKzdw2zHeJE6fQ5KYKVOQAApP7qULHXRC7yPyhRmI1kkycG2qJmaomNrEU+7uXNF3mx0eZi/UwfMFelDvieyHhHQzVKmQRdJEnSCFQYzv/LAK9woXBTNOcEBHP0E4KQXs66xWq+F64ltWfMcywLoy/3A3vftEwj0PvIi4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Bcnp1de+; arc=fail smtp.client-ip=52.101.201.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Bcnp1de+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=aqyF0U0/+RiyYl+oA/xmyUDO+gvNbq8I0KgD0L44Z9f9XFXoeSXZm9TrbglyxzPWGDgTgO8vIsx/55BXrsx1mY8/I4+3td4ChraYaM1sIdLZAwimb8OUbQqeDaJeKuPtmPMJi7BmWAhZTMHDSX9Uj9YDfrWTMYD8VOu2ZyKQjuJt+uB3RR5xuwa1DsR1v/hp6XT5hhOe8LjuSkJiYFh8ojW4XVq6YS6v2VqdCoQyauJDMmWOX40sANiGagHvmRW9AAHWSi4A/dGYr2a107BXVLAwq6h0LfzBXpR85o0O23FcFifDIAlBu6HnAbdW5Z1NMFv1n71SpzEwrQYmTd/4JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9n5LlS+SZWmSFhbpfYTx9YlhKJiF8ZICEM2Xlfoam/0=; b=F3TBHgKUGTr8zYNICmqT8Bf9mK0zouCG2VTczJd9+66H29Y/gLmvdGn53Mb0bI+lncdLmMFnLmXP1pnmpumjJfEO9elGF+likx+HCgP6ndMRKUMP/pHzDrjiukvS+5AVr5FfzmQCsb+bHK7m20uUDJ7RnjQCFuHRESK5KDk9TW+mO+72Wp2ct2aqvDNAlPK1cvgTFOVJbmdYhO2HqxX4YXRw91NUYiqZeUZDlNaaSGlKoxE0LbdRzs0+Is4C/94l2/Ia2J2fnigSkdb9Nr+mPUS3XMaAmFjfy6hEpJMzl1cTrTdDMHV8HbC6kMiOiV/P65dZg5IXdN+3m8rzHJo9GA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9n5LlS+SZWmSFhbpfYTx9YlhKJiF8ZICEM2Xlfoam/0=; b=Bcnp1de+vjlSNDf9itOZ3s6JWGB+WnYybvm0q+q4qKqYb5J8WyKSJOpHx1tS8aB2MD9ZdsOqsG37XuyyhxeH1kILDP8dRUidCP4bl1wSTL8Ts6v/RXsdZjc7oPWp0iVFLDJMmxFfKDgrAgpwWwWFSfpfkJH9tTkw3+lV7aLJETg= Received: from CH0P221CA0041.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:11d::25) by PH7PR12MB7380.namprd12.prod.outlook.com (2603:10b6:510:20f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.8; Wed, 26 Aug 2026 06:30:22 +0000 Received: from CH2PEPF00000145.namprd02.prod.outlook.com (2603:10b6:610:11d:cafe::21) by CH0P221CA0041.outlook.office365.com (2603:10b6:610:11d::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.7 via Frontend Transport; Wed, 26 Aug 2026 06:30:21 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000145.mail.protection.outlook.com (10.167.244.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:30:21 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:30:15 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 03/16] sched/fair: Use enqueue flags for DO_ATTACH in update_load_avg() Date: Wed, 26 Aug 2026 06:28:47 +0000 Message-ID: <20260826062901.2137-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000145:EE_|PH7PR12MB7380:EE_ X-MS-Office365-Filtering-Correlation-Id: 0f795cc5-1b02-4de5-a5f3-08df033b8165 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700016|23010399003|7416014|1800799024|10067099003|56012099006|11063799006|22082099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: xKFWTH26Ki4FeENm2V+og5z4gacUPWJmUz4xWdp8xKtS5mFU7NsHwjdP1w9XH9Sm5LDLt2lzoioOFRn/e38J3o+cDy8lBlvcMASFIvnwfsmVknfH4IMR7U2qCm7b34FYzHEeaPvpkbswDGCMchE8AmPrIY1ILAvXkEW5a/1BIt9dtRxrJXJerlMZDxLrdUcMUCZCeYQAUbD5v+vE9L1t5MUK4Oy1FrFc/rJP/0qqRMK7ski15GC9DZ1AXaVdFNuY1zgEqafj/u0SMwkmnb9ZCYX4ZGBZk9BIUxFfjcnnxM9pxE3yeB9Pz2YsUhgB2tqq1pmuQVMM2J3YkJ0VcJIjazyWIswyv7ASy7TT0UwcgyT2vt9MdBusfPXzfLr/WZQqnqfXy291jg+KOvseUa+2WKgbHqpiijyGlrkQMniKqYyd2wjR1aaxiUJmyVz7tVATIrCMol4+yaZLat6QhC8ej7HZ32o0J+10/vI0noJnedSj7oga6TsXDpnnRuNH1MIZRg5YjdRsO2y0mF4kF6fpokymrCgr9jZhROJhmxQVwg9FeJ1s5kImRokKAagIDSusyuciNnozMPrnfsPbJvvBflftMMip/1pIZGLPpbvEp/TGuCuuQ/EQibFan+iBHd503cjkZUrMG1dxiI8yHRj+px5s8WwweNgDTqWds8LaU7L84nk2FOSjNIRcVUCiXYJ3RDf+jNMY8AVWNCODbZZRKM8elAdWxA5cbZLhkaEi3yytLKMWL6OpJiytTl7oXb2a X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700016)(23010399003)(7416014)(1800799024)(10067099003)(56012099006)(11063799006)(22082099003)(18002099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: j0cZ+efSGRbUJf9+T/koWHL2vSqa+TrjWdQrXI5OLdTnH4+C67eGPoUqHcTa+jWLNp44dx56JmManzo3IzTEwdtJyy5meGLcv1BczADn9tGjufGUqhDIyhQf9jO4Prr98y2RTtM4ikd6jHUe9rGxnS0CyjuARSNTgcGQKM8cc+dSydkanW3Rz5xFdPUMCIAVlahGn4V8IiGgTku933vK3Bry1eRUN3PRgPZIhqhYKcEXdsBGqDVF+L/LtZf0yug6z5X5W7ZlhWPbkJJly8JVOE6/giTzorHFTovoAzM+HdC+TByqXFJsGSvxcVzwvwN8RZKMmtWUWqRSNmFH1jSW+U1moib1TZULMmTc9xhDhV/vBmz4YsP6LJoE4EI9yAdg0YFxDvFyYyPQV2icDRi7+MfMqZ8dBf8hMg+rhVgThWkm+QaRb9p25Us5hWMQ6VcD X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:30:21.7362 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0f795cc5-1b02-4de5-a5f3-08df033b8165 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000145.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7380 Content-Type: text/plain; charset="utf-8" proxy_set_cpu() skips calling ->migrate_task_rq() callback for blocked donor migrations and can lead to incorrect PELT accounting. enqueue_task() -> update_load_avg() depends on migrate_task_rq_fair() to clear the p->se.avg.last_update_time and use this as a gating condition to do DO_ATTACH which is skipped for blocked donors. Use migrate instead of se->avg.last_update_time indicator for adding DO_ATTACH action. Add a defensive WARN_ON_ONCE() to catch any cases that are not covered by the MIGRATING | MIGRATED | INITIAL flags. Fixes: b049b81bdff6 ("sched: Handle blocked-waiter migration (and return mi= gration)") Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 244f1c68c041..ed397c259509 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5654,7 +5654,7 @@ static inline void update_load_avg(struct cfs_rq *cfs= _rq, struct sched_entity *s decayed =3D update_cfs_rq_load_avg(now, cfs_rq); decayed |=3D propagate_entity_load_avg(se); =20 - if (!se->avg.last_update_time && (flags & DO_ATTACH)) { + if (flags & DO_ATTACH) { =20 /* * DO_ATTACH means we're here from enqueue_entity(). @@ -6110,6 +6110,7 @@ static void enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { bool curr =3D cfs_rq->curr =3D=3D se; + int action =3D UPDATE_TG; =20 /* * If we're the current task, we must renormalise before calling @@ -6120,6 +6121,11 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_e= ntity *se, int flags) =20 update_curr(cfs_rq); =20 + if (flags & (ENQUEUE_MIGRATING | ENQUEUE_MIGRATED | ENQUEUE_INITIAL)) + action |=3D DO_ATTACH; + else + WARN_ON_ONCE(!se->avg.last_update_time); + /* * When enqueuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. @@ -6129,7 +6135,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) * its group cfs_rq * - Add its new weight to cfs_rq->load.weight */ - update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH); + update_load_avg(cfs_rq, se, action); se_update_runnable(se); /* * XXX update_load_avg() above will have attached us to the pelt sum; --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011026.outbound.protection.outlook.com [40.93.194.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA5213B4407 for ; Wed, 26 Aug 2026 06:30:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.26 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725857; cv=fail; b=fNi0rJpmwznaWfXymk2au1p6GQXhFxLh6WNxLVY80Qz/xNcf9m2aHDaKOq28WXOq9ZMIo33fSGbP88LeLX/tcNl2AJV3Qd3Mte5XHh+2hELD39p/DDdSXQEWDANx9EMeUzzxXPmuKLmcXq3bKX9V+Dyf1vkbX2E51Ef1oyMw/QU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725857; c=relaxed/simple; bh=T3NMGknl2wQ+6uJGbazuRhpwkzUBa6XS52uVW4iTw8M=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DJPhkCIClEkNKY+C5EaKc0l1rcvJeXXW+XJFItlXWvi4hBT3pES7GFWyUnO9tOdLwV9s3LkNBcph/3/E/bRPdFe6BpFo1J1+NLErL1AiMWpD1IPHh58ewm1UmemIb6tzGYRQCOdadUNc26bwS9B945U2luGlIcpEzdlvzroFm4g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=tnwlGrcC; arc=fail smtp.client-ip=40.93.194.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="tnwlGrcC" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Y9zzSS7CXPxRYxo3s3q4OFGD7dyEOTJAlxr+qtPYrdzWfqhnA2rn9HKdtAEcensbvKJDN3bx4A+HNu4yn7E7KSNpvbqXzPDsnFav+4lTlJ97HAjjbuxjvnWTrLLipUDdYMpzmjenctAfqcYxAJIvRQ7uYYb3sRe50QgYdqWANUtywDtj6y65nqwfHtUNTgQrBcDUa3r763POjxnuwWjwmHe45RRdwZVFrPK5g+8kj9acHFrZ7WYLN9E3ny1sjgv3EulKlPUWkCcaw50tPFK+DrP6c4/a1FiLvtsr/VthZBE/7RnI9rXNtknmDJ1Xdd8y7tCEZlwE570LVvKTnPhcWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8sh35/wwhhiaPhou4AKCiqdBAOZwTcYCdh9QAXmgBeU=; b=lJOmtxruV6nLlms1GxXoyCIY4CDRaU/jI0OBL6vcvhk4Vvl+CSdwN4mu8/2kYZqSTHpTg91NLzPkbgT7pO3IQzMYRMK1QQZyoqmWB3ZUk2CflV72DJ7oS/Ry69hhLDzvYHcE4k2EsFgfRYb2/UW/naXUBzfM/xjCEq5jmHdzvKUj/rswm0Jw+sre21nOns1a1Y6Y3MI5y4xfY4++K4PAQegiwVUW20LRSgBDbVsz+rcZYI/9Um8vOxGjnqVuCUEd/AgGOmS9ivCDy4OW2mh+wNZZtKnI245oLWgoeO8tk+LSkahqBRGS0R7mJtMQmVppFk7wqQP57nveG+OatSBvmw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8sh35/wwhhiaPhou4AKCiqdBAOZwTcYCdh9QAXmgBeU=; b=tnwlGrcCwIcXAnliqFiMS5tPgB8/kXhWVXYQsqqupsozHEbbBxiBuv7IFW7K0CV9Li9NHPSYfgLd9qNlO/kOaOWXqZTCbIbYmMsqwTVzDQsAeSoHMSpK5TtR8rwfiHyKhhO4RE22ZgvtPYBktqfCg4UYJ60t6zVa2fB/vNGHhFs= Received: from CH5P221CA0009.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:1f2::14) by SJ5PPF2CD49510F.namprd12.prod.outlook.com (2603:10b6:a0f:fc02::98f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.8; Wed, 26 Aug 2026 06:30:44 +0000 Received: from CH2PEPF00000143.namprd02.prod.outlook.com (2603:10b6:610:1f2:cafe::ad) by CH5P221CA0009.outlook.office365.com (2603:10b6:610:1f2::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.8 via Frontend Transport; Wed, 26 Aug 2026 06:30:44 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000143.mail.protection.outlook.com (10.167.244.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:30:44 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:30:38 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 04/16] sched/core: Activate blocked donor when no owner is found Date: Wed, 26 Aug 2026 06:28:48 +0000 Message-ID: <20260826062901.2137-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000143:EE_|SJ5PPF2CD49510F:EE_ X-MS-Office365-Filtering-Correlation-Id: 5c7c6b61-8aaf-44fa-6671-08df033b8ed9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|23010399003|7416014|376014|36860700016|921020|10067099003|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: DwGDdzWfg+JA1FhHRXV+W7pQdHTXQLllpFLz+C0MsAWpAPxdRJdRw6xrk7XTyEgRJxtmRPfNer7tCmRG0Cpp9jng4rQQ8IuLlnodI+tt0FusBcu9DxXJYLTvRPLkLhjDhXL9U23j4sdXgETsxRx2+1OXNbEeeNVbZI3p4qpkUzmczmKbXwa5dLid9L0yaI409UsNM3t7q2ft8bbnGWiVeF5bCKg4ANXrwflvsuohnjko5RXjQxRR3OEXBCSjpJahpOG2wjnbkwTSRN76AvTcSDLBknYHGfq7P+2FSuR1AffDisjoPiN/LxHIbiXLMoDaxL4BB9abIvzgUurDn6FJ6Wifwi99UvKOmds67r/oLc8Gb6/XAwmchseYg0x57GN0dX70WVCeNJg/2VMZBnVi+mr72ALbgk0p1t2Jd0M23W2ruDXh1XgxtDcdryRjKt/tRZzU6sRCwwJ9Qvu1xmTn4pXicMElC1fj33fChnQXiNt0Vd0RZ6DJV8dVQG0G1kKaPvqml578qJT5gzNt7xmTEluHxGQLtm0r1B02phcjUyo+np6qr5aC95flNS+zowtfJ9uDHTHkjpIA+YoroVoVxh81Il2PTyaxWmDNIXcRLvQl/WuHtyqIAI61tb7bJxOr4X4vuc/nLswR67TZTsSF8gFm6zemqIjuG2sqsLFpNQIJRhISutUG0vbBfqrzx7xFT+6QjvLNIBZ92H3+1dhVfEIBwkJXSd8xiYRdETfKt4dBzKiUF/o2mi/gFzlCerEj X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(23010399003)(7416014)(376014)(36860700016)(921020)(10067099003)(56012099006)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: A4GQx7ldkjeb/FwlZv6z2+GXUOXEGvfiwvRCgfCH7RIYjoxbdPAIUfRwWCvnt9VBnrhRI3XSCAQ4Sg/KpGCWrPMLP/QcajgtE57Rv6CnfU64xadq3aeXFfjUXQNr09faxOBUbplBT2mnywqNe+vSt8iZ/DwhFbomAMIKyNK2PYFFOktSW8fH/qiqHdeEdytZhxQd/IdpBasSnleDYJCPMErtA61k4CRKlDfVgUTMeSgxKPAtFG/+/dstecX3qOafWEkErFsF8WMyhO0ndG5HZQsQ1/1YZ7ft2y1+56uKMaaebBu0Ine4DOteeUdd4Mh73b+lymuogbi/kvOt/YGGzTapditQCYThl6+rG+mn7pXvN1/ezC3h1zdSRgKvrJZ2Wf/7i1pv5OGdOjOn4nBRtFn/QrarYSfmfOBHKhW/ZsPp2RgBqvRPncYYyG0mHXYh X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:30:44.3105 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5c7c6b61-8aaf-44fa-6671-08df033b8ed9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000143.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ5PPF2CD49510F Content-Type: text/plain; charset="utf-8" mutex_unlock_slowpath() follows: if (owner & MUTEX_FLAG_HANDOFF) break /* ... and do __mutex_handoff() */ if (atomic_long_try_cmpxchg_release(&lock->owner, &owner, __owner_flags= (owner))) { if (owner & MUTEX_FLAG_WAITERS) break; /* ... and wake up the forst waiter. */ MUTEX_FLAG_HANDOFF is only set by first-waiter after it has been woken up and in absence of MUTEX_FLAG_HANDOFF, the owner clears itself from the lock_word and wakes up the first waiter to try a __mutex_trylock_or_handoff(). MUTEX_FLAG_HANDOFF exists to prevent new optimistic spinners from trying to hijack the lock from waiter all the time and potentially starving them but it is not necessary for MUTEX_FLAG_HANDOFF to be always set in presence of a waiter. If a blocked donor is deactivated when no owner is observed, it may not be woken up until it becomes the first waiter and is naturally woken up which breaks proxy in the interim. Wake up the blocked donor and allow it to grab the lock when no owner is observed. If the task manages to grab the lock, the block chain will follow at the next proxy migration. If the task fails to grab the lock, same situation is restored and everyone migrated to the CPU of new owner. Fixes: f13beb010e4a ("sched: Have try_to_wake_up() handle return-migration = for PROXY_WAKING case") Signed-off-by: K Prateek Nayak Acked-by: John Stultz --- XXX: Is there a better way to handle this? If we can confirm a owner in find_proxy_task(), we don't need to do a spurious wakeup of every task observing !owner. proxy_resched_idle() until owner appears in an option but it will spin until next owner appears. --- kernel/sched/core.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 4cd69b08b415..9d9db7ccf01f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6809,6 +6809,19 @@ static inline void proxy_reacquire_rq_lock(struct rq= *rq, struct rq_flags *rf) update_rq_clock(rq); } =20 +static void +proxy_activate(struct rq *rq, struct rq_flags *rf, struct task_struct *p) + __must_hold(__rq_lockp(rq)) +{ + lockdep_assert_rq_held(rq); + proxy_resched_idle(rq); + proxy_release_rq_lock(rq, rf); + + wake_up_process(p); + + proxy_reacquire_rq_lock(rq, rf); +} + /* * If the blocked-on relationship crosses CPUs, migrate @p to the * owner's CPU. @@ -6934,14 +6947,15 @@ find_proxy_task(struct rq *rq, struct task_struct *= donor, struct rq_flags *rf) /* * If there is no owner, either clear blocked_on * and return p (if it is current and safe to - * just run on this rq), or return-migrate the task. + * just run on this rq), or wake the task to try + * and grab the lock it is blocked on. */ __clear_task_blocked_on(p, NULL); - if (task_current(rq, p)) { + if (task_current(rq, p) || p->wake_cpu =3D=3D task_cpu(p)) { p->is_blocked =3D 0; return p; } - goto deactivate; + goto activate; } =20 if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) { @@ -7029,6 +7043,9 @@ find_proxy_task(struct rq *rq, struct task_struct *do= nor, struct rq_flags *rf) } return owner; =20 +activate: + proxy_activate(rq, rf, p); + return NULL; deactivate: proxy_deactivate(rq, p); return NULL; --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012065.outbound.protection.outlook.com [52.101.53.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D07A3A785A for ; Wed, 26 Aug 2026 06:31:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725872; cv=fail; b=Ak3MbMStHJTBkhz9Tai1MRDyn563C8gdTOtbOQhLoOy1f6rBf6yNbIR95cvb1ey3egSdJdvFJ+ZYBmRm5FcPNsQwwEYF+6TKg0ylVF1suIggU4fvkSm12EA/DYVLGGMBamIjQujXZGHmfTs3Iq7SF1rI3cAxcLKsiVnhNhSEd1w= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725872; c=relaxed/simple; bh=J0BJGDULnaAUjcrqMqxNnyPZ56YS43ix+B4YnnMYri0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=i67my5i3wqdDUrBciy66z+HaceOwIkZ6LSEcQktU7Wzh7NmP5teWH2xwNCGAeC5IcoYiKU9Dx6jluk7XRgTg5Ms1js2ERPnfMx8knRsReyKfEgNsNFfyYLqE1DYAv/ynbLx/5mElgzquG26ih4LesRyTPeVy0QrEA1xfYJKkQ08= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=UMMZxq5/; arc=fail smtp.client-ip=52.101.53.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="UMMZxq5/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lTYl6Uqh55tqqi/CmKpeYddQq2pgGf1hJjv3JLIbhSNz4Z9q0lfyLfrJ7S/7uGAcXspDwekJ6Ywry+IfRYuWt7vkjCVyTl1CEKRKKasVv/PbCo2YlzNyBPqpLjan7Lr3v1b59Ybxz84r8EImJEr/Ejur7eK2GQ3bQ7uLNVdgRDfrwvEA0rcuF8T4M+K1yTWR+WSAhBn2FTbC6m4q1lMXikzbrsqtk/wbJ3q5QAsjbRoTLHfPKf0sYXDOgn8HTAdMZBtIQi1TJb4KUYRuHMYjwhpIZI3JXCWg+Ft3UB8A1Z/GXi4Bh2+KYPqT0SAaV4RNW63saZmaVLHbi3wF8Yec+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MmqLADXKIjhpF9KXugYwWqT3fpS7qtgC2OG2hY15kMA=; b=mFdho8fozRyl/GSfjiGK45NW/XsLgQ1rbSqdcwgDVrKUl93XoSmwuJ2bZSkW15EXimA4qsnLF/pOmbzwKvxea+jDfSQqstN1PECeUbN6y4ahCSPn6DnHUeNXYRVXujwpSuHYMdM/NNHsqLZFfV0trQI8rB7vLfWPM0JZehkGcgmcO+SaIBxKIdmrwbzyefiGAvt9O1gDd96axB+AwsyNAegZrBnoJe36bTALdZBns+dj9em3BZ2haFx5ivpoJYPMPN1AK9jPEz+Bc3wkjDwX1ZaoHjbqVIMcJ6m0m2aAdSEdr3tAFTmkyLYENuAozrPBfx4QPurIhWxEB5qZq/vgog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MmqLADXKIjhpF9KXugYwWqT3fpS7qtgC2OG2hY15kMA=; b=UMMZxq5/iEzDSylnNXieSl02QWEQZkCHQ6/5D7FRHS7ZtbWZguurmuMq/4SannTjZOF/+dWZZ/Ie/S/pBhv44LDGv4vFUnfryXw+7EvT354S8vXqxK+Yk9eq2VuhaQZUvBGCZUhQHW5v0M0UQ+yeCK/1ePeIUfviM7sXO3LIcbw= Received: from CH5P221CA0019.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:1f2::26) by SJ0PR12MB6967.namprd12.prod.outlook.com (2603:10b6:a03:44b::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.17; Wed, 26 Aug 2026 06:31:02 +0000 Received: from CH2PEPF0000014A.namprd02.prod.outlook.com (2603:10b6:610:1f2:cafe::9b) by CH5P221CA0019.outlook.office365.com (2603:10b6:610:1f2::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.8 via Frontend Transport; Wed, 26 Aug 2026 06:31:02 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF0000014A.mail.protection.outlook.com (10.167.244.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:31:02 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:30:56 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 05/16] sched/core: Do not queue blocked donor on a delayed owner Date: Wed, 26 Aug 2026 06:28:49 +0000 Message-ID: <20260826062901.2137-6-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000014A:EE_|SJ0PR12MB6967:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d4c2a6d-ceaf-4623-ddcd-08df033b9973 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|1800799024|36860700016|23010399003|10067099003|921020|22082099003|18002099003|3023799007|56012099006|11063799006; X-Microsoft-Antispam-Message-Info: 53ZDgTZaNxkq0Z6NH7OgCyuS0cu+LRC1LnGNlshicfkmy497p62hbqL7p7Xb0nsrp+9WmwSIsWwYrC4V2qhZXA02Fk3ILIkyPoIliKE9ENuwSKdvByccWPojpFn4uQYozEgnHttuDXuSKnGLB7HrIr3EDV23EaM+bWHkcl6Scnv+vfyV0el4SUbtLGN+CyOpykaQj4+dIwNULN0qwIUrsAAsy4shl98grMlZR6g6ZbaKq86L6HuoW8yxRxeySFMXsLHHlsqkXLl886dDXPnqt8JEidBXHa/QDRgzPRcDNiDDNaXQtZ/WtMMzP0hcfHOO5j+47pAEZbg4kwKqU6uiPP2BeiV8MV7xZW6J/IBIASU5OWaAjy2+foCtHvRS4jUlUWAoVfvn4ZcCwk6rYkSALwGjJLDS5ZM2KtLDHBFk0BWdJbi329uCnLVJ5dMdWl1by/f1Lvt6t2WnJJOdjQoTsFWyYtdywsGuTA23cydQZb1+/fUAXtCC8cIE77M9k75CHc9oS1xu6q1UOCaTu8yKRiDP+9No6HlktSLSCQj6S2bD1o27Js1hAhKKagYqsYTCPW0sOHfgz6fZyL03QyXqR/mWISF8uyAZkGvYTtyOY0jeKQmJIUFjOU5fPu23luYXQ3xJ/az4Szw1SG63khRxBQvRqUJgleFCwmjz2d9Ezl10mnhnvbH/TMryFvjHBOTqgQ/+wh44jZIbM959jqt8jNSsGTRnKF74C0vXJC3tRksL3AzBezybWxL9Z+16S3La X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(1800799024)(36860700016)(23010399003)(10067099003)(921020)(22082099003)(18002099003)(3023799007)(56012099006)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: O6bnwluvpRuIp60APcKyv7yOb0jfGy2WuVX9tAu34IgBOntQgAjeLBIMyDBTBmGU1t30JyIcLcHpZFbce7F8hy8f4HwnxmLkYSVmWp3m7ObthR+/zPdMYb5iL7Do3gdmEw+47o9XDyUAlnfSaXXWNsSHWfmxFPenTM/xqRmslJw4Va8p64HS8J1YWgPeExDrtVmv7YZWZiqciK9UcVrj2Rii+0ukUBbTrrtJC+NB3NdBIwVfiCvJA4zeUQVaWQssenT+bav6Nh6pWycczBbHPkE5RLF5PN/FgQpLqMyKKeZkMt8FJ/d+VSXUbi2XP7m4DNjyE0t9/BVTrg2s8aL345aK3QCEkYdgiMXdyaja3eDy7+FwvmWT/x/M8MmfAEyZZ+3chav9tiRuWXT26/ykI8ptneQq6mCa91i3GFsT/nxO9hfzF4I7GpjHpDMqfuky X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:31:02.1271 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4d4c2a6d-ceaf-4623-ddcd-08df033b9973 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000014A.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6967 Content-Type: text/plain; charset="utf-8" Delayed owner have a stable task_cpu() since they are ON_RQ_QUEUED. Move the donor to delayed owner's CPU and block the delayed owner completely if find_proxy_task() resolves to them similar to what pick would have done. This avoids needing to handle chain-wakeup for delayed tasks, simplifying the ttwu_runnable() handling in the subsequent commits. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 9d9db7ccf01f..b4a2771eb290 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6958,8 +6958,8 @@ find_proxy_task(struct rq *rq, struct task_struct *do= nor, struct rq_flags *rf) goto activate; } =20 - if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) { - /* XXX Don't handle blocked owners/delayed dequeue yet */ + if (!READ_ONCE(owner->on_rq)) { + /* XXX Don't handle blocked owners yet */ if (curr_in_chain) return proxy_resched_idle(rq); __clear_task_blocked_on(p, NULL); @@ -7025,6 +7025,18 @@ find_proxy_task(struct rq *rq, struct task_struct *d= onor, struct rq_flags *rf) */ return proxy_resched_idle(rq); } + + /* + * Delayed owner is on the same CPU and rq_lock is held. Treat this + * like pick resolving to the delayed owner and block the task + * completely before retrying the pick. + * + * This prevents the need to resolve "owner->blocked_head" chain for + * wakeup of delayed task and fast-path them. + */ + if (owner->se.sched_delayed) + goto block_delayed_owner; + /* * OK, now we're absolutely sure @owner is on this * rq, therefore holding @rq->lock is sufficient to @@ -7052,6 +7064,9 @@ find_proxy_task(struct rq *rq, struct task_struct *do= nor, struct rq_flags *rf) migrate_task: proxy_migrate_task(rq, rf, p, owner_cpu); return NULL; +block_delayed_owner: + dequeue_task(rq, owner, DEQUEUE_SLEEP | DEQUEUE_DELAYED); + return NULL; } #else /* SCHED_PROXY_EXEC */ static struct task_struct * --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010041.outbound.protection.outlook.com [52.101.85.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCCC03B6C18 for ; Wed, 26 Aug 2026 06:31:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.85.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725915; cv=fail; b=mSrNHG6QYI1MdSLzEceubpY/gXGvOXrXYaLAI4hXyYNujZCQ6NlnSsLKM7W4tDWX7mm0JqXcUU3qPSHvlU5YQDNbjRbplYZYz167plzF9n+NyrUXfDfrXP4EoXL9uE2ODflglUht10IxrQbt5IPkVoO8oWsFZ8Kr7PHIGEDvHiQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725915; c=relaxed/simple; bh=I5B8CRJUX0Z9y/JJPNubHnTU6633zFjaPDasLepyk00=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ShJGnWYB70tkSnh/iD0wG340AHHd/Yqf2P41wPKvJ4HjgmAF141ZFUH9SaNApQBfk55+FUHOQwj8J+kwzbEaLjeRJgnFVdnZ5Eo+W4OMoBqk78G+igbQ3L+sH9pZDRjR2VrJhNFUOVKzwUEyByjg5php5YkRebazZqjpmLrJtyo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=LBqfT50F; arc=fail smtp.client-ip=52.101.85.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="LBqfT50F" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AXX5y0ncF7it6wZdAcb8AXCoHfH91kesLuctnTU0pY+/iQF3EzVqrpoMX9dyS8hQP2tlImwDWOoehcxxGhAE9UNbzGse9rUw1KrGU9gpmmmLrK2QDje3xQ9Sifdqb9+CUeGJ4OP0IbJAdU9Vi9focMjptEW0oLCpffZqH9I1NOxG/FmglYU3yGfWHr0TJf7ECPRgxsyuWqAE68bgKqMMQFpxOTU2eTV8gNBYj1co6n+R/jMvjJIWCv76jHojO9NS4nHMwC9PkF0+ID4xasKzZZcCBgoTNuxMbEns2+2cM0nFymq8AbNz7J1br3+FWfboj0rGgB/KTXpzgWRTxGQfXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UrIhgTeRXP9uV+VqG3fSVK8czaRuZZfhJlF9VmJbTyc=; b=bvJQeHu21V7aHw2LLUYf0F8qt+MZI5idqpPgAe6B+12gwaItoUfTvbBcNpyXUH5erYWqhk/HT318MNeQ9qHGOgM5DLY5vZ2lDFmoNqg/qhwTh7jyo0zn3mksDMkzCq3P9TNrNnab3mULcu1L3/jipeVDoJUU/7vppB0BW/UBRBcGTUCbzhNmZj5QueXoEFw3TasBXmdeApeeBZnGBXHBf4gXSxLhINPuRN39oINkMcRVxJHIAR0P2dRfk9U7cCWznraMS2260zkINoeb3QNsQHDC7V2yG0wsaUrExFzZKHO6QHDU1HUEegt4UCwB/lHYYf/stkP/zlkgubnlD6Pzhg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UrIhgTeRXP9uV+VqG3fSVK8czaRuZZfhJlF9VmJbTyc=; b=LBqfT50F6jlOu5gUP1gBSATXtwhC5P4127ld3OjEbpApeXZUgb2xoc/vfXzn+HLpU4lM3UKN6rLLO1BzyvtstwUmZYSavJJZZSGALsbkvxJwwWQh2GJXQh942n2clIUrcooA0BqVPkjAZdPd37a8iFCVXGdVMLxS43QmI32EY0o= Received: from CH0PR03CA0105.namprd03.prod.outlook.com (2603:10b6:610:cd::20) by SA3PR12MB176418.namprd12.prod.outlook.com (2603:10b6:806:513::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:31:49 +0000 Received: from CH2PEPF00000146.namprd02.prod.outlook.com (2603:10b6:610:cd:cafe::17) by CH0PR03CA0105.outlook.office365.com (2603:10b6:610:cd::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.7 via Frontend Transport; Wed, 26 Aug 2026 06:31:49 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000146.mail.protection.outlook.com (10.167.244.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:31:49 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:31:14 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 06/16] sched/core: Queue blocked donor onto sleeping owner for chain-wakeup Date: Wed, 26 Aug 2026 06:28:50 +0000 Message-ID: <20260826062901.2137-7-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000146:EE_|SA3PR12MB176418:EE_ X-MS-Office365-Filtering-Correlation-Id: 9116e675-5a80-4d12-bc4a-08df033bb5b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|1800799024|23010399003|36860700016|7416014|56012099006|11063799006|18002099003|22082099003|7136999003|10067099003|921020; X-Microsoft-Antispam-Message-Info: sHCbWFImd90Qkjwfbna7uoavOJudNTE8wtajjnThvXOO1NsY1IQSUST7kaFyYh70PB/+J3NmPgf3v2JhexEpM4ORjvRP8OP+qrhC5RytdEgpqRzMfjYW7LRQLHRze4daUom6nkgch1uC1cAjZ9Z9rxtpx26RixMKYldaU8OCD2advYFLV48RWZE8WR3drL7oWliUruSu36wQYUSmueRL0RGIoTvW4KE3fb56V6HgGJsGzGr7ontdiIoK4S8DcZVyswFvs3SxynAOUAcHqz0ZpMQ8qHOnbZoSSLGxR9blOkeRd24GahKqMx7LsrE6xwrsMxipwoS3vjKq+7gH++1WDAbjraT6DNHUUH4UaAIbXrxdtoZPVuARy9qHo08A6TpHReXECrAJM+zrFS95i6s0/u5rckEWgba7G9MwKtxvZGKGokNL5shpRjNMBnksS5A6Dlj2U8rtpc1FmmwDIyYcZHXR0d41UOGGC6sjcqOIp4xZ+zJOCDxUbT1YtmDDm/b1rQ22nQXeNfVmLMlyW5IilJUANsyxLlpcpTVOl//F5FEEMEUzYIiQOdep+sD7kDfSe5+bB+/59xogBuq9eg233F93XF8K1kQ6XuC9R7CcSOGoqmMN95kKR9CGTwSK3B2FAbdSmqd6uqR1E5ljv7E3QIP2VyAHjzasOhAyvp8G6EeRJE2NMxxluS9Xpzs3guKY9KFhIRWCtRhKJ2lra7/fSn78K/E8UVKUqj2effJEFl0DUbWsk52krP4AQxAZzVue X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(82310400026)(1800799024)(23010399003)(36860700016)(7416014)(56012099006)(11063799006)(18002099003)(22082099003)(7136999003)(10067099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iaounACrVVVfIImg03KKcY6t8ZAskGBmZ8aotZff3WTb9YeD1Hxcn8k4V2Ly2SsuVEGr7LSbAURsSy5KP7Qxs81trEwQ0sfNwcMHWc2F/5D1nLtEcBltRktDjKSs4rVouReo/9a8NYxG57jQoRFhN0j5+cQ+4KaqdUwAZgxs3iRIwaaS1nyBB0fbXpAXfLBDY/JdMRKmavUY+JknAailNXqGUKbm/QrbEIMZLYPdQkcWzb/+nCi9vhcCOutsz1Fbyt5vymNjeRuIoEchGnyM+2PtvJGOI75/Pe9UVO6H8zTJKDhP9n2zV+nHNMORLkQ9u5c3pe99qhz5BvmR+4CoD+H6TXENAybsUsGcjhzFY3iDcgb1htKmqtvkrFxGvfkZxM2pYQ1/QQke9NH4u/Zk5W2bwes0DLkt26/KRuIiC0O9JqDJlA1HDqX6RSwLUUeW X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:31:49.5341 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9116e675-5a80-4d12-bc4a-08df033bb5b5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000146.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB176418 Content-Type: text/plain; charset="utf-8" Add a blocked_donor to sleeping owner's "blocked_head" when find_proxy_task() resolves to a blocked task. Once added to the list, double check if the owner has woken up by checking "owner->on_rq". If the owner has woken up, remove the task from "blocked_head" and continue try find_proxy_task() agiain to re-evaluate the state of owner and take the correct steps. Since find_proxy_task() is called with wait_lock held, which the owner needs during unlock, it is guaranteed that owner cannot disappear under us in the process. The added data memebers in task_struct serve the following pusrpose: - blocked_head: Contains the blocked donors queued on us - blocked_node: The list head used to queue onto blocked_head of a sleeping owner - sleeping_owner: Sleeping owner on which the task is queued. Co-developed-by: John Stultz Signed-off-by: John Stultz Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 5 ++++ init/init_task.c | 5 ++++ kernel/fork.c | 5 ++++ kernel/sched/core.c | 68 +++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 81 insertions(+), 2 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..bf0f4b6be7c4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1251,6 +1251,11 @@ struct task_struct { =20 struct mutex *blocked_on; /* lock we're blocked on */ raw_spinlock_t blocked_lock; +#ifdef CONFIG_SCHED_PROXY_EXEC + struct list_head blocked_head; /* tasks blocked on this task */ + struct list_head blocked_node; /* our entry on someone elses blocked_he= ad */ + struct task_struct *sleeping_owner; /* task our blocked_node is enqueued= on */ +#endif =20 /* * The task that is boosting this task; a back link for the current diff --git a/init/init_task.c b/init/init_task.c index b67ef6040a65..a097c0def4c4 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -211,6 +211,11 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES)= =3D { &init_task.alloc_lock), #endif .blocked_donor =3D NULL, +#ifdef CONFIG_SCHED_PROXY_EXEC + .blocked_head =3D LIST_HEAD_INIT(init_task.blocked_head), + .blocked_node =3D LIST_HEAD_INIT(init_task.blocked_node), + .sleeping_owner =3D NULL, +#endif #ifdef CONFIG_RT_MUTEXES .pi_waiters =3D RB_ROOT_CACHED, .pi_top_task =3D NULL, diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5..88f2b6e08c46 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2247,6 +2247,11 @@ __latent_entropy struct task_struct *copy_process( =20 p->blocked_on =3D NULL; /* not blocked yet */ p->blocked_donor =3D NULL; /* nobody is boosting p yet */ +#ifdef CONFIG_SCHED_PROXY_EXEC + INIT_LIST_HEAD(&p->blocked_head); + INIT_LIST_HEAD(&p->blocked_node); + p->sleeping_owner =3D NULL; +#endif =20 #ifdef CONFIG_BCACHE p->sequential_io =3D 0; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b4a2771eb290..e53967a126a9 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2229,6 +2229,16 @@ void activate_task(struct rq *rq, struct task_struct= *p, int en_flags) __activate_task(rq, p, en_flags | ENQUEUE_MIGRATING); } =20 +#ifdef CONFIG_SCHED_PROXY_EXEC + +static void __proxy_dequeue_from_owner(struct task_struct *p) +{ + list_del_init(&p->blocked_node); + WRITE_ONCE(p->sleeping_owner, NULL); +} + +#endif /* CONFIG_SCHED_PROXY_EXEC */ + static void activate_blocked_task(struct rq *rq, struct task_struct *p, in= t en_flags) { __activate_task(rq, p, en_flags); @@ -6872,6 +6882,43 @@ static void proxy_migrate_task(struct rq *rq, struct= rq_flags *rf, proxy_reacquire_rq_lock(rq, rf); } =20 +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owne= r, + struct task_struct *p) +{ + lockdep_assert_rq_held(rq); + lockdep_assert_held(&owner->blocked_lock); + + WARN_ON(!p->on_rq); + WARN_ON(p->sleeping_owner); + + WRITE_ONCE(p->sleeping_owner, owner); + list_add(&p->blocked_node, &owner->blocked_head); + proxy_resched_idle(rq); + + /* + * Order against __activate_blocked_task_slowpath() checking + * owner->blocked_list after setting owner->on_rq. + */ + smp_mb(); + + if (READ_ONCE(owner->on_rq)) { + /* + * owner has woken up and may miss activating us. + * Remove ourself from "owner->blocked_head" and try + * find_proxy_task() again considering the owner's + * new state. + */ + __proxy_dequeue_from_owner(p); + return; + } + + /* + * Owner is fully blocked. __activate_blocked_task_slowpath() + * will see us on the list during wakeup and DTRT. + */ + block_task(rq, p, READ_ONCE(p->__state)); +} + /* * Find runnable lock owner to proxy for mutex blocked donor * @@ -6962,8 +7009,25 @@ find_proxy_task(struct rq *rq, struct task_struct *d= onor, struct rq_flags *rf) /* XXX Don't handle blocked owners yet */ if (curr_in_chain) return proxy_resched_idle(rq); - __clear_task_blocked_on(p, NULL); - goto deactivate; + /* + * If !@owner->on_rq, holding @rq->lock will not pin the task, + * so we cannot drop @mutex->wait_lock until we're sure its a blocked + * task on this rq. + * + * We use @owner->blocked_lock to serialize against ttwu_activate(). + * Either we see its new owner->on_rq or it will see our list_add(). + */ + WARN_ON(owner =3D=3D p); + + raw_spin_unlock(&p->blocked_lock); + raw_spin_lock(&owner->blocked_lock); + + proxy_enqueue_on_owner(rq, owner, p); + + raw_spin_unlock(&owner->blocked_lock); + raw_spin_lock(&p->blocked_lock); + + return NULL; /* retry task selection */ } =20 owner_cpu =3D task_cpu(owner); --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012049.outbound.protection.outlook.com [52.101.48.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99772131E49 for ; Wed, 26 Aug 2026 06:32:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725935; cv=fail; b=pjEOIrdiI/KcK1wDqSAovBQAHl2u60RRA97PbIG3Vve8r1QUQFUjamS9dQFRm2P3hdNgS/AaE647HcRjiLK2ijwQxFPn3yBNLvHaMVUk1kOXLcuFIULYWNJrywe1r43Kk0MuMQBejPoNzobI6sY5bB72XnZsiTLKogzD1EWzBAw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725935; c=relaxed/simple; bh=6ZDm84LaGmwQU4z1AX3kg5/U26dAK9z+1ibPSS7j3gA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eisXMacWl+Oe0NGMln/t+uM2z8u0GJX3vt/DcGn/20uaDUu0Yd0Yqt1OzK6pVRUIVggkWomqeQ88RLAnDg2h/ZdNKT+lqixUKsRlRrth1S2BH/XnV/G61dWogox0qupPaJQ3zc6Uu+/vOzPHpg1fixK+jPX+2YY3het/4AZs4MA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=JzRzCi+t; arc=fail smtp.client-ip=52.101.48.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="JzRzCi+t" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kJqh5C/p+4jJVkqs/N0b9xwcWWdEl+6U1NwHMuVLSbh8Q7AHa+WRbDqmzX5/QhGoDNxxQZQstVuDn0UBQIzUit6IH/peCd1geGMm1ul92LC1SncklNqb3CquxTBceDKD0Q69HjmsCQcCj/cl02a2fjlQyl2NQ7D5vvpHtZWa8tEZ10SNhxkb5UpkmroFG50pYzL/e+r02JI8gLYlUw6MGj5dJM4HIZeRkaFDViQ36hQa76q8UnjRpISB26Jt4HYPE4abrOTyz+qIxP5JAuvLbKhZA6nMaZcFjcyKXYfJcTCKlmle587JOq9IGZ1H5rCsxEM1GR6OMmt0EcyO9aHn/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nQga5/CLKVJbKErpp2tJummIwIaFo68uPzKXSqQel/E=; b=Q5ckcBKMpOXeA58SzHULfz11eFPcGUU9+J2g6QTiHn8CwrbMeEy4gz5PXJrefWkvwoxHM1O2mQ2jglGJnfp7hBoa44pDood36vUFRxwQvzW838hn4lJFSRjdRzFqpnCHiuZQE01qem+1vbybz1V4aIVWKZV5rRVxQYGZdD/ldfkgULab+e3kcbD4G9g1+ToM/Xv/y85ij2JRfbZKkyUdyXCDOleNGp2TlKvBZA1elb5kPqnuqcyDGmlZElTI3qG9vPnAC+1wEKNFojdUbqzd9BGEBQJl9vQ9LTEBaG4Uo6zYMvE+4dZExCFOFtprKefGz26sYFkVczy6vo03KuAoXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nQga5/CLKVJbKErpp2tJummIwIaFo68uPzKXSqQel/E=; b=JzRzCi+t6wzbGIHCrsxlk2d9rXfbChwTdLrqGzK6eLPAYa2rjmWLguVs2U51UieX3CuDnonlAdqaXc74EhnjwbyEiMyLqbxjBjYmD9fTc/ADocuqcnyppkMdSE8lLpGU2K2NHTdAt3TiN7KkGdXDcQvYmKYdldhlsz9IJmyhWk8= Received: from CH0P221CA0032.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:11d::11) by SN7PR12MB6669.namprd12.prod.outlook.com (2603:10b6:806:26f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:32:07 +0000 Received: from CH2PEPF00000145.namprd02.prod.outlook.com (2603:10b6:610:11d:cafe::4) by CH0P221CA0032.outlook.office365.com (2603:10b6:610:11d::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.7 via Frontend Transport; Wed, 26 Aug 2026 06:32:07 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CH2PEPF00000145.mail.protection.outlook.com (10.167.244.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:32:07 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:32:01 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 07/16] sched/core: Avoid delaying blocked donors queued on sleeping owner Date: Wed, 26 Aug 2026 06:28:51 +0000 Message-ID: <20260826062901.2137-8-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000145:EE_|SN7PR12MB6669:EE_ X-MS-Office365-Filtering-Correlation-Id: 239f09f5-7f6e-4d7d-c20d-08df033bc02c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700016|1800799024|82310400026|23010399003|376014|921020|10067099003|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: tZ4azmS7sxwkdJ5NdMKZQR9GS4cTwCgCZP+5xD6cv5INk/SXmzefDmgtpHGNA2N7AT5wrZxe/uWp6iuy1e7GjQdRqqnjwwKJFmAqSOAqcvJPrQ7w6fyFiydDNg8jZHpsGgN++4qF7yg8UJCIb9BgTMXSPYEis3jkPQn18fz5hO0drtVV1+gUw//ZKizozenDKGQRTk8U4d/lRWaPDrbGOxqo3mEYp/gUozl5jomCNj8nL67pvE0Ljcj+qNYhz5VyNEwZU4a+2oEHKE/cX/x19x7R4mpwhcBsWS87fMQtdQb6stGmxrkGhstWJOu0DuN2B+MHRKOXnzhGAoBZc4vJSfgPfG7cNg58oCoYaW1rfraVBdiRpN0XaaZZBg6Ih623dGDfE/F1wvDpqSVxb1CYTZKEV26wR2CWPkgG9tVlSNh56ZW+mOyRkF5suYsbaFEeKZCtXhZI0rXp73r91q7Gos9C2kiLbMIXjvrIONLioX3J67jQCUz/mnQyjS+Vu4I0jMDCoG7vb+VcVd/Onuflt/4oegeezVjDB4RY+VyPKhHEQKO0SmnKEmvtGdOr0kh9cwZ0V5odMSZTjmKZoTLDCHk/zn3qJ/9CI81Z6TPMoaKRk5xYELtwfvChNNDn3KMPz83TpF61LU/0ms+xo6w1JUswZqcqknpnO2kYgN3/mVk7ewaCmGcX7rFxeoultc9p5sIqG5VjlRO17vhoMEm+wiSRcraM7o0t3x9SUwBujaL+GEntQ4dlWuP8hVzejDhZ X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(7416014)(36860700016)(1800799024)(82310400026)(23010399003)(376014)(921020)(10067099003)(56012099006)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: hR5sltNH8RCeBbB53a0D+i7ryIuRg7rlh5jShLB4x9tQH8DKg0ISkEYy+/QbMYAZIOSCnxIjAEwUm0H3WBXhFY0PXJcz8WBSO7s4TQRkddFtP7p4+q73eHXdmdWva0mnhzQBQzrHYIJnldcsVTLtcr/3gzVauun7ip5EEjvvzvZkOt2IMkQ188GrLD/0yTUzjiPR9lUGraru0EwiZMz1mU/dP3mHgqY93QsaEBOrRJEK2eZM12PxbYDS7QLNS78kGQmHK/jqtUlpT5j0nHgGfFkxdOHuBPejHMKCaxeo/AMQ7CJUpQmi+QNpJnV+2ku01QKv78wjOWPEW3OOBZ2vGV+RFXkoXyEqynNCKjavxzBYAfwuwIfEN+sZZaybnVutA1UKbA0hWM+mZjBgRmSBdTP3yQqJNxO6a8L2JeAyOHdBZiOHNzjEdAJMuzohzbQq X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:32:07.0834 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 239f09f5-7f6e-4d7d-c20d-08df033bc02c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000145.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6669 Content-Type: text/plain; charset="utf-8" Subsequent patches will rely on the fact that tasks on a sleeping owner's chain are all owner by one CPU (task_cpu() will all resolve to a single CPU). With delayed task, this assumption will not be true since the delayed task is owned by the CPU where it was dequeued until the pick fully blocks it. Block the task queued on sleeping owner completely to avoid this complication. Further optimizations can explore moving the delayed task to the p->blocked_cpu and wait until it is dequeued there but for this PoC, the simpler approach has been adopted. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e53967a126a9..d635f5452a1b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6916,7 +6916,7 @@ static void proxy_enqueue_on_owner(struct rq *rq, str= uct task_struct *owner, * Owner is fully blocked. __activate_blocked_task_slowpath() * will see us on the list during wakeup and DTRT. */ - block_task(rq, p, READ_ONCE(p->__state)); + block_task(rq, p, READ_ONCE(p->__state) | TASK_WAKING); } =20 /* --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011061.outbound.protection.outlook.com [52.101.57.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73A3B493644 for ; Wed, 26 Aug 2026 06:32:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.61 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725953; cv=fail; b=SZ3oOIhPNR8xrkrdNv9Za855YDnSfTMSbmtgXJgoxR2d9hB4hGtkW6OUi9cbnt7IXDPrlta4mphNX0VSEJ2D+l/gHME6k3zZy6ymEC9SJbGqfoBIjoWk+YrwOtBloGAszVwly5P8ertRXWMt6IxXgVJQTyQGSWSltb13Q1cFqiA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725953; c=relaxed/simple; bh=giC1iICNCedhTFOYbPbrQRdvrU7XymKvi2vwmkJQFtk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=j3/LdgCjUTbZml6miwaE0OblpVCxNSORz/bGmVg9535oMZxS2fIxPeHF4+O+w2PA5o3BYan/BYqeOeq33N4COUwhYvnostbyzD04V+yIslLuexJtkE6jbErEqiYt1eUzjqjSLF86lRJIwbGQGaC7To/NRm6gZ70JMy1mEbWqSmE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=amHvVRRT; arc=fail smtp.client-ip=52.101.57.61 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="amHvVRRT" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=G8lp4KvMEAGy2Zie0yTLlC4JYrD36nHkGN2YqWZxGEfXb0TQf6pZ9pJMGmH1nDBcPvvQeqf1gg8UmGI4hVdQYj5wU9CCf7X8CQcUohCSNEZ+BsqH1So5c89KwUcOjU3iVKfFgL6UFbh8uP0IngSAp/86jI0Jo1KF3aDHl7il6e/l2zIuCqIejrQ4F1ODLEZV1QkhBhjkjTsZvmpBfVUVM1NrZoxcUY82kGm/+9ZJBUCcL6+8G7tBgSKRNr38WKDEUWi/NqyJUp4NmfZ5VxDevCVJWFmRBM1ktShF3Y4u+XEcyk2yh770Qp/dgHYMhLcZukciT6e2OEXJGsTuiEU88A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0pQ/Ntwg+e+dxUzJ6Idw6IUV9Bk6PgLTdJc1rCkiiDg=; b=BFu1QowTeGVR8oh2nc1FtOUMJiVaa6pwn7TOh+rL8bgth2kIRWgxwR0P3pYpkFhT0k8VAMRuoSqqcwve5Eb+N88rcF640ePvMn0cEaRlivlhwHeaKD5mAvvQVviN9GIZrt8AtuF/WReT3jsab3n2m2yrqtbvsoE+iU0/rY+EtZoUbAK7yaxnEVZdL0yFLuFRGX17/m3fwyNqFj2Puy8zGQUTnC3lPKyDPEBbcuj0asCSWwVTrILm11WQcoAqk3bzMJ9Dn6bAmVxcNz/NCS7Htt9TP1ZUpXJLIXUa6R/gEeMcx4OUdvTxgpnKOSnoI+nJPeVwbwZVlhUexnn7oRjsDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0pQ/Ntwg+e+dxUzJ6Idw6IUV9Bk6PgLTdJc1rCkiiDg=; b=amHvVRRTDrFhpUdNFqHsEmZFYzkwunuDygQ3oNgzHF66XjCHcNe7ZRbDnBKQ7lFQYg0F4GNKS/w1HF+QFLsuXYdv67pmBrVmB1WgMd/qh0rU1M3XIi0HYmBw+Y5LVHvj1S8PKOZTv+6ECTBoTytXN5ZMLaVzW6+npDvPScm9r44= Received: from DS7PR05CA0100.namprd05.prod.outlook.com (2603:10b6:8:56::23) by PH7PR12MB8016.namprd12.prod.outlook.com (2603:10b6:510:26b::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.8; Wed, 26 Aug 2026 06:32:26 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:8:56:cafe::1c) by DS7PR05CA0100.outlook.office365.com (2603:10b6:8:56::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.5 via Frontend Transport; Wed, 26 Aug 2026 06:32:25 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:32:25 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:32:20 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 08/16] sched/deadline: Prepare for blocking and proxy activation with MIGRATING flag Date: Wed, 26 Aug 2026 06:28:52 +0000 Message-ID: <20260826062901.2137-9-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|PH7PR12MB8016:EE_ X-MS-Office365-Filtering-Correlation-Id: 759a5867-d74d-4262-598c-08df033bcb40 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|23010399003|7416014|376014|36860700016|921020|10067099003|56012099006|5023799004|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: 1zocTk2tNzhxFVHwAKa8FM0eiEfBQm4ueW1et9Rda4URVpX2UgskQX+OIfhTvbdkSel3o2bRrOjAmNb5jvlTRMKPQEB+mm/l64Utg2R+ZodnH2G000PCaovdwELyTzSj8lHZ1bFtAgNCKLyy9Ln0+IWwgBGVo0qRqJTwKfo8Mxq2mPzua/lYHq42xWiBJvsAm6TObicB7IYqTZfVnnRFLOosewG5qe6lG9fyU3JUUFhLseHQksBFEezTuCkJWtprGdpKpchWCn/2qpj2jNby+OifEyZvUHJjL2VGvtFXZQ/f8gOoouyppfvQzv12/NUKvs42nrYXIoZgq1QJwMVQQmgO77tjgR3BSwjDBpn01KXXp8+tzqTDsVzGG3+TCiCCXWmA6vIAw9q8PmuaPzpp/Ojei2PGK1oxzdj5MXPhs2+0QxCAeqmSInfwf9KBksJfNBHfkSvA/spsTeFttyvu+WLbYZF78t/g7EM0zmIbhmzjrENKJJIvGmhq1jjgqBYV8UPZNuaV72Aj0armEzhu9kIe9DRgPnse9wvRJadK61LF6TujwtMyTkM+4e2PbGzH/A39oPJzQxEk9h0VmaD7FfJyKrO+dOlXZDZNj7G3iWKXh25fJz5FhrTY+qxyYarR26yMlgBQUx/wvU64gHW9xJ/NAV71c8Jicd3l/P2+COtLOu0E4YD+QMhpZ01LdnsB5eRh9pbD1BApUIhNizWZyDoR3ZDzV7rmByycJn2ejppbgmhTH7YFcpctEk6oGKex X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(23010399003)(7416014)(376014)(36860700016)(921020)(10067099003)(56012099006)(5023799004)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 3jb+kc+2HPtBYRrSvPkFg1CwmPAGwKPMT9W5W8P46rE09mShUTqQjnkn2NLbXa2/RfnK+28P20bou3t2lOOJTK8IhyUJ5C2//ZKghVwPAONPO50TKL5CphSs0cFQNByZDOYnqveftAmyXLgBSLMYWrU9W0A+pW9eeRU2v9Rq8Lm3drYlKmqWV98BOWGYt3hp6y1/0+/iz391leIPDIxZy8kHhqeB2cAFNxSMZv3h47whjJFleqAQ3tXJrndnt5Siki9BMBh0tEbcv2nK6pQeXxibSbgVELjI1FzIPUsnk+3iuJIp7nUdDEswLTEBmmGO/fis8qKSUlBKTKehtW718CkfnPRa8WvfvT+Wkoa3VrILgo4RsxlUlvFpGYcfkeIq9D7c9fx6XiRTxUu5wY52Iq2rE6mmLCBRzy19D/6McAgUwWNdZWgzQDLA1fSzxxEH X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:32:25.6942 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 759a5867-d74d-4262-598c-08df033bcb40 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8016 Content-Type: text/plain; charset="utf-8" Blocking deadline task sets task_not_contending() to retain bandwidth and start zero-lag timer on the rq and depends on migrate_task_rq_dl() via set_task_cpu() to remove them if the task migrates at wakeup. When DEQUEUE_SLEEP is coupled with DEQUEUE_MIGRATING, the task_cpu() can be switched via proxy_set_task_cpu() after the task is dequeued. To balance this, enqueue path will add ENQUEUE_MIGRATING on the activate path for symmetry with migrated blocking. Treat SLEEP or WAKEUP with MIGRATING as simple migration and prevent bandwidth modifiction, inactive timers programming on the rq since task_rq() may resolve differently. Signed-off-by: K Prateek Nayak --- XXX: Does this break EDF if we are blocking with 0-lag point still ahead of us and not set task_non_contending()? --- kernel/sched/deadline.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 9abda560c633..886ba6bfa868 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -471,7 +471,7 @@ static void task_contending(struct sched_dl_entity *dl_= se, int flags) * If this is a non-deadline task that has been boosted, * do nothing */ - if (dl_se->dl_runtime =3D=3D 0) + if (dl_se->dl_runtime =3D=3D 0 || (flags & ENQUEUE_MIGRATING)) return; =20 if (flags & ENQUEUE_MIGRATED) @@ -2477,8 +2477,13 @@ static void dequeue_dl_entity(struct sched_dl_entity= *dl_se, int flags) * way, because from GRUB's point of view the same thing is happening * (the task moves from "active contending" to "active non contending" * or "inactive") + * + * XXX: Proxy execution can block task with DEQUEUE_MIGRATING and + * switch the task_cpu() underneath without going through + * migrate_task_rq_dl(). Do not leave any bandwidth or timer reference + * back when DEQUEUE_SLEEP is coupled with DEQUEUE_MIGRATING */ - if (flags & DEQUEUE_SLEEP) + if ((flags & (DEQUEUE_SLEEP|DEQUEUE_MIGRATING)) =3D=3D DEQUEUE_SLEEP) task_non_contending(dl_se, true); } =20 --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012056.outbound.protection.outlook.com [52.101.48.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58F3F3B0ADB for ; Wed, 26 Aug 2026 06:32:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.56 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725969; cv=fail; b=PKrhTcDU9IXvGjtdXJ8WCmAM61zQU7wrGpHW9lrd52rYfw0QZ8WkJy+EQBm1Viegq0k0IJ1ow5SdvfXQX7bE/Dv0jL5GUVUOM+LzTJ8gUUzGcLBCqg6RN4WlnzZWPD9x6SMRnLm1snnNDWEZa+0/rJBTHkeKYBpV8gxKLLRK0+I= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725969; c=relaxed/simple; bh=LsLmrIRszRIUzjH5DAvm3gWbgiWLs5UqzAMQxLqnt70=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=IQF0mUBKX+jdABM9k5iJO99hvRc6kBnF+KBwP3wtAtA82/rXna5pSk2D5LAxNpcE437jkxKO/3QE/Dn0qW8ilZj0gsegVfxK0P//vu2XXDBU3pfSWnluiQtyN9fR+gLIi+hJoLgfEyqwHIziSXDpGHuOhxpDzqoJS7WzcGA4d5g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=G/KCgWS3; arc=fail smtp.client-ip=52.101.48.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="G/KCgWS3" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=WE9px8nFEwMwaX0rNqnV8aq/kLU6yrhupWdh/1GVw5YNL6IiTuG3pVnEBPbWhZY9KnsFDE69ci6NDhe3qhA4nAJ2iX3zk7Qc/6vIePsbOjlnGFNnI+bk6XupfP+eOoFXzhG7UzdaWU5aQa2qNnIkCVS0fzzmGjZf3GdC68tW8L5LLFElDBldBn0SWmPFK0Y15P+5fpXshM/2XWNbPDsmiA+V1+HNX1NWqzSjAI1me2/yNIxpVS+yqG0ctHOJ9//aX3nw7TljbMmexZcl0iEqhKnKUupwo9Ds12vJeEcMFzEtnQqxknlPAX/f/jOJGI+7SZkfFtKL2H437a5HjPLzvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P0yMohpyRdd0l6l1x+hbhb4zJ0nWOSacRiCkL0tnmUY=; b=NmjzzKAfNVjuTabNL0I5Vw9LyXejce2hRqiTEM/GUd4iuVJaaP+2OOBRnduZZQnfxQaiooTCpJA6+sS4Vb4fA9yKhCbrUToxP/qfdss3XjS+CIwM5MAWPUgx4vRNKbz30mGiW+fhPkGxwonRcPIVkvOBMRncMgMNBTQqJOu1GZKUI6y3p3+7pvkI9QaoNBe2bTnytn0TWwiTY+YpmDHDrdyWHqh79lYsH+dF2omqnOd8WCwklfsH1tubmVZjXWJ8CrWMzs90ZBsdiG33EHB37krGPTFgPLa9xM5i+8ygZd5HK/kDC03Zd4Rsl5cfYEMM9jnguqYPIpheb+bcJc16DQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P0yMohpyRdd0l6l1x+hbhb4zJ0nWOSacRiCkL0tnmUY=; b=G/KCgWS3sYDlrUaENH0psXo8vjFQdlKdwJPijUQ04s9li3cJipeiqnPwfpgo2bHo0Y6xtMpjDMGCavt6BtUFzHrrE+Bb07VFZ7J/U6AWY5AhXIXtJtHpEm8we12fWtli5EvIfKuc1bqNefU9UpYAPRvX7Srbcnf+oiVldcYd79I= Received: from DS1PR05CA0026.namprd05.prod.outlook.com (2603:10b6:8:23f::8) by MN2PR12MB4159.namprd12.prod.outlook.com (2603:10b6:208:1da::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.7; Wed, 26 Aug 2026 06:32:44 +0000 Received: from CY4PEPF0000E9D9.namprd05.prod.outlook.com (2603:10b6:8:23f:cafe::7c) by DS1PR05CA0026.outlook.office365.com (2603:10b6:8:23f::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.5 via Frontend Transport; Wed, 26 Aug 2026 06:32:44 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D9.mail.protection.outlook.com (10.167.241.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:32:44 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:32:38 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 09/16] sched/core: Track CPU where the task was blocked on Date: Wed, 26 Aug 2026 06:28:53 +0000 Message-ID: <20260826062901.2137-10-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D9:EE_|MN2PR12MB4159:EE_ X-MS-Office365-Filtering-Correlation-Id: 1f786194-2edf-426e-cd86-08df033bd62f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|376014|7416014|1800799024|23010399003|82310400026|921020|10067099003|56012099006|11063799006|5023799004|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: DtRK5GagehO2tjZ5S24r08G6N42QQMthhJJerfV/ExnRT7MyGs5kZfLxyHUHRG5NN3JL6GTynWjGrN6cHBdOlUYjbGe5tY+dw2C1XdtSaGIJG4FY6vibgCnUWiprN9PGopdUDnXY5rRWx3oxaWGLV4aMQQ/d6RFBdsS94QuNfsMMw7L933JVyouaYnSpHhT42Bl7qbOq0J+KlYRukXH+ocjL70VqWuEJXRiG7ivOUqs7gxYd3YuUh3pgAn1L6Ii0Djlso/QJ44bP3ImS5pibO6cq5SsvESTYg3dXFb+GVp6svSZfP02B4FQqXaxO0s4xf3Tx/+s7Cj2hocKfLU0ioty08+aXZvC/5Z6KXWvKsKeop2Ol8vg18PA86Fi4pjwwjY44ultOADtonUC0NeDcgvWwyJG0Huvyj/n0bRA4nOo+fZLUOv2TyYVhMfmVjESLq+pyseF+NfkqqeWFlIGamQCensPmIKjhANApzSWkLH1bc65g8gK9/GlGzU6dXN5k6UytZmb79crbHSNFVQ+S3KAe4ZTQKN1/g4F3Is+jg8Ag+BIqJ2hjrQormAj6rpNX8C3S8zThCrcKVbpbEmjdrzBYVn9gsmM1R+I3qxKt2KkIv8Dc9CWuvEP3DagVbYpNQJadi2C6kAuvEtITkjwvw2drVOJfHuynpC0pwHDH5b8538UDG73/Eb+S/H7pi9WbFChq9TJTlP0h6Ow+LQYQKhb+dghdl733sD/pjtX4VGsj5RUnggTHxX8Kz7+nOud5 X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(36860700016)(376014)(7416014)(1800799024)(23010399003)(82310400026)(921020)(10067099003)(56012099006)(11063799006)(5023799004)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: w/2PRdsv/G6J5+FLNG7qHCGs0MfIbRY0zUB6NoxIc8WHX4Z1iQ98bhHo4jmbY50l27ThUOvMjXqNZUYoT44ztMvga+D+tKijYpFTZI/N91VrX5AWLg/J1aAHuPJepZ0CBCJCh5nNdDRHYNcg8BOPjQMXDj6ENiONHwSYW8Dgqjs8XrBwY3p4sq/V/viRAUr0fL/9SasKib+EEgaTHGaTS9fA5VZlPo8Lcug5Yn7ccAmpBtgn1hvqIJzXaHqK/htk5J5KlWymJ8zS5f1ykp90lPWIWKkioPKnGrxn6johe8be7m07qgfx7n+lTEzZEASdGO5US37TB8ceLne/d2p6KTgTe/xrCKno9FsHBMe2c51HOFl9uWgKMq+G+TJN4CPHO48iNU4mHuv8aaCNcWf4q5QRii4uUd00ITF0p4CtDLTIbqBiIt/HPs7VrS1PtDdb X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:32:44.0427 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1f786194-2edf-426e-cd86-08df033bd62f X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D9.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4159 Content-Type: text/plain; charset="utf-8" Track the CPU where task was blocked on when running with sched_proxy_exec(). This is used to re-direct the activation of blocked donors queued on the blocked_head via the said CPU in the activation slowpath that will be added in the subsequent patches. p->wake_cpu or task_cpu() is not sufficient for this purpose since p->wake_cpu is not stable when !task_on_rq_queued() and there is a window between set_task_cpu() and activet_blocked_task() in the wakeup path that needs to be plugged in. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 1 + kernel/sched/core.c | 16 ++++++++++++++++ kernel/sched/sched.h | 6 ++++++ 3 files changed, 23 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index bf0f4b6be7c4..2360fdb1ce3f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1255,6 +1255,7 @@ struct task_struct { struct list_head blocked_head; /* tasks blocked on this task */ struct list_head blocked_node; /* our entry on someone elses blocked_he= ad */ struct task_struct *sleeping_owner; /* task our blocked_node is enqueued= on */ + int blocked_cpu; /* CPU where task was blocked. */ #endif =20 /* diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d635f5452a1b..b5cc5612a12c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3773,6 +3773,22 @@ static inline void proxy_reset_donor(struct rq *rq) resched_curr(rq); } =20 +void __proxy_block_task(struct task_struct *p) +{ + if (!sched_proxy_exec()) + return; + + /* + * This is the CPU where the __active_blocked_donor() slowpath will + * resume activation of blocked donors queued on this task. + * + * This exists to cover the race between set_task_cpu() changing the + * task_cpu() before task is transitioned to TASK_ON_RQ_QUEUED in + * activate_blocked_task(). + */ + WRITE_ONCE(p->blocked_cpu, task_cpu(p)); +} + /* * Checks to see if task p has been proxy-migrated to another rq * and needs to be returned. If so, we deactivate the task here diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56d9c09c485e..0d90506c34f1 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1445,11 +1445,15 @@ static inline void rq_set_donor(struct rq *rq, stru= ct task_struct *t) { rcu_assign_pointer(rq->donor, t); } + +void __proxy_block_task(struct task_struct *p); #else static inline void rq_set_donor(struct rq *rq, struct task_struct *t) { /* Do nothing */ } + +static inline void __proxy_block_task(struct task_struct *p) {} #endif =20 #ifdef CONFIG_SCHED_CORE @@ -3039,6 +3043,8 @@ static inline void __block_task(struct rq *rq, struct= task_struct *p) delayacct_blkio_start(); } =20 + __proxy_block_task(p); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); =20 /* --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11012034.outbound.protection.outlook.com [52.101.53.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 251AD3B8948 for ; Wed, 26 Aug 2026 06:33:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.34 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725989; cv=fail; b=NBoOmgNrahS+PPDi3ABY2qNcaRjqUQvug2uyjKoVzUtcp/OApwkgQ4vNdr/NoEEFSkDqBDW4UsR3ZIgWAInK3baaTXGhsAKSJ8cOHULiHOfvclf1CTWr4pBi+fTl5mq48BJjHR+r2BVfpGakn8giTaEKTvsYSKP5q/xBlfxBYLo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787725989; c=relaxed/simple; bh=dfMtUCZgy9Ut3Q6GpU8ipnZzzN7Y9aSW0ErC4VDHMM8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nVyjfaurMes63MUng1MbRTtW7P599kaZqdUxBoH9COEUnwKR11H+91kQ1tTbrzx0yTJk+jOO4tyI6YpcXTbvA6cE0A0oV3KGBS6BbDh/5Y4WhRwlB2+ZPhIzzWa6AuZOEvVks+3Mt54zDR4V2vSvSM+WFNFW320PW3JWq17NYBk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=kMnMmKz2; arc=fail smtp.client-ip=52.101.53.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="kMnMmKz2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DawxolCXet1hpm5nsdWDvb8CZDz3dFikbYrC4RUZEg3EPO86YbeGJ9NNYBlYm/ME1xmi1BQF1NBuVabmX7ovypEP3qEKa2i0BwHyJ33AlH53vZJKhkI9YUX5HwyDOxKhG/bbstri1TSaBsTDVADJKLi7p00m3SpQwiGaopq0FH4Ta32c3J6yaEgIKMf+kUOBrX/hrkuCthCxymCIMly/3nSDXYj2/OuOeua47wMdxJuT/1+KZolBtn1pmC8/QLVEV/85wKnP1UGiW+tcowTjVP3Teq8mx2Ydl4n3ucT7EEDk5wCoQ66rnkf0heI4tzoc40f72mp/Bc2j5XfxTdsiQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6lPoVGGWxORbBW2i87mlfX3qeAtBiFKybHeIKxpiiYE=; b=fkbuQjJ0VdJroahNo1za4/7befK7p32nqu42d+dQO84iqo+SEfgi0c/HcOoGO/nTc1IAUyKMhzT1L+il6rCuYZa+M3ivK62JQpbRTdJYrQAyWsJ/HTge/cxRMc//LOrij2JaioGCCf59G3oymfEtFINHZbVBRc58hqwr/fRi07Sc3zowabw9W4sBHTbr0JalpH2JdCRzjKWn5a9DhI8TYNb7Zt+j8X+Kj6T+wd7Sw37jggaqa+aRFXn8hTKsWT3c4BVC8JCVfq7SVVgZNkNXCs2KzpaiXBBnSzZB/emoswQNzwpph11lWKALGe1/YhjYxzwMhLUnkA2HjguZW7+S+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6lPoVGGWxORbBW2i87mlfX3qeAtBiFKybHeIKxpiiYE=; b=kMnMmKz2X8GRPBjglsHwU73pe6OT4pRjJvAUxAFa+J8N8U0pTZlgOKeTLisB6hCeutAnbmANumDwWZF3zZcNafkpZYR+hZe/Y4nAoNV8fzyuACsDiIjqSGzP9QjqoMT0i2nF23gWuQWRZYk3+y65xlV7X3yfqdfA90cNbh3Tfj0= Received: from DS1PR05CA0026.namprd05.prod.outlook.com (2603:10b6:8:23f::8) by SAVPR12MB999121.namprd12.prod.outlook.com (2603:10b6:806:4e7::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.6; Wed, 26 Aug 2026 06:33:02 +0000 Received: from CY4PEPF0000E9D9.namprd05.prod.outlook.com (2603:10b6:8:23f:cafe::a4) by DS1PR05CA0026.outlook.office365.com (2603:10b6:8:23f::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.5 via Frontend Transport; Wed, 26 Aug 2026 06:33:02 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D9.mail.protection.outlook.com (10.167.241.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:33:02 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:32:56 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 10/16] sched/core: Introduce p->is_linked to track if task is queued on sleeping owner Date: Wed, 26 Aug 2026 06:28:54 +0000 Message-ID: <20260826062901.2137-11-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D9:EE_|SAVPR12MB999121:EE_ X-MS-Office365-Filtering-Correlation-Id: b09b6e3d-78d6-471b-6a6b-08df033be0eb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700016|376014|82310400026|7416014|23010399003|10067099003|56012099006|11063799006|3023799007|18002099003|22082099003|921020; X-Microsoft-Antispam-Message-Info: 7uhZeUk7G2omVtWepr1HQpy7K7NcKOrEpv0MufWnG2zsvjZIO9gNykTx42Sj7HjDQxykqAbqEdqilSb/FBMk56uP4qS5kqW4W2hyn10wi/bfBLPXy4Nxefoc/Ozc8TD6t4NciJ3nltUWXiSeRFDk16uRkpXwJVEj/VbWwG0EmmfIwJLs/cpnGo1LkCdOQ63BdnJHqNUOtpdmyFnyEdzFt+Enc9UqRjhca4hR6ucaZRiWr8G52VFh2ZxSKzKkD8BdJn2kWxgN9f69oAUq+Q12GzFQin05tuDwBlFfMHC4+BDphcwTwwmPDB0DsqiXbW+he/i0GY2bzTBMsaQgB+cpOekMmv5Hnh/1dNdzR6WN5pRyhpUkb0s27u5CjQXlD2S6Tbh94feHWsIQSbW4QV1vGIJb/rN6KxgjRzQSRWF+C8HDXUq5qFY4OAMI7L4bDWppwLkiBuUyLVTSdNwD6PnMaVjiN5vsvQeBERoepYnEOHhgm/vr19oxOHp/SbAFF3+14xS+Rloa0pSDCjbG58tE0lSe7fNTZyKu0iSVzcZpAqEZv6e2lzo2hplZPcQ545M6IJ5OoKBB0ysjsjTDUedwEkL/uXDWaJ5+nrMGx4XC6wrggUiOXuZ/tYtg5nqMhIVSdyAyGc7eZZS57zZTgBGUvdlgek1nY6exqVdMeLdxorFuIUbqBJe95Q5Z3z6lCqPLL4Ov3ZseSAmi21R1zo+5/E3v5o9E/HrRPRoczeePZ68Uv+Uo1SYn2tpgvTS2//+O X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700016)(376014)(82310400026)(7416014)(23010399003)(10067099003)(56012099006)(11063799006)(3023799007)(18002099003)(22082099003)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: i2wXGfRKnj/2gpGZ6vUq3A/UiO0/wenb+nl0TWHu7ZNEicO52V5qYI18xTlfjjuyDJ07cBRbFF+L8JKEWpPr7Bu0rEUK9Voier7SA/8nKgtLzUHKfUSFwYxby5/ovybvPmhqgwBS2TSg5/t4MZC5ROZNUKMdBlE49Tq7Pal7xYWjdpOSlhpoC6ehegtvZAcsryd8HuNPiQUItlwmPyAiE6Yfr1QxK0o1660f9HQ2rI0H8C63DBoB8dAjz3YQSyNHOwVzGYMgWFwQJl0Fd6zWson/X3ek78pbvcc7LWS3iooxVDUmMz4B3EwbzMP9e2zvfNrYYO//mShdXROhWBdKrCVsplZkl9PSUrYV/uPkBck94mKAFF2cC/fD1M+b4Aa6pdyU7WxNsXnMMaiX1KSXx9ARav5YIXYH2dZ4v9hCoM/SWcM1vXevbp9hMCsUhhTO X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:33:02.0475 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b09b6e3d-78d6-471b-6a6b-08df033be0eb X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D9.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SAVPR12MB999121 Content-Type: text/plain; charset="utf-8" Add a new p->is_linked state that indicates if the task, currently acting as a blocked donor, is queued on a sleeping owner. This will be used in the subsequent patches to influence ttwu path in addition to ->on_rq state for linked proxy donors to consolidate their removal from owner's blocked_head in a single location. Clearing of p->is_linked will be added in subsequent patches when ttwu and activate_blocked_task() paths are plugged to work with the new scheme. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 2 +- kernel/sched/core.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 2360fdb1ce3f..ec65abf090a4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -854,8 +854,8 @@ struct task_struct { =20 u8 on_cpu; u8 on_rq; + u8 is_linked; u8 is_blocked; - u8 __pad; =20 struct __call_single_node wake_entry; unsigned int wakee_flips; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b5cc5612a12c..d3cdfcb400d1 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -630,6 +630,18 @@ int task_llc(const struct task_struct *p) * if the task is blocked. Traditionally this would mirror p->on_rq, how= ever * due things like DELAY_DEQUEUE and PROXY_EXEC, this can diverge. * + * p->is_linked <- { 0, 1 }: + * + * is set by proxy_enqueue_on_onwer() and cleared by either + * proxy_needs_return() or proxy_dequeue_from_owner() and tracks if task= is + * queued on a sleeping owner's blocked_head. Although this could have b= een + * a p->on_rq state, it really represents the state of task when it is o= ff + * the rq and is tracked separately for convenience. + * + * is_linked is always modified under __task_rq_lock() or when + * task_on_rq_migrating() during chain-wakeup where concurrent + * __task_rq_lock() is stalled for the task. + * * task_cpu(p): is changed by set_task_cpu(), the rules are: * * - Don't call set_task_cpu() on a blocked task: @@ -2235,6 +2247,7 @@ static void __proxy_dequeue_from_owner(struct task_st= ruct *p) { list_del_init(&p->blocked_node); WRITE_ONCE(p->sleeping_owner, NULL); + WRITE_ONCE(p->is_linked, 0); } =20 #endif /* CONFIG_SCHED_PROXY_EXEC */ @@ -4307,6 +4320,7 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) */ WARN_ON_ONCE(p->se.sched_delayed); WARN_ON_ONCE(p->is_blocked); + WARN_ON_ONCE(p->is_linked); /* If p is current, we know we can run here, so clear blocked_on */ clear_task_blocked_on(p, NULL); if (!ttwu_state_match(p, state, &success)) @@ -4611,6 +4625,7 @@ static void __sched_fork(u64 clone_flags, struct task= _struct *p) /* A delayed task cannot be in clone(). */ WARN_ON_ONCE(p->se.sched_delayed); WARN_ON_ONCE(p->is_blocked); + WARN_ON_ONCE(p->is_linked); =20 #ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq =3D NULL; @@ -6909,6 +6924,7 @@ static void proxy_enqueue_on_owner(struct rq *rq, str= uct task_struct *owner, =20 WRITE_ONCE(p->sleeping_owner, owner); list_add(&p->blocked_node, &owner->blocked_head); + WRITE_ONCE(p->is_linked, 1); proxy_resched_idle(rq); =20 /* --=20 2.34.1 From nobody Mon Sep 28 05:45:29 2026 Received: from CH1PR05CU001.outbound.protection.outlook.com (mail-northcentralusazon11010014.outbound.protection.outlook.com [52.101.193.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B0B52E736F for ; Wed, 26 Aug 2026 06:33:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.193.14 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726006; cv=fail; b=M0FDgif48xxnqRXxBfISSwqv3zedJ/JyZTM0mji1fzBmIYgX+mjg9gaITZPjrsPHP7PAwSlJtWuGTvjQ+28Dltsb1Sp3A+D8gYxbgu6OJu1i+sTEBReSKC62rJg3sTNKZxXZzq8GDDvU93wlsXjN+ZcPN1T3JiFwXeSwtohtJBM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726006; c=relaxed/simple; bh=ic3u5AlAYbG6zDX6bQb3Ud6gvHT2cLi+mM/1nqNTmRk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fXZaC9FbwBKKkHiUTl/Y9ITDVzuMjbcb1ZiEz1YQRnjQsow61c8AcAAGcqX4lxME6oTzhmz+fROdHiPGT/KZIboxlrgAr2ibYNj9RmRuxlzetS1bk05sElSCOt635J3sWX2fxf7Fv3lo63e1z59jwn3yvjC+kPl2Ynk78ZzjQ5M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RoQ5rlsV; arc=fail smtp.client-ip=52.101.193.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RoQ5rlsV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rMG1JkaImVS7I503260HTQlPgy+pUPFXctM+AcTUw0jFASMNmWNEAaA1CpMT6xe+vm4JTn19xhh316NBjDFAhAQDkbukgI66s9V5w+89/q4D8VueZTwdIKPLwKmO7+ZXz1JuvmbGx1La74C8uBtK1T6kEG/w0Eua877v0Q7ac+JwhgZNWx6gdwMdnC9T+Q2MDkaUtbAEgRF1X9oz6dgglr1r7yyjtvR6KDWx8OGmpZhYF3Q2BL8t2ZeHMLq13rTXwYKmAhLZ2ZJeZIDIpUMsUZ0dw35v6bxFs1XYAa3k8CvWrqU89RjJt4Of2xtz+bZGxZMZN1/fGn/Y88Swo2O4hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9KiYYUj56CZcaGJ34n2xTLli85bT5S9KrcPISUv9tZo=; b=cjlmflauZgjPot2xCAwnXJPNBTZfUzROLtCMnGjg7OPSLOVyHsOymWjCAE12pQk1XsLEiwMERwiHdhByRkXsElqso+a5v5eE5iRDrBd3ShWIUkayeuHxa7IzARLK9k9V6r30PHOIuG9eOmVi7nvc/KMVmVLsAs9Jg5nc66F8ncuNNYyqmrBRsTM0DHYMNuyiVPZei6hnI3Zu/QoK1SCEbMNtrpFO+7J72DmhSahHmyz4AxIdRbzzrkS8yz1KzfG562nYX8xcMENPt4CBMnF5rBONDmyCAh7ggUPLoonpkKVG7xZ1zAryk59PHg282iJpXbytx/FxVvMUSV3+GDbplQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9KiYYUj56CZcaGJ34n2xTLli85bT5S9KrcPISUv9tZo=; b=RoQ5rlsVb5ymcRj5SVTLpAVs4oDe9D3GT32Yt7YD6pqslPgF5P8qs6poKwURKYjMQR54jGAabCJt29BWZcpI0Erg2XpQA82VwZT6tKNLqwTFWmEwVE8DB5y58VGs33SMX9/EplXXblFL6KjWklmjOxIQE4uk8cT/cvWChm2drvM= Received: from DM6PR05CA0058.namprd05.prod.outlook.com (2603:10b6:5:335::27) by BN7PPF9C6E5285F.namprd12.prod.outlook.com (2603:10b6:40f:fc02::6db) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.6; Wed, 26 Aug 2026 06:33:20 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:5:335:cafe::af) by DM6PR05CA0058.outlook.office365.com (2603:10b6:5:335::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.6 via Frontend Transport; Wed, 26 Aug 2026 06:33:20 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:33:20 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:33:15 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 11/16] sched/core: Prepare to inspect ->is_linked alongside ->on_rq during wakeup Date: Wed, 26 Aug 2026 06:28:55 +0000 Message-ID: <20260826062901.2137-12-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|BN7PPF9C6E5285F:EE_ X-MS-Office365-Filtering-Correlation-Id: a9b5c33c-8da4-447e-ca6f-08df033bebc2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|23010399003|82310400026|36860700016|376014|7416014|56012099006|10067099003|22082099003|18002099003|11063799006|921020; X-Microsoft-Antispam-Message-Info: z7zJH0uAbG4RjMfRvuDKmltAHHj9zYC0/Om97/I2wS0BEdoySYzgp14bRPhVzfCES+V1lbX9M+acJ+FvUFrLZ/yQ/inQNKeWrmu7sqZy525ByOOoImpS3Apffg/3FMgRuVhTeTUU7+7g2JemjCz45b7cHF3zxTYzeMJpJfu98TGoXaWvCgwRMs6ZeCuXpRHF9bsDPoznA19USdFmB6HAg7bAR5VPbilEn00IsBTLlv37dz1pyGcWVz88p52roxoK/xA13TLktdmmPwyf67skIDjACLqof33ErIi93rmKjAp0bKMlSJSPEOVDSvykyyO/3tGTcDwMfGDOen3fjR5hy+TzmSgbdC7JtDtphv3lh4iohTtP9INLE9icftXKm10GfgW0bv0BtV6dr64w+pGou2opAaZn5uTO0cBQLI6Oc0wX6UxsD6VhsnFMjAAOrM0IyH4lI1nnN9YzzKPWpFdUuJSjbchSSyjM9YkgQXaeezoVKDdtSzzkR9Gg+eysZ6xhhSAtZyb2ZdD6ZFnYr6gYnR1D33yQOXT2nV7ULG84OkPPlEsod12YlH1pWsKF//aqu1qdYKopbVZLkBBVVJbyxo0uUf8NlljvkITBrDM6YKa+ON8Th0sYF6G65PIzSJW3pJkyLzxp3Dcs/i1nt41XdMtPjCmH991DGpHBmuWL36MRmqryYLOiZxnXNaIRNspD4gdi0XuHQRqkRLOOuch/BSEBB+hkAwGHWC2Tjyv0Cx7/5vYfmiK6BScm3gs6ZA9b X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(1800799024)(23010399003)(82310400026)(36860700016)(376014)(7416014)(56012099006)(10067099003)(22082099003)(18002099003)(11063799006)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fbhNYbP27XQTEc1nyeb6xjA11h5kfbax6Nsi9izvVAvZZYJAQuJaiAMa3Gpu4lJ0ztOOGAm47udYd8mqlbQXSeACu+b6sv9e5xwjuexgdlirOCw8OywQ52MDd5fKTS849RhhHLZl9LvpWe4y3jokC8YEB2Lw382q+3gYcfUPVGkROapvRCXhNq2dfyroaxjolT1vaBeYf6bUifuChEq35nlmFkST8vljRHzN5GwDkJZf7bdMSAAIGCbpIhmyBQuyc01ZLfhpRBUg1QkvSuoTzMPqpK82dUqAXW+8TQhNHcC2I6n14y7t4EH0gx+hjuw77WnslqEJFUTFP+Vn8M8l42d06EGwoZoGnX8dRifcl3XF5beyGEqlY8YdtMXlcYlcsDqSbb1LYTVSBiH5fKkGWBINOIiI+eJ5CawX8llAchWFpIoKApQ/5qJKSYlsErJa X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:33:20.2375 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a9b5c33c-8da4-447e-ca6f-08df033bebc2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PPF9C6E5285F Content-Type: text/plain; charset="utf-8" To handle wakeup of blocked donors queued on owner's blocked_head via ttwu_runnable(), prepare to inspect p->is_blocked in addition to p->on_rq on paths that demand grabbing rq_lock to prevent "->on_rq" transitions. Unionize p->on_rq with p->is_linked to derive a 16-bit p->needs_rq_sync state that can be atomically inspected on all architectures. p->needs_rq_sync check follows the same ordering requirements as p->on_rq check. Subsequent tasks will guarntee ordering of p->is_linked against p->on_rq transitions ensuring that loading p->needs_rq_sync state is enough to cover both and the readers don't falsely observe them to be cleared if read individually during their transition. get_wchan() has been reflowed using guards as a pert of the change to prevent awkward indentation. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 18 +++++++++++++-- kernel/sched/core.c | 53 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 64 insertions(+), 7 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index ec65abf090a4..76de348fb88b 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -852,9 +852,23 @@ struct task_struct { struct alloc_tag *alloc_tag; #endif =20 + union { + struct { + u8 on_rq; + u8 is_linked; + }; + /* + * Allows inspecting on_rq and is_linked + * atomically with a single read outside + * the task_rq_lock(). + * + * See the comment above the load in + * try_to_wake_up() for ordering + * guarantees. + */ + u16 needs_rq_sync; + }; u8 on_cpu; - u8 on_rq; - u8 is_linked; u8 is_blocked; =20 struct __call_single_node wake_entry; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d3cdfcb400d1..957f30632b82 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2171,12 +2171,16 @@ unsigned long get_wchan(struct task_struct *p) return 0; =20 /* Only get wchan if task is blocked and we can keep it that way. */ - raw_spin_lock_irq(&p->pi_lock); + guard(raw_spinlock_irq)(&p->pi_lock); + state =3D READ_ONCE(p->__state); + if (state =3D=3D TASK_RUNNING || state =3D=3D TASK_WAKING) + return 0; + smp_rmb(); /* see try_to_wake_up() */ - if (state !=3D TASK_RUNNING && state !=3D TASK_WAKING && !p->on_rq) + + if (!READ_ONCE(p->needs_rq_sync)) ip =3D __get_wchan(p); - raw_spin_unlock_irq(&p->pi_lock); =20 return ip; } @@ -4365,10 +4369,49 @@ int try_to_wake_up(struct task_struct *p, unsigned = int state, int wake_flags) * Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in * __schedule(). See the comment for smp_mb__after_spinlock(). * + * Additionally, this also guards against activation of blocked + * donors queued on a sleeping owner when a wakeup races for + * the same task with sched_proxy_exec(). + * + * proxy_enqueue_on_owner() + * STORE p->is_linked =3D 1 + * + * block_task() try_to_wake_up() + * smp_mb() + * STRORE p->on_rq =3D 0 LOAD p->needs_rq_sync + * + * The read below will atomically observe either p->on_rq or + * p->is_linked being set and put the task on ttwu_runnable() + * path. + * + * On the chain-wakeup path, p->on_rq is first transitioned to + * TASK_ON_RQ_MIGRATING before p->is_linked is cleared and the + * task is transitioned to TASK_ON_RQ_QUEUED + * + * activate_blocked_task() + * task_rq_lock() ttwu_runnable() + * STORE donor->on_rq =3D MIGRATING task_rq_lock() + * # STALL + * smp_wmb(); + * + * proxy_dequeue_from_owner() + * STORE donor->is_linked =3D 0 + * + * activate_task() + * STORE donor->on_rq =3D QUEUED + * + * rq_unlock() # ACQUIRED + * LOAD donor->needs_rq_sync + * + * In this case too try_to_wake_up() will correctly observe + * either p->on_rq !=3D 0 or p->is_linked !=3D 0 by atomically + * reading p->needs_rq_sync when the task is being activated as + * a part of chain wakeup. + * * A similar smp_rmb() lives in __task_needs_rq_lock(). */ smp_rmb(); - if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) + if (READ_ONCE(p->needs_rq_sync) && ttwu_runnable(p, wake_flags)) break; =20 /* @@ -4488,7 +4531,7 @@ static bool __task_needs_rq_lock(struct task_struct *= p) * See try_to_wake_up() for a longer comment. */ smp_rmb(); - if (p->on_rq) + if (READ_ONCE(p->needs_rq_sync)) return true; =20 /* --=20 2.34.1 From nobody Mon Sep 28 05:45:30 2026 Received: from CH4PR04CU002.outbound.protection.outlook.com (mail-northcentralusazon11013014.outbound.protection.outlook.com [40.107.201.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8FDE26B2D2 for ; Wed, 26 Aug 2026 06:33:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.201.14 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726025; cv=fail; b=uN7gV2s9rA32fmj3C/CMqX4F7ErQ4pXQFoImMMBLHlEY//+p9ZLROwnw+w1ocRL6K5g2LIf6i/DLoyVeAuEVanjec3poZLX+GcSOj+HetJKKc5okMfgM+ZTVFgAL3O11Mgg7WEckCoxLLoEaHIWIMfRHX5UeszB/T37/2RZvq80= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726025; c=relaxed/simple; bh=CLl7M1+3mzgRssWVNB0j4ga+g8OroO44c6VTuJVR7aE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FG53QPjrWkW1s2hBvkrtHL+QBSOizbGB1Xr4XHzYMS9JbpFGo7aC4cazMEkFE5jnKx5JGDCTUjx7NaCk0X5mqbubGmL59P/TcIiujvGJKCI/MMmKP2AZVUNPQ3A+jPtn0axdUZrpdEJ+uoSdElcohufARXL42xVaVhDg+SEXlMI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=leDwPQCX; arc=fail smtp.client-ip=40.107.201.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="leDwPQCX" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=aeUGgbnW3TgB+P0uWVRyOuS+lVe+dIufAjo1YTRI+xoGB1IUBTdJIMFKlt+PFIMIzvUkaLUS7CNXZCtAoltbKQwBmbVQAipgQGmSzT+CwFQQ9C3TRH23zaCkpEs6Ik9AXQFS7KYQ64X93KzI+P4IKk0P3BBuF8rTdESIfUjHtyzVkH9dnBcU7GzPMIO2ej+/Lj9qDOz02tYXiEmPwaEn7msXTBNST2Rr+qqBcV4FCVMC62Yp9YIcGeLaX871IF/EbEk03EKGCpcpmI12KyxRbtR6Kkp7mztHeXQ+m60P+uX8c0bPzN2Jps9IQ/7xDl50U25S0SqAOSxPQaJ1bYVEbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fijKUT5GzTNnRK6qaG9IU3w6hOhCn48/0C8aBKc08lo=; b=bFDVBjgkNLVf375t0TbYeZzP9jsNsxEJCYKUXbwtwP25j7g3poIH8OdY+AWtNXD6Bwi2cmPB35jiwweksuwZH+K9F7YueZ/ANkPFPg6xis+yWfXx6lkjTjydIsh8dhvGQhe2aAPVjncwOaNOkz1fySeWvTWDGC7+eSVAe042paA2VvQFINAkPNPqOjSsN4EEfwFhhSmwNH9hfWGzYvCC96FIeP3abiWCxGSilZRPyNPGi2jSq8GoML7JCb7aL+aSq8aBZfJL2OMPFNIJ6BCvLvedrcrY1vlzTrMTHwEHgq1o7mA11uDYXZ8IMMBA/WCd+IAZiDOW8uWGzuHXfR4VDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fijKUT5GzTNnRK6qaG9IU3w6hOhCn48/0C8aBKc08lo=; b=leDwPQCXsj+VM+XKtS2Cuy8+SDritJK5GKzJBQ7esmB6yapv9/AZK60ZYDzMbuFJzaM4apTGshctswv9ULIvdnSZajYs0Dq87YrimzH4ZFhy1gQHO/ZjO+7qyuZoF4VkS6wU/vjHXKBPZ1LUMcu2R21Pz93xmXY93GRoEcfcC24= Received: from CY8PR10CA0005.namprd10.prod.outlook.com (2603:10b6:930:4f::22) by DS0PR12MB6605.namprd12.prod.outlook.com (2603:10b6:8:d3::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.8; Wed, 26 Aug 2026 06:33:38 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2603:10b6:930:4f:cafe::6d) by CY8PR10CA0005.outlook.office365.com (2603:10b6:930:4f::22) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.8 via Frontend Transport; Wed, 26 Aug 2026 06:33:38 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:33:38 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:33:33 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 12/16] sched:core: Add MIGRATING flags when blocking and activating linked donors Date: Wed, 26 Aug 2026 06:28:56 +0000 Message-ID: <20260826062901.2137-13-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|DS0PR12MB6605:EE_ X-MS-Office365-Filtering-Correlation-Id: 71ee60d9-2c5d-4bb9-4155-08df033bf68a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|23010399003|7416014|376014|36860700016|921020|6133799003|10067099003|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: 43l/kp54XFZMWm9Pbb546iyRUsceqSMpEMyFJlr+OJH/LkSPzsBmK0ap5kSmSW5O6eO3t/yiwGqFHRQM/CIGWkH4PXDvvfZMCMYawO21J4YE4wSDQB/BOci3+i1mlJ7tb/Z/YPI6XssQFUiYnckAGWLGQ6Hb4QPZKtTbdLBh1yjXh0YF/1Vzi2GbcHfUMLqb+TS9IMoAPgVyXCTCGoJ/MvZ8iqt8yLWG+nPrFzu8aDrhJOXWi5Iudu0W5ou4n9hf7Y2573DimjS3NigdMdyGcESvkGXey8ZfdKgLfp4+8ex7DScbg6RXyxB4g689fbJA+f7FYdWYfMlhRM+aywlEYsrBaHA2znOvifnsGRpH6xigz4z/lVYcx/X9g1Dz57hzc1KAahE1XOqKMvZAHBK6RYom5kwTHr16+zyoMKm5E8N+pJV9y4yyq7a0KWZh4pv/pJT2KGlc5t7wTDpBVFKuug375+TTH4Qg84Ox1YxxBxwihoR0OUP9OBwZAU1gDVg2OCw1Pq3wSts7+4WrMVl+mUdXoMWYQCl6haRGylvdpghbSChq981TEE0DPPEg+mJNtJvjuXCudH+i0Lrn7Rit1SpVkI2tpD35e0ZywCdkGyLZqQGzHYBk6yiy8hJQ/UYcezdB+iPt94dymvYcEzqFZA5CuJIeiGVXITIqlhFDVMyCy9tUkvSte5r+hpRXeh6JjSkZrTLthxrm6ZJEqebjZbirZTJAdhkTL+S3ytLcWnLlXo8ELxz2y+KJvFm50N2w X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(23010399003)(7416014)(376014)(36860700016)(921020)(6133799003)(10067099003)(56012099006)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: V7kbnFNQwS+r2O64HT8QmYR2OPbEtr4/m6hiv0n2047B/q5WGm60boEsKt6rwgo6ZgU29hxvUfd/1zYqetVMM5SbjwH6Jb0/0aOKcspyahbuSXYagWXwzWYjHZNbMRIlOOYbFQzv0SU6593Q+Sz10+raxfCHozQZIqexEsMeDUSeI9mXRsKmbaypu/ERMrc3fdQvhQzTCi9PKT7l46jSGgwpYXy/pDdEQsGmjU9jSq+LksMlaIEniVrD2l8JVRu2tvE1Q+N4PpWlAoc0bAUU2O1wfd1lX9o4resgWLTL5DJLWg3f5uDrbw8LkgEU/KWbMm6cryh48rMMzMIwqHi0JS2laa/lMXvPNS6eQcGdpZRfs4ucJp3E38LkgnhGaxLFVU+Mq3xWC2tgrXl/v5ipCJBu1m3PFxfvdlRhnKCtQhGVyV7F8/DWoXXACKIBVAUR X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:33:38.3194 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 71ee60d9-2c5d-4bb9-4155-08df033bf68a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6605 Content-Type: text/plain; charset="utf-8" Linked donor can go through a proxy_set_task_cpu() on the way to blocking. Add DEQUEUE_MIGRATING with DEQUEUE_SLEEP when blocking proxy donors and add ENQUEUE_MIGRATING on the activation path if the task woke up for reason other than proxy chain-wakeup. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 1 + kernel/sched/core.c | 9 ++++++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 76de348fb88b..a05240b52806 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1003,6 +1003,7 @@ struct task_struct { unsigned sched_contributes_to_load:1; unsigned sched_migrated:1; unsigned sched_task_hot:1; + unsigned sched_migrated_on_blocking:1; =20 /* Force alignment to the next boundary: */ unsigned :0; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 957f30632b82..ff499d4532ad 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2285,6 +2285,10 @@ static void block_task(struct rq *rq, struct task_st= ruct *p, unsigned long task_ !(task_state & TASK_NOLOAD) && !(task_state & TASK_FROZEN); =20 + p->sched_migrated_on_blocking =3D !!p->is_linked; + if (p->is_linked) + flags |=3D DEQUEUE_MIGRATING; + if (unlikely(is_special_task_state(task_state))) flags |=3D DEQUEUE_SPECIAL; =20 @@ -3415,7 +3419,7 @@ void set_task_cpu(struct task_struct *p, unsigned int= new_cpu) trace_sched_migrate_task(p, new_cpu); =20 if (task_cpu(p) !=3D new_cpu) { - if (p->sched_class->migrate_task_rq) + if (!p->sched_migrated_on_blocking && p->sched_class->migrate_task_rq) p->sched_class->migrate_task_rq(p, new_cpu); p->se.nr_migrations++; perf_event_task_migrate(p); @@ -3863,6 +3867,9 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p= , int wake_flags, if (p->sched_contributes_to_load) rq->nr_uninterruptible--; =20 + if (p->sched_migrated_on_blocking) + en_flags |=3D ENQUEUE_MIGRATING; + if (wake_flags & WF_RQ_SELECTED) en_flags |=3D ENQUEUE_RQ_SELECTED; if (wake_flags & WF_MIGRATED) --=20 2.34.1 From nobody Mon Sep 28 05:45:30 2026 Received: from CY3PR05CU001.outbound.protection.outlook.com (mail-westcentralusazon11013067.outbound.protection.outlook.com [40.93.201.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2009426B2D2 for ; Wed, 26 Aug 2026 06:34:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.201.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726046; cv=fail; b=AbsDmI+PQFXQb5WsICECjshufp8TiUJlaqueh9FEWOJWUxM2kIm72rSKJ3okQ70nt2tKLiF1msuMDkc0lJGnrhOK9N66OuO1uND0LG603lRhcluo9OR7KzRy591l4/cB3uXse+HCEtJ3nRVDKJqQx+x/Qr692AeRqgC9okPWtig= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726046; c=relaxed/simple; bh=YzZhWuxtveg3ZNEOx5BRhRHfdNturIs9qCnulljZzf0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FNriCf6KAf7fyumCTAckOfEa0FOV0MitQI3ybePJ9p1bk39zbBt1Rw1Em9HbYQEx9DrzCOW7BIF2jWW7+S8jwoQ3TuXhc3oFKurGB5a438RJsQ/zPlltONTaVTnzaTHsGs0+TpImmvfzyAEQwVavVHvSOxg5/ypNq7tGscokCHc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QeQ8tp0C; arc=fail smtp.client-ip=40.93.201.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QeQ8tp0C" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lPedsPMFcxCMWFwF7MVHBcDXlE5GMDQVev3AFrfMXbyDbK3TGOBWs53rANOaj3i78E110zNVHdUlOx6ITZFR8R6YZsX35OTpVja6uVzcX9yBvq7NOkfUZuns9/daoqebZ2MG1DV7XbcHs5ob2EPd6DTVNQ6H8Tm5fFcdVlG1OKYYranEBdT+foOKjTFvLGIob+VC9RHkdJY8r+pkZZOLlUIjOdojmEJLxUSl+TbyA1d1MzRYYXvQiNWuIOgmwNcMhrGUzOkwmYGwDgiN2CfW4gnrX/WIaGIkuG6PT+1srxc47rwNmQwOPcf84n6/Jrr1ScwrgvuMcytLxE9BB1WRhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zQV08FaGagoR7EY/zZz9C9jlCjK4pXuei/Fxt1upO+M=; b=G/RErbQ4BwzvZbpM1fqKK7j4QzDE+qFX2aDvS4w/eI0cAXQGY2UatvseRmA+QJAjLNYHRq7VCkx0O2Ve1EGU1tYFriFAGqjbo5dn3FG1kyWWjsYfbxfCRVbq2o/w3gvtXkmpflFAEVGlOlk/PE6O6+cBy0GGZoIsp/cmo527Zrcbi0aEr+mLMqmE45d0Urv/TTDUaVquCx2kKaFKOjNz3MjblVrTaNfYICbf4L2SrvrnwwBJE4HfS/3X+957Q0AgiCtUN6dXOy4s+3I9iuDkFUvprQvVgl7NDU+ShfY8b8R1fewqDaGXU8u2CW99Y6SrfWqA9ABVhyQDds56NPrFyw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zQV08FaGagoR7EY/zZz9C9jlCjK4pXuei/Fxt1upO+M=; b=QeQ8tp0CXUv6+inE0DU2TOYiXnqohgjJJH0w5X/c2ucbYfJvJ66luqIcr4GD0KqKWTuz3vvwAfN8omLUJ3ytCJXN1gZIA1l5uPexJuuKSWq4TDAoTNSprYBVvz8JcdouL9yG2f+Su8lWTPCGQCktLlcOnGusqCGySXNH4mPPqYs= Received: from DM6PR05CA0052.namprd05.prod.outlook.com (2603:10b6:5:335::21) by BY5PR12MB4052.namprd12.prod.outlook.com (2603:10b6:a03:209::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:33:57 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:5:335:cafe::16) by DM6PR05CA0052.outlook.office365.com (2603:10b6:5:335::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.7 via Frontend Transport; Wed, 26 Aug 2026 06:33:56 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:33:56 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:33:51 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 13/16] sched/core: Use p->is_linked state to unlink from sleeping owner early Date: Wed, 26 Aug 2026 06:28:57 +0000 Message-ID: <20260826062901.2137-14-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|BY5PR12MB4052:EE_ X-MS-Office365-Filtering-Correlation-Id: bbc2172a-a63a-4142-56be-08df033c014b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700016|1800799024|23010399003|921020|10067099003|11063799006|56012099006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: 1uJMeSyrpNazLKrZAgF4BbxAvEfv3j4rpVtUkv3vdegHRjXKXApn8JlCE363e/oTOGLG+baqpSP3kQ+1W5mjz6H+qUhTE4GQmblEb6IDyIZ8qwmHfm1YBZ7Sc+4s89QPvAlrjBdFaUs90RlJf/ibZx9iI7HfTg9rGqCrhM8C4Sm7nQmTfdqgnCxOUfbVbOTjPHZMzfPIoMBiapR4n02HcoPhJSafiWJ8JpX0lTU6qvhDKAtgsE6fKcL4G9gZG89r+1gam3PwHR88agxS1EjrEr/rNjPnRS2IimvhLXhx4jbX1pRUvfxIIq8CNtIT54AL1QPDy+uSzHcrSCi61Z8QYh4mbI5Or7TwoJF4+BjTk28hDSmVmEJ2ZDxGd8OjxzUdfNmwb0Jg6kSJPbJildoqQuJ05ilKjyEFQVUvl/a4wYEodGIa2pkq3l/JvGEj8L/2pc7RYKzsXlBmLzqUh9AluN+H3EaO+4DgqYiEkD/gt3URIB0ACEiuB1P/xUGOIFNwdUriL5ajl6gAiNM50JkfJbsHD/w0cInvT9GC7WizLNESex25DVKfiRNo0Mu7x3Sy4zzlv9++yQFnNY4P5z5JSX34Inaazw207ZCUtMC47dGbizPA8EeQiMduzZC9L+OlLMFeQJi8ceBR5WIWUvYAyg+PBNLHjG269+Aw4IpTsONfa12ehtQ28BAjKl/ry5qwPABTgH+YQ4c9FUQecxRi+j9JsHszZgiUV7eXBTmaFWlAeiMjHM9tm2dZRTjbGZIK X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700016)(1800799024)(23010399003)(921020)(10067099003)(11063799006)(56012099006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sowFeQTswjtzEmL7N9DC2FkkY44RyRtixQGYt+XWSVHNgVzD+TWH+vr8LHTn3wnjm3GUdsHo3mQU+62xGJaWZfOzNhsRJWLc9B9azDcQkfUsvZUn/Va7JCmVgkyGfyoxY5yoo3einiKtyVOQd96DNqSJp7iDnOxy7JFzIy5uQ8lOfVZ2g/Tfe6eFLqH/goJGkkKYChNnILSzQbcyLggbtWdyQDdGMGmu1gFZY7y+WCOzES9MMexKAdGWE8wn9SNRgKO1ip2CfEM+vbhGrbkApca0nTHIp+pXBbfKwQojt10BwR5IxR6cgYG62vN08Ptx1VvNJ7zfOxP7sbnEnmC0lJlh19JhkIyqr/bdt/ms/dAA60G1bu2Q/w/1HITrvjgLu6DP5bHhT7b6Jcj5vbd+LH40J2huQnAlu6juyho0jkBg6MfC1QiNvDRNvt9d0I3v X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:33:56.3638 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bbc2172a-a63a-4142-56be-08df033c014b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4052 Content-Type: text/plain; charset="utf-8" Use p->is_linked indicator to unlink the task from a sleeping owner's blocked node early and continue to proxy_needs_return() on the wakeup path. proxy_needs_return() is now the single point of proxy donor handling on the wakeup path. The cases that handle enqueue of owner is as follows: if (!owner->on_rq) { guard(raw_spinlock)(&owner->blocked_lock) activate_task(owner) enqueue_on_owner() owner->on_rq =3D QUEUED / MIGRATING smp_mb() smp_mb() if (owner->on_rq) /* Raced */ if (!list_empty(owner->blocked_head)) dequeue_from_owner() guard(raw_spinlock)(&owner->blocked_lock) activate_blocked_donor() Case that handles concurrent activation of blocked_donor is as follows: try_to_wake_up(p) guard(raw_spinlock)(&p->pi_lock) /* State matches */ if (p->on_rq || p->is_linked) activate_blocked_donor(owner) ttwu_runnable() if (!list_empty(owner->blocked_head) task_rq_lock(p) rq_lock(cpu_rq(owner->blocked_cpu)) /* Same as task_rq(p) */ Either activate_blocked_donor() wins and activates task or ttwu_runnable() wins and completes the wakeup after dequeuing from sleeping owner. activate_blocked_donor() changes are added in the subsequent patch. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 176 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 174 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ff499d4532ad..e6abcea97683 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2252,6 +2252,96 @@ static void __proxy_dequeue_from_owner(struct task_s= truct *p) list_del_init(&p->blocked_node); WRITE_ONCE(p->sleeping_owner, NULL); WRITE_ONCE(p->is_linked, 0); + /* + * p->blocked_cpu is intentionally left un-altered + * since the wakeup path requires it to grab the + * correct rq for chain wakeup if we are here from + * the activation path. + * + * "p->blocked_cpu" only has significance for blocked + * task on "blocked_head" and a stale value in the + * field is harmless. + */ +} + +static void proxy_dequeue_from_owner(struct task_struct *p) +{ + struct task_struct *owner =3D p->sleeping_owner; + + scoped_guard(raw_spinlock, &owner->blocked_lock) { + WARN_ON_ONCE(p->sleeping_owner !=3D owner); + __proxy_dequeue_from_owner(p); + } +} + +static bool proxy_try_dequeue_from_owner(struct task_struct *p) +{ + struct task_struct *owner =3D READ_ONCE(p->sleeping_owner); + + /* + * Task is outside __task_rq_lock(). These fields + * can race with a chain-wakeup. + */ + if (!owner || READ_ONCE(p->on_rq)) + return false; + + /* + * The chain-wakeup slowpath grabs both the rq_lock() and the + * owner->blocked_lock while removing the tasks on + * owner->blocked_head. + * + * If there is no concurrent chain-wakeup for this task, it is + * safe to just hold the owner->blocked_lock which serializes + * the removal like: + * + * proxy_try_dequeue_from_owner(donor) proxy_activate_blocked_donor(owner) + * LOCK rq_lock(p->blocked_cpu) + * LOCK &owner->blocked_lock LOCK &owner->blocked_lock + * + * # Defensive checks STORE donor->on_rq =3D MIGRATING + * __proxy_dequeue_from_owner() smp_mb() + * + * UNLOCK &owner->blocked_lock UNLOCK &owner->blocked_lock + * + * # If proxy_try_dequeue fails + * __task_rq_lock(donor) __proxy_dequeue_from_owner(donor) + * if (donor->on_rq =3D=3D MIGRATING) __activate_task(donor) + * cpu_relax() STORE p->on_rq =3D 1 + * + * # Sees consistent state. + * + * Either proxy_activate_blocked_donor() manages to dequeue the donor + * and stall the __task_rq_lock() until the task is actually stable + * or proxy_try_dequeue_from_owner() wins and manages to remove the + * task from "owner->blocked_head" first. + */ + scoped_guard(raw_spinlock, &owner->blocked_lock) { + /* + * Chain-wakeup has raced to queue the task. + * Try again with __task_rq_lock() held. + */ + if (READ_ONCE(p->on_rq)) + return false; + + /* + * Something changes in the link. Try again with + * __task_rq_lock() held via proxy_needs_return() + * where both p->on_rq and p->is_linked is stable. + */ + if (!READ_ONCE(p->is_linked) || owner !=3D READ_ONCE(p->sleeping_owner)) + return false; + + __proxy_dequeue_from_owner(p); + } + + return true; +} + +#else /* !CONFIG_SCHED_PROXY_EXEC */ + +static bool proxy_try_dequeue_from_owner(struct task_struct *p) +{ + return false; } =20 #endif /* CONFIG_SCHED_PROXY_EXEC */ @@ -3799,6 +3889,40 @@ void __proxy_block_task(struct task_struct *p) if (!sched_proxy_exec()) return; =20 + if (unlikely(p->is_linked && p->blocked_cpu !=3D task_cpu(p))) { + WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); + + /* Move iowait signal to new CPU. */ + if (p->in_iowait) { + atomic_dec(&task_rq(p)->nr_iowait); + atomic_inc(&cpu_rq(p->blocked_cpu)->nr_iowait); + } + + /* + * Moved the linked task to p->blocked_cpu which is now + * resposnible for chain wakeup. TASK_ON_RQ_MIGRATING + * above stalls task_rq_lock() until p->on_rq is + * transitioned to 0 (See ___task_rq_lock()). + */ + proxy_set_task_cpu(p, p->blocked_cpu); + + /* + * Preserve p->blocked_cpu for any potential future + * additions to p->blocked_head. The chain should be + * re-directed to the same "blocked_cpu". + * + * XXX: Technically, it is possible to keep the + * "blocked_cpu" for this sub-chain different + * compared to our ancestors but the chain-wakeup + * would need to juggle multiple locks then to + * ensure it doesn't race with concurrent wakeups. + * Matching "blocked_cpu" requires only a single + * rq_lock which is simpler. + */ + return; + } + /* * This is the CPU where the __active_blocked_donor() slowpath will * resume activation of blocked donors queued on this task. @@ -3819,6 +3943,25 @@ void __proxy_block_task(struct task_struct *p) */ static inline bool proxy_needs_return(struct rq *rq, struct task_struct *p) { + if (!sched_proxy_exec()) + return false; + + /* See try_to_wake_up() for ordering guarantees. */ + if (unlikely(p->is_linked)) { + /* + * Task is queued on an owner's blocked_node. + * + * Dequeue from owner's blocked_node before going + * further. Do not take the fast-path since this task is + * fully blocked and it is necessary to clear + * p->blocked_on indicator before waking up. + */ + WARN_ON_ONCE(p->on_rq); + proxy_dequeue_from_owner(p); + clear_task_blocked_on(p, NULL); + return true; + } + /* * Typically per __set_task_cpu(), task_cpu(p) =3D=3D p->wake_cpu. * @@ -3922,10 +4065,34 @@ ttwu_do_activate(struct rq *rq, struct task_struct = *p, int wake_flags, */ static int ttwu_runnable(struct task_struct *p, int wake_flags) { + struct rq *rq; + + if (sched_proxy_exec() && READ_ONCE(p->is_linked)) { + /* + * If p->is_linked put us on the ttwu_runnable() path, + * try to de-link the blocked donor outside the + * __task_rq_lock() first. + * + * If the task state changed, a wakeup raced with + * ttwu_runnable() (see proxy_try_dequeue_from_owner() + * for interleaving). Try again with rq_lock held where + * state is guaranteed to be stable. + */ + if (proxy_try_dequeue_from_owner(p)) { + WARN_ON_ONCE(READ_ONCE(p->on_rq)); + clear_task_blocked_on(p, NULL); + return 0; + } + } + ACQUIRE(__task_rq_lock, guard)(p); - struct rq *rq =3D guard.rq; + rq =3D guard.rq; =20 - if (!task_on_rq_queued(p)) + /* + * schedule() managed to block the task and there is + * no unlinking needed. Continue with wakeup. + */ + if (!p->needs_rq_sync) return 0; =20 update_rq_clock(rq); @@ -6972,6 +7139,11 @@ static void proxy_enqueue_on_owner(struct rq *rq, st= ruct task_struct *owner, WARN_ON(!p->on_rq); WARN_ON(p->sleeping_owner); =20 + /* + * See __proxy_block_task() on how p->blocked_cpu is used + * to re-direct the task to a handler CPU when + */ + WRITE_ONCE(p->blocked_cpu, READ_ONCE(owner->blocked_cpu)); WRITE_ONCE(p->sleeping_owner, owner); list_add(&p->blocked_node, &owner->blocked_head); WRITE_ONCE(p->is_linked, 1); --=20 2.34.1 From nobody Mon Sep 28 05:45:30 2026 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012017.outbound.protection.outlook.com [40.93.195.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 642773B0AC4 for ; Wed, 26 Aug 2026 06:34:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.17 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726062; cv=fail; b=Pfl/Yd4wz6ew6TiPeP47QEhIyMVMPBz5NIWKkfRL3suZHJmduhhd038FBych2ykuIUBkmPYYPZO6dMu146gojcgbd6HBxV7JgtSB8nWLlGe9xgitKfG4qg5RU20+da7gsyPhi+mtLmRrBrXEprSlLgJiovdfCy595+wf1j+FnoA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726062; c=relaxed/simple; bh=Ur/5m/baFfECEGg6Zrpl0SQ1HdH6G0mLG8RH93eqHRg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=t+OUSfqnULipJX2Iyz/GZhXFaO95A1FWxrRmvQgZExoCDS0LFYxwRbFzme/6EFfv5JH75tpLwtcjB212iF1Xe4tS6ZH2BsTgGo6afCNrTh7dgIcuK0kfNWidFhKiDEgk9PSBcY1XoVT6pxxbLhfM3dwL2RBFudE1pA7KVY2m+pk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=unV0q6Bt; arc=fail smtp.client-ip=40.93.195.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="unV0q6Bt" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cf8K3TWrXbspYz+keRcjiTDdgq9ySqRch3qsKc+CUxkYAzbp+wS/xDmCpmra428TlBEXCZ6p2HXjNo9Ov8a2dDvv23kAcWCJNTERoBzYofQJ8tL+DmzUfN4VOmsMP1LNV1NfFj2DHcrKUbFEIBAy3PEX8ya5CQtxOji6O3l/mDjsdKwxj+X56eFu0rDMmK0JN4Qcez1jK2tSqYGkYu9lorQoMHaX1ZDyFjXnnoNOeKx/fSgCqZEoKOUXPfSfnEE+ekbOHso2VzJXNIvbm2mmxmBkbVilspg0isfu90Csu3oflWYCllQO2og7JFRM702KUIIRa1QLDunUCovhiFXPxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6KYDFc/+Y7XqWlVIILwkLGo87HLzsHRcJP8pS8Zz2EM=; b=egXkHfzroEqp9GJ6mPtnBGB1NpcZ52f4W8OwECpg9jgBAIqNt+TJMH3A0CjikAtRXRo5jJ/3f4WayRBKsGE2n9UZJl3CFVlamBGd4L+x1AoBIZA1Hh410GBgdpniLtAR1DljPBt6Va5w+AVie9u48Yw/VkzaTcQ4f0x9qeO6ANTaFocCk6GcsTnsREf3zfjUFVOoEGMGRWftS0niQ2PbszdaTEd25OPVyGZUs07QrL8l7vYghS8FA4vE4F0UwToimJ0yM5v00LiUhJYCaTrmyLwXfLuA30KgL7ZtdlFf66dWwEqGtSddab8lNZvkkRonxRh30U40n7VZtJXlZBNEag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6KYDFc/+Y7XqWlVIILwkLGo87HLzsHRcJP8pS8Zz2EM=; b=unV0q6Bt2CXHO6y26UsP8hgDMwQ2c0mSYKdeXPL9vY3jRpP94L3scv7Q2Dfy0e2jb5+UvcnwHOneghEKwxSLDX4DC2TIqV56gKf7Zr+Hc0DbhBzToqhaiy9imxN1sUJkMUkVgQQq1Y8a1idRoNkawn5Hp7S6RNg0bPfe+699ceU= Received: from CY8PR10CA0022.namprd10.prod.outlook.com (2603:10b6:930:4f::23) by DS7PR12MB6358.namprd12.prod.outlook.com (2603:10b6:8:95::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:34:14 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2603:10b6:930:4f:cafe::62) by CY8PR10CA0022.outlook.office365.com (2603:10b6:930:4f::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.10 via Frontend Transport; Wed, 26 Aug 2026 06:34:14 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:34:14 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:34:09 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 14/16] sched/core: Introduce chain-wakeup to activate blocked donors Date: Wed, 26 Aug 2026 06:28:58 +0000 Message-ID: <20260826062901.2137-15-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|DS7PR12MB6358:EE_ X-MS-Office365-Filtering-Correlation-Id: 24088f05-6d7e-4c93-a34f-08df033c0c1e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|23010399003|82310400026|1800799024|36860700016|7416014|10067099003|56012099006|22082099003|18002099003|11063799006|921020; X-Microsoft-Antispam-Message-Info: rR0j1mQJDU6I1TxxGa0Cn/aIPfiR+XpHS9N1e1thIOKPZSWO26leMFTYLnarvisC/NI06eOMlRxS/XnBp0d0JAmx/0KFjKKvlIIM+xMLKYTqMo3g6g+9fZ8xcBZ94uEEcHwSnPmvTcMNnPjU0nhOd+qXEf2I1zkrPviL9DcHflB70ciFjXwPLt/wq0ATKv/N/upjIW6pydKOYgRXinEm8USLQgKED59s4CEiKShV98MZiTwx2kwDNsu3WkgiL0jKItMrwIsORs4dMs3tSz8+wfZjPVgmcH0KmpWKWDZIzZ3sLzRi+Sc0HLaBtMWD1s8LtNCK7lhtIsT0/w7wZCiGKEZCrVjPIfxKpsEnze/DL+pOW/rsufzkDTOMRlkZoFDjWeAV3FNzE/GyWSbsHL1foYe5fOtBdLrR3Vxr8q3c6S8uR7/xWo6AgthSeLbgY1OBO7MSLme6wL9xmhK11YktJ++HtKdeEFfyMYQPFeFQamnLYv8z78Ifh1+toMgrn9bPErMTVag8EMkX1lnTz0daB+TRMnKbJtSSJMXmQ5ZOZbFyzGfSEttCte2i1SYWfKwwVX+DkBeEMgiwNPG+YCK/GuO2gDe6hd+U6Sk7wgtum+jAelUnHyTb4uKnzUury18E4N/AVi6rrKFe2rsVWZjbmPgO9n+WUmquj5ymYApFMtRRGK8QDpLS17CHW3GpwbvtRCZ33ZI1XH9qMGqYfx0wT0rk+MnSvvVEDfx78/XrAsTBOqIlLfB7yhY6jw+jq3GG X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(376014)(23010399003)(82310400026)(1800799024)(36860700016)(7416014)(10067099003)(56012099006)(22082099003)(18002099003)(11063799006)(921020);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: W+sew0AFRlPBKTmGXQgHU753rf86fakJ/0cpQHVJmiAuR0pKfIC+AFwwG74IkWnb6lxjdoWCJRg012HhWPWDzwMogg/8kH5gvv3GUKCWgbI79gEYHHLJs3Nj5p2PrjrFqiRfHw8IuT7J898ISX1iqSVIAEJIoucJqYmvfQX33C+qPHHPb/OE0GTUgyuehpguPxohgJzKITlJOcLOtYvSAE9tDVlou/8lxEj8AzM++FoClLEbL7TWLFT9mD6KRvdEpGrmKXXd7JlrpQ0zRrYWvlQ986GZXvO3LRXoVIlpOh9BUxtOU//+RLLX/OL0gMfMBJpZoJQrxo+EGOzOUNrdomnEDphxAbg6JIkABlN2wo5Uh7D4NJ1u+HQmzf7gk70F4zwmM5+l42FRGQXSpkrD7C1RBn5mGV7hlFxjptf2mlulkrz/Pelvtwf87PUzggJ+ X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:34:14.5266 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 24088f05-6d7e-4c93-a34f-08df033c0c1e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6358 Content-Type: text/plain; charset="utf-8" Activating blocked task with sched_proxy_exec() needed grabbing p->blocked_lock for all wakeups to prevent missing additions to blocked_head during wakeup. With proxy_enqueue_on_owner() converted to plug the race between enqueue on owner and owner's wakeup, simplify the wakeup path with the following three step process: activate_blocked_task(owner) /* Part 1: Enqueue race */ enqueue_on_owner(owner, p) owner->on_rq =3D TASK_ON_RQ_MIGRATING; list_add(&p->blocked_node, &ow= ner->blocked_head) smp_mb(); smp_mb(); if (list_empty(&owner->blocked_head)) if (owner->on_rq) /* Raced */ /* list_del_init(&p->blocked_node) * Fast-path: Future additions will return; * see owner->on_rq !=3D 0 and dequeue /* * themself. * Owner has to observe !list_empty() after */ * this point so it is safe to block. __activate_task(p) */ block_task(p); /* END: Part 1 */ enqueue_on_owner() holds mutex->wait_lock and "owner" needs to grab the same wait_lock during unlock so "owner" cannot disappear during enqueue_on_owner(). /* Part 2: Prepare migration chain */ rq_unlock(rq); LIST_HEAD(migration_list) LIST_HEAD(wakeup_list) try_to_wake_up() /* Sees proxy_task_is_linked(p) */ ttwu_runnable(donor) rq_lock(rq_of(owner->blocked_cpu)) task_rq_lock(donor) /* Same rq */ do { list_for_each_entry(donor, &owner->blocked_head) if (task_is_linked(d= onor) donor->on_rq =3D TASK_ON_RQ_MIGRATING /* Part 1 */ list_del_init(= &p->blocked_node) if (list_empty(&owner->blocked_head) activate_blocked_task(donor) list_add(&doner, &migration_list) /* Same as left */ else list_add(&donor, &wakeup_list) list_add(&owner, &wakeup_list) owner =3D list_next_entry(&migration_list) } while (!list_empty(&migration_list)) rq_unlock(rq_of(owner->blocked_cpu)) /* END: Part 2 */ Since owner->blocked_cpu matches with task_cpu() of the entire chain (including tasks queued on the blocked donors, and task queued on them, and so on), grabbing "rq_of(owner->blocked_cpu)" is enough to stall any concurrent wakeups for any of the linked tasks. If try_to_wake_up() wins, donor will remove itself from the chain and continue its wakeup as if it is a blocked task waking up. If activate_blocked_task() wins, it will set all the donors to TASK_ON_RQ_MIGRATING, repeating Part 1, to prevent other tasks from queueing on it form then on and iterates its "blocked_head". This is done in a breadth-first fashion until full chain is set to TASK_ON_RQ_MIGRATING. "owner->blocked_head" traversal and manipulations are still guarded by "owner->blocked_lock" which has not been depicted here. /* Part 3: Wakeup chain */ rq_lock(rq); __activate_task(p); /* Activate top owner first */ /* Activate the entire chain next */ list_for_each_entry(donor, &owner->blocked_head) list_del_init(&donor->blocked_node) __activate_task(donor) /* END: Part 3 */ The wakeup steps skipped for donors is performormed during the chain wakeup. Advantages of this method: o Does not involve juggling p->pi_lock, rq_lock, owner->blocked_lock in addition to checking task state. Task on chain is fully blocked and only one rq_lock + onwer_blocked lock can hold off ttwu_runnable() from waking them. o No need to borrow reference to "owner" before queuing. The mechanism ensures p->blocked_owner remains valid as long as is task is on chain. Disadvantages: o Since chain wakeups funnel through a single CPU, it is possible for increased lock contention however large chain wakeups and concurrent wakeups of donors for reasons other than an unlock is rare. Co-developed-by: John Stultz Signed-off-by: John Stultz Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 183 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 181 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e6abcea97683..3153ad007447 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2247,6 +2247,8 @@ void activate_task(struct rq *rq, struct task_struct = *p, int en_flags) =20 #ifdef CONFIG_SCHED_PROXY_EXEC =20 +static inline void proxy_set_task_cpu(struct task_struct *p, int cpu); + static void __proxy_dequeue_from_owner(struct task_struct *p) { list_del_init(&p->blocked_node); @@ -2337,6 +2339,183 @@ static bool proxy_try_dequeue_from_owner(struct tas= k_struct *p) return true; } =20 +static void +proxy_activate_blocked_task(struct rq *rq, struct task_struct *p, int en_f= lags) +{ + int iowait_count =3D 0, load_contrib_count =3D 0; + struct task_struct *donor, *owner, *tmp; + int blocked_cpu, this_cpu =3D cpu_of(rq); + LIST_HEAD(migration_head); + LIST_HEAD(wakeup_head); + struct rq *blocked_rq; + bool needs_migration; + + WRITE_ONCE(p->on_rq, TASK_ON_RQ_MIGRATING); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); + + /* + * Pairs against smp_mb() in proxy_enqueue_on_owner() which + * orders ownwer->on_rq state against the blocked_head addition. + */ + smp_mb(); + + /* + * Fast-path: Tasks blocking on us will see p->on_rq updated and + * will bail out natturally in proxy_enqueue_on_owner(). + * + * proxy_enqueue_on_owner() holds the wait_lock to prevent owner + * from running and disappearing before the transient task can + * observe p->on_rq change and dequeue itself. + */ + if (list_empty(&p->blocked_head)) { + __activate_task(rq, p, en_flags); + return; + } + + /* + * Slow-path: Since we need to wakeup tasks (and task queued on + * them, and task queued on them, and ... you get the gist) we + * need to grab some locks. + * + * Lucky for us, we have simplified this via p->blocked_cpu + * which is the task_cpu() for the entire chain. Wakeup is + * broken into two parts. + * + * Part1: Under the rq_lock + blocked lock, mark all tasks on + * the chain as TASK_ON_RQ_MIGRATING in breadth-first manner. + * + * Part2: Active all the tasks in bulk. + */ + + /* Part 1: Prepare the blocked donor chain. */ + blocked_cpu =3D p->blocked_cpu; + blocked_rq =3D cpu_rq(blocked_cpu); + + /* + * Blocked chain is linked to a different rq. Drop the rq_lock + * and migrate the chain over before wkaing it up here. + */ + needs_migration =3D blocked_rq !=3D rq; + if (needs_migration) { + raw_spin_rq_unlock(rq); + raw_spin_rq_lock(blocked_rq); + } + + owner =3D p; + + do { + guard(raw_spinlock)(&owner->blocked_lock); + + /* Iterate the tasks blocked on this owner. */ + list_for_each_entry_safe(donor, tmp, &owner->blocked_head, blocked_node)= { + /* Everything hinges on this assumption. */ + WARN_ON_ONCE(task_cpu(donor) !=3D blocked_cpu); + WARN_ON_ONCE(donor->se.sched_delayed); + + WRITE_ONCE(donor->on_rq, TASK_ON_RQ_MIGRATING); + ASSERT_EXCLUSIVE_WRITER(donor->on_rq); + + if (donor->in_iowait) { + delayacct_blkio_end(donor); + iowait_count++; + } + + if (donor->sched_contributes_to_load) + load_contrib_count++; + + /* + * Pairs against the smp_mb() in proxy_enqueue_on_owner(). + * See the comment at the beginning of the function. + */ + smp_mb(); + + /* + * If the donor has more task queued on it, + * add it to the migration list else directly + * queue it onto the wakeup list. + */ + if (!list_empty(&donor->blocked_head)) { + list_move_tail(&donor->blocked_node, &migration_head); + } else { + list_move_tail(&donor->blocked_node, &wakeup_head); + } + } + + owner =3D list_first_entry_or_null(&migration_head, + typeof(*owner), + blocked_node); + + /* + * Move the owner to the wakeup list before preparing + * the sub-chain queued on it. + */ + if (owner) + list_move_tail(&owner->blocked_node, &wakeup_head); + + } while (owner); + + /* Part 2: Bulk wakeup */ + + atomic_sub(iowait_count, &blocked_rq->nr_iowait); + + if (needs_migration) { + raw_spin_rq_unlock(blocked_rq); + raw_spin_rq_lock(rq); + + update_rq_clock(rq); + en_flags |=3D ENQUEUE_NOCLOCK; + } + + rq->nr_uninterruptible -=3D load_contrib_count; + + if (!(en_flags & ENQUEUE_NOCLOCK)) { + update_rq_clock(rq); + en_flags |=3D ENQUEUE_NOCLOCK; + } + + /* Activate the top owner first. */ + __activate_task(rq, p, en_flags); + + /* + * Add ENQUEUE_MIGRATED for all tasks on the chain + * since they are all sched_migrated_on_blocking + * and have skipped ->migrate_task_rq() callback. + */ + en_flags |=3D ENQUEUE_MIGRATING; + + list_for_each_entry_safe(donor, tmp, &wakeup_head, blocked_node) { + __proxy_dequeue_from_owner(donor); + + /* + * Correct the task_cpu() before activating. + * + * Since chain preparation already adds enough barrier + * after storing p->on_rq =3D MIGRATING, no additional + * barriers are required after modifying p->is_linked + * via __proxy_dequeue_from_owner() above. + */ + if (needs_migration) + proxy_set_task_cpu(donor, this_cpu); + + __activate_task(rq, donor, en_flags); + wakeup_preempt(rq, donor, en_flags); + /* + * Blocked tasks do not add push callbacks and it is + * safe to skip calling ->task_woken() here. + */ + } +} + +static void activate_blocked_task(struct rq *rq, struct task_struct *p, in= t en_flags) +{ + if (!sched_proxy_exec()) { + __activate_task(rq, p, en_flags); + return; + } + + proxy_activate_blocked_task(rq, p, en_flags); +} + #else /* !CONFIG_SCHED_PROXY_EXEC */ =20 static bool proxy_try_dequeue_from_owner(struct task_struct *p) @@ -2344,13 +2523,13 @@ static bool proxy_try_dequeue_from_owner(struct tas= k_struct *p) return false; } =20 -#endif /* CONFIG_SCHED_PROXY_EXEC */ - static void activate_blocked_task(struct rq *rq, struct task_struct *p, in= t en_flags) { __activate_task(rq, p, en_flags); } =20 +#endif /* CONFIG_SCHED_PROXY_EXEC */ + void deactivate_task(struct rq *rq, struct task_struct *p, int flags) { WARN_ON_ONCE(flags & DEQUEUE_SLEEP); --=20 2.34.1 From nobody Mon Sep 28 05:45:30 2026 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012023.outbound.protection.outlook.com [40.107.209.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C28732E7162 for ; Wed, 26 Aug 2026 06:34:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.23 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726079; cv=fail; b=NcB1Y3SR+15hxoazwqZnvjfFYSdUJoiTsYxJBigR2bYizXcvZbNUC2GIEvFpdpNepUhlhLUqVdZtyvHuJsIQIQ5t4kkM3Zy8WlI4W26QedRzJ21NPOqIUbVoCgy9iWhrOzsdM8tjnPfph5P2LEeOP1ScQMlS89iElBsIZlt4kio= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726079; c=relaxed/simple; bh=CqXUgQZFz4PTFo2oD7ogkUbezg2VWT1Na9MewITSpMo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=lpXfJeNkIFfMUL80ZcmrhTMv+ut98Y84cO0Y2+J1UhKQ7xSxtAEuLa2AnqKVG4um6FDnsn51J+d9IUITJm71tE7tWjFyfwQyuknpdCu0LZBEu/wMEOXj+I98AeBnuOlUSVVzExrGVKufTXNXeAc8bQM0VAVcCjTw4O1a4f+WnkU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=5pgbC/Tw; arc=fail smtp.client-ip=40.107.209.23 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="5pgbC/Tw" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ikmfwnlyrNx+76iYS5jBh51VIhVtEeJs+/+5XyyXK06zymsphhW5haLT9poi7Z2Gg269cKAiLyq7FTu0hfOS0Ohx1vHy5RcNM3+HrN2a8xqWE+VZeU5S6KFryBehS9AYcjHFeJx0r8E4GLOCLwQTU3joxcFylW03jvymsa54FejXEgcbigXJCkNjMDp7gCLveE7RwOpJMZkI3x872VIldjX4bziP0Th+Pydro8bYyc9grxTlUo4oFn09J3ZKYU3r1WzwvkqpHtIc4mt5fevtR8DtPDPZzvrqJuAmh7eNgrDD0CH7BH0e0MsWSgmcf3PXHLSYE0zVYUZ96Seo3MGkOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f6kVmQL16rORhVmLfjDlY+924TDnTsWH99rTIi8TASA=; b=MNdSAb+MOSba9gRsxUN5qJoFFNEgEn7eN3DHpRgRGbDkSz8wRuRmzPkxPPYbnuAD4Z4Po3mlgrLFmEhEZMxqXh+IRB96jp/tiMrv6LycYsTakoOBjjdjuJnQmExlinF2vOJBDRPuxtnWOhdm0tOdomFiK6G2VaEckdnbGrVOskReApiu+oo3yU8UuiBn4LdDIAkPmF2NOE5HrESS6KgXXB1lDZ4Z4qtgMeK/UocF4AF0R9rnWrSyLSr2xZb6OWFFcGl8o8ABZVsom0ZArqUam0HaT3EkEOENNOyCbnVh0uVabaZklqVsUHycg3eauqzhhavVUiZujK7DFDXkKRUsZw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f6kVmQL16rORhVmLfjDlY+924TDnTsWH99rTIi8TASA=; b=5pgbC/TwLMCcQMBhMjRq+6phoBLLdR//zq1lURYGcwKUQGKidXszMT0pvq38xaMEFE350XaRF11uY6d7MHi1bvsC2+zdF6Tir+zrMrHZ7j1x3LjFv5LRrc+hf2kEJ59EC6lN/TC5utH23gwzApnM2pE0NWCyd4NwvqqwKqX/HJg= Received: from CYZPR14CA0003.namprd14.prod.outlook.com (2603:10b6:930:8f::16) by MN0PR12MB6342.namprd12.prod.outlook.com (2603:10b6:208:3c1::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.6; Wed, 26 Aug 2026 06:34:33 +0000 Received: from CY4PEPF0000E9DA.namprd05.prod.outlook.com (2603:10b6:930:8f:cafe::3f) by CYZPR14CA0003.outlook.office365.com (2603:10b6:930:8f::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.8 via Frontend Transport; Wed, 26 Aug 2026 06:34:32 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9DA.mail.protection.outlook.com (10.167.241.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:34:32 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:34:27 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 15/16] locking/mutex: Track locks owned by a task in a per-task counter Date: Wed, 26 Aug 2026 06:28:59 +0000 Message-ID: <20260826062901.2137-16-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DA:EE_|MN0PR12MB6342:EE_ X-MS-Office365-Filtering-Correlation-Id: 74406ff6-785c-48a2-d8b3-08df033c16e6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|82310400026|36860700016|1800799024|7416014|376014|921020|10067099003|11063799006|56012099006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: f/bZ2RHswvnCHI3/zpcotoyqwya/SlRzC7KHVCTBQe+bYJagnIBJ/WYfgnPdAtbsYNuBFeitOBmABgXZ6t8tmi0gOS5aWY5Dehh8TtdyKQvbje3BuW4A36wi81C6KH2pgqVTGr1Iz+hK8iDByWNBllC8Z4pMRV5+rVRawo0TVL6YUep3sOROJkl5si2yInploQD/PKFmQ/q5pZAW6Jmp0cobIKBjKCUt7OtAeQjr3MJ7RVMgLyKT4JLcoTLvptJlMRy3tg/HfrrNTlwtbBVjo6Xm8WZLWnbYlnmrQbpBnzc+wGUnvHV/FL+TIxaOMXvi96rH+5cQmLXIpDodMtCMwFGnK6gT03d7EjhanRFQ8yna3pG56VLF0Tp/veUkAH6rYmgoxaixt931pQBYkQepe8I+DDL14P/eESlvasLrMbDSLc1yfiZikO6bhqzCCSO9C7h8lyglr1gD1GjcPBeU0E2rawzs0T9Q1Z+7UhKZY/fCyKILXypeiaJ6WXWLm42db2B4Hayj/SZ9QZO/I1dKk6DUOumAQ6zGnpH0OPgs3dwT+gQ/6s90u0BelM2sNdGp02X7Ku5OM6KS7Ow7Y7vONyzdrH3seYEgXtB/GM+UgWjmxn69X8xziTyklXFTcGnW34n0s+2ejuks6hELdGCojSLb3k1Tk/jRKiuYu2icmzAEgL1vzQHH2Xt/kWIMORBMqzLBRhp2qx3XFh7MIcZFzE25q2kGLcE5JOh/MmX8gN2/JKxvlkT85ZBxDrA8F4lO X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(23010399003)(82310400026)(36860700016)(1800799024)(7416014)(376014)(921020)(10067099003)(11063799006)(56012099006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mPIyEr+R6ob7pQwFfN7nT3TvUN1Cb5fh9hILmnwl12sHQ2bl5KhfLLjX4bYh6+baXPGThL6HW8IBbuVh2z/3dgCGbuI0gHYZz/J6ITEZeRjIMz20BccKQ2Q1Rebs2XgWGmZhagYD8gJUh7nWQAnjcK4sNO9Lu4n4mm/5mI1zovMZl38zUF5JwukSd0P9e6/6SXbY3+GEW+kt5Ql/y1Ew/1ch9kneWnKASy6PjRYyZGJtBTBickSg1+R/J2+OeXUb8ldNVo3fjgNXgAShvsat87D8DQwGZVRHfSsRdD6idT+uNUDVI0vJMpdArvEW4PXIccBONaKaf0gaXNOExp7+UY7dJHmjma7GgWs2hQ3Og9ubrvWcMbpZ/wxTV+bYauUokGxU8Al8RaHna0ZUS/Hz+gyzmRXV4tesq8vWEcfKnc1NKL6rsRXupCT/unEp+IAP X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:34:32.6166 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 74406ff6-785c-48a2-d8b3-08df033c16e6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DA.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6342 Content-Type: text/plain; charset="utf-8" With sleeping owner handling, the task activation path will need to be extra careful when waking up a task that can resolve to a mutex_owner(). Track the lock acquisitions in a per-task counter to selectively put mutex owner activations onto a slowpath. The per-task accounting is done locklessly on the current task. This is only relevant for a blocking owner and barrie in __block_task() ensures the observer sees the correct "lock_nesting" before p->on_rq is transitioned to 0. The counter will be used in subsequent patches to optimize activate_blocked_task() and __proxy_block_task(). One exception to the lock tracking rule is the increment before schedule_preempt_disabled() and decrement after to fix the former mis-count. This is necessary for mutex since a handoff can set fully blocked task as mutex_owner and, at that moment, other tasks can start queuing themselves onto the blocked owner which requires the wakeup path to go via slow-path and ensure blocked waiters (if any) are activated. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 6 +++++ kernel/locking/mutex.c | 57 ++++++++++++++++++++++++++++++++---------- 2 files changed, 50 insertions(+), 13 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index a05240b52806..cb58992814fb 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1272,6 +1272,12 @@ struct task_struct { struct task_struct *sleeping_owner; /* task our blocked_node is enqueued= on */ int blocked_cpu; /* CPU where task was blocked. */ #endif + /* + * HACK: These bits should technically live in CONFIG_SCHED_PROXY_EXEC + * block and only account when sched_proxy_exec() is true but for the + * PoC keep it outside and assume proxy is always enabled ;-) + */ + unsigned int lock_nesting; =20 /* * The task that is boosting this task; a back link for the current diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 8a85912d7ee6..23a63e117c3e 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -84,7 +84,8 @@ unsigned long mutex_get_owner(struct mutex *lock) */ static inline struct task_struct *__mutex_trylock_common(struct mutex *loc= k, bool handoff) { - unsigned long owner, curr =3D (unsigned long)current; + struct task_struct *cur_task =3D current; + unsigned long owner, curr =3D (unsigned long)cur_task; =20 owner =3D atomic_long_read(&lock->owner); for (;;) { /* must loop, can race against a flag */ @@ -109,8 +110,10 @@ static inline struct task_struct *__mutex_trylock_comm= on(struct mutex *lock, boo } =20 if (atomic_long_try_cmpxchg_acquire(&lock->owner, &owner, task | flags))= { - if (task =3D=3D curr) + if (task =3D=3D curr) { + cur_task->lock_nesting++; return NULL; + } break; } } @@ -153,13 +156,16 @@ EXPORT_SYMBOL(mutex_init_generic); static __always_inline bool __mutex_trylock_fast(struct mutex *lock) __cond_acquires(true, lock) { - unsigned long curr =3D (unsigned long)current; + struct task_struct *cur_task =3D current; + unsigned long curr =3D (unsigned long)cur_task; unsigned long zero =3D 0UL; =20 MUTEX_WARN_ON(lock->magic !=3D lock); =20 - if (atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr)) + if (atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr)) { + cur_task->lock_nesting++; return true; + } =20 return false; } @@ -167,9 +173,15 @@ static __always_inline bool __mutex_trylock_fast(struc= t mutex *lock) static __always_inline bool __mutex_unlock_fast(struct mutex *lock) __cond_releases(true, lock) { - unsigned long curr =3D (unsigned long)current; + struct task_struct *cur_task =3D current; + unsigned long curr =3D (unsigned long)cur_task; + + if (atomic_long_try_cmpxchg_release(&lock->owner, &curr, 0UL)) { + cur_task->lock_nesting--; + return true; + } =20 - return atomic_long_try_cmpxchg_release(&lock->owner, &curr, 0UL); + return false; } =20 #else /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -723,8 +735,23 @@ __mutex_lock_common(struct mutex *lock, unsigned int s= tate, unsigned int subclas =20 raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q); =20 + /* + * Mutex handoff can set the current as the lock owner while + * it is blocked. Increment "nesting_count" on the way to + * schedule() to notify the wakeup path that __mutex_owner() + * can resolve to this task on the way back. + */ + current->lock_nesting++; + schedule_preempt_disabled(); =20 + /* + * Fix the guard against handoff. If the task got the lock, + * __mutex_trylock*() will increment the lock_nesting when the + * acquisition is finalized before exiting. + */ + current->lock_nesting--; + first =3D lock->first_waiter =3D=3D &waiter; =20 raw_spin_lock_irqsave(&lock->wait_lock, flags); @@ -981,7 +1008,7 @@ EXPORT_SYMBOL_GPL(ww_mutex_lock_interruptible); static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, u= nsigned long ip) __releases(lock) { - struct task_struct *donor, *next =3D NULL; + struct task_struct *cur_task, *donor, *next =3D NULL; struct mutex_waiter *waiter; unsigned long owner; unsigned long flags; @@ -997,6 +1024,10 @@ static noinline void __sched __mutex_unlock_slowpath(= struct mutex *lock, unsigne * missed. */ guard(preempt)(); + + cur_task =3D current; + cur_task->lock_nesting--; + /* * Release the lock before (potentially) taking the spinlock such that * other contenders can get on with things ASAP. @@ -1006,10 +1037,10 @@ static noinline void __sched __mutex_unlock_slowpat= h(struct mutex *lock, unsigne */ owner =3D atomic_long_read(&lock->owner); for (;;) { - MUTEX_WARN_ON(__owner_task(owner) !=3D current); + MUTEX_WARN_ON(__owner_task(owner) !=3D cur_task); MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP); =20 - if (sched_proxy_exec() && current->blocked_donor) { + if (sched_proxy_exec() && cur_task->blocked_donor) { /* force handoff if we have a blocked_donor */ owner =3D MUTEX_FLAG_HANDOFF; break; @@ -1027,7 +1058,7 @@ static noinline void __sched __mutex_unlock_slowpath(= struct mutex *lock, unsigne } =20 raw_spin_lock_irqsave(&lock->wait_lock, flags); - raw_spin_lock(¤t->blocked_lock); + raw_spin_lock(&cur_task->blocked_lock); debug_mutex_unlock(lock); =20 if (sched_proxy_exec()) { @@ -1036,7 +1067,7 @@ static noinline void __sched __mutex_unlock_slowpath(= struct mutex *lock, unsigne * current through this lock, hand the lock to that task, as that * is the highest waiter, as selected by the scheduling function. */ - donor =3D current->blocked_donor; + donor =3D cur_task->blocked_donor; if (donor) { struct mutex *next_lock; =20 @@ -1045,7 +1076,7 @@ static noinline void __sched __mutex_unlock_slowpath(= struct mutex *lock, unsigne if (next_lock =3D=3D lock) { next =3D get_task_struct(donor); __clear_task_blocked_on(next, lock); - current->blocked_donor =3D NULL; + cur_task->blocked_donor =3D NULL; } raw_spin_unlock(&donor->blocked_lock); } @@ -1071,7 +1102,7 @@ static noinline void __sched __mutex_unlock_slowpath(= struct mutex *lock, unsigne if (owner & MUTEX_FLAG_HANDOFF) __mutex_handoff(lock, next); =20 - raw_spin_unlock(¤t->blocked_lock); + raw_spin_unlock(&cur_task->blocked_lock); raw_spin_unlock_irqrestore(&lock->wait_lock, flags); if (next) { wake_up_process(next); --=20 2.34.1 From nobody Mon Sep 28 05:45:30 2026 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012009.outbound.protection.outlook.com [40.93.195.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAD4C140E5F for ; Wed, 26 Aug 2026 06:34:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.9 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726100; cv=fail; b=rjnRf0gxjOYR/OH4z5VL984/aqdVRA44rwWh5PLnG3HVCj48w2onghQNT4TK//XeDKkP3XmLv+YT/t0xDzdrpFnDyyoHwrCMZW+gE2/4j8LI00I4dBCdAbpGNPiYomuJQP8ihIQRLH+fKlGS2FmyPiqwnQQM4Sr6/NgUzEIo6eQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787726100; c=relaxed/simple; bh=lQdAcFWwl8Ftdhc0/5k+Tu6420QxpWMiRvBBakunjUQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gityM/NLJa76/y4Bp+/1eW1z/uaxT8fAlmDW35wt+GDofYBDu80hSXc2FKZgbTwmMYqLl0H84zCJlFX4qOb1HiZmnd0sTY7uhLNF7H9hX1RQ5mBGLNey4c/zXWGgXDDHgE1Rk3c7H3MAeW+K9QuJS4d/hUqbdEghXjQSbxNswwA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=ik0saz7M; arc=fail smtp.client-ip=40.93.195.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="ik0saz7M" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=r5wuubP/pe2GOmtWw68eiR/YEaY9h5y3iHxCMrhWqTrSvz9lgraiWtM6kn+ED4fSuHAT8TN4wNzN2+VjxAitQa1iwbXMnOgOOM1P4qDvG52YOp7foUtdNagbG7eEXGjJTrXa4yUPqM+7BG4yfh3Ipq8q89CiQ9xE0j4DkLxu2SXJnzzSybqnAzV7+u+aheVymvKu/3MGeHv3QO/eokBmsVkUCRpJLmLpDfz7SjGG/CquXVAgWvl5W+Mj6L8xJtPc5gptDgJaN/Y7anchDIeqXC6C5NtRVl4hjPfJTiR+s3pu0+4zQH72QMOtrFothvmRABPZRnxRqpg/x3vMWWm+ZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FmfhvTTLDdo6BEI432kGlMX0bSWOD/NU+bLLBRNXGgk=; b=um+d5gLbfnEhCRV1cU7JibqASKcJnUrGp5R6CbYVPOQa5B7YNJ2B/dHiToSMJ+MxRD+DXrR3AsQmK75kmxzTQ2cmop0BBLkf4mnyLYWk2c4wmaiQ3rwQNgvHKzFmbHXGTxBawrjnued1YcTL5LCoT5W6tkOse7v88zuvDzW+us2/WBdmXw2oq5PzBQYQx2vsY2divzXJc/d0SVFrW/Ne1n/UdbpFvqRQs4bhe4G//I4HwWEWaWrGzG8Wes63NpHxn9vjxufaJKLpyrCk1KOzjY90ryh18C1KEPoUPlSKvJYWG4iSXG7fu3ZOkpROoKYmpdMVYigshN+ZyV5bUPHBuA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FmfhvTTLDdo6BEI432kGlMX0bSWOD/NU+bLLBRNXGgk=; b=ik0saz7Mwg/baBSlhRUKS1d8CwLL1aHwxZSp+wovjmQRsShIm6p/PfLtVMEjK/a/owFgszom0Kmwn1nagMMQrcNfNJMv3DC1c7AG+e+6l5E6kiyzLgUwRQLEBIvgVVUPhXaoEVNWr2grZeehG1HEx8e8thLkAexdIj6pAk7rWhg= Received: from DM6PR05CA0046.namprd05.prod.outlook.com (2603:10b6:5:335::15) by CH3PR12MB7572.namprd12.prod.outlook.com (2603:10b6:610:144::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.339.12; Wed, 26 Aug 2026 06:34:54 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:5:335:cafe::17) by DM6PR05CA0046.outlook.office365.com (2603:10b6:5:335::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.6 via Frontend Transport; Wed, 26 Aug 2026 06:34:50 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb07.amd.com (149.199.90.133) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Wed, 26 Aug 2026 06:34:50 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Wed, 26 Aug 2026 01:34:45 -0500 From: K Prateek Nayak To: John Stultz , Suleiman Souhlal , Ingo Molnar , Peter Zijlstra , "Juri Lelli" , Vincent Guittot , Will Deacon , Boqun Feng , Andrea Righi , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Waiman Long , K Prateek Nayak Subject: [RFC PATCH 16/16] sched/core: Set activation of non lock-holders to fast-path Date: Wed, 26 Aug 2026 06:29:00 +0000 Message-ID: <20260826062901.2137-17-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260826062901.2137-1-kprateek.nayak@amd.com> References: <20260826062901.2137-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|CH3PR12MB7572:EE_ X-MS-Office365-Filtering-Correlation-Id: c74f03a6-68b2-489e-68ba-08df033c21ab X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|376014|7416014|1800799024|23010399003|82310400026|921020|10067099003|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: g92U38c8pF0ZvXGq/AyYouh67PQ1iLvAjUdh1l2qEECbLa0ryAFB5v5tC3RshRkE4TKqUfrQ2ftMif9Q1kf1zM4koAt5IKxTvy5i1HXy0C85BHTIKAuxV4Ccsd6MIKEB7s8QWT50vhsyIcvpQofThxo6CN2DfQdZm2uM7BHaToV+jfNbnWHw663o3MiGnZh6P0IS/uXsw/92mMG82yh0ZJgX+J+BYY6KGksSTgkQKr/PseA5yhhjQ+4mZk8gkCk6wW9yaK4NurjayzRhY9PW/uoWmERwzkmfnGsbH7ad1rqwWjjsy2O0MCsJVfIQ4UsXc/jcoK6CWvmC0Kalnr8cWdsLKGYihktUz3lr8h0sV3a79mkexZvgR+BsldtNfIdcis5BgKjO3cKmIKlecyQYAXuHZNfqDmPiCQVqcbgBj2QHkst5eudXnLssICH17F9R0IbYkv+EQS7PiZWRHNWVGlBo/rw7k9G+huuwsEHzZKF0N6S7ZTWB/en3gU096rnEVMCAN112LqIBeWNXhUn/W3wIKfsWPzsPFZ8HLQvt7LEvVc4dQefqsOtuMEQ6HbwHAxj5DwZ4226jfx4gQRXNl+ESFk/2q7HnKs6rZEGWoOZPsvnFps7qbyXVYDg3gAiqG6D3NvrPnqSJVqJZwHV3wGGNpl3vXKiHp86xhaHZklV1TyQCwLuGicIhrJhDbTYC+1n+pua6R5bqVA/CFToD7lXvaokoAW+IAaF/cSRJwtYOUiTc1op1ZWy86wkEE32k X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(36860700016)(376014)(7416014)(1800799024)(23010399003)(82310400026)(921020)(10067099003)(56012099006)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uxAN84lt1g02zpGTb6VymAwOLr41iA32L0ej6M66gRXizx7Yhdj8dpiBaRIEImE7Drxe2LHzxsaC8ncDtMFnNB0SzIZ8PizpT6ISn/qyBxV1gGvCOkwWptnCl6WyfUuKF33oawr7zEk4cmv/hFcOQ8OAk9BynGZv7NEysTkHJQZW2e9ZKMY/4jPGN4qzCDTSKVJpYegKsUWXMus7CyNvj1/D88cN5hGGWl/2DOvhsvlL/b+pk3lszY11QvWXYCD1erCai89nkK5txgGG7M9tx2ktlPtZEFOkVktGOHcxryzecOa6vqHcVVG3IVheqI1bFI1SgACiPRpP0PUV8MlCeBPKyWIDnkXfO1orxUOo6MY2YTrdrBlx+Fbn3dlhZKbw2YMP71DVIrRtGkDY5moMvxyLAppa5Qt6Wli8IdgttcFwrypBxj1Fkj4Gh87BA4zo X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Aug 2026 06:34:50.6772 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c74f03a6-68b2-489e-68ba-08df033c21ab X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7572 Content-Type: text/plain; charset="utf-8" Tasks with a 0 count in "p->lock_nesting" cannot resolve to a __mutex_owner(). Set actiavations and blocking of these task to fast-path (plain __activate_task()) to avoid the overheads. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 3153ad007447..3ab4cd36f887 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2508,7 +2508,7 @@ proxy_activate_blocked_task(struct rq *rq, struct tas= k_struct *p, int en_flags) =20 static void activate_blocked_task(struct rq *rq, struct task_struct *p, in= t en_flags) { - if (!sched_proxy_exec()) { + if (likely(!sched_proxy_exec() || !p->lock_nesting)) { __activate_task(rq, p, en_flags); return; } @@ -4065,7 +4065,7 @@ static inline void proxy_reset_donor(struct rq *rq) =20 void __proxy_block_task(struct task_struct *p) { - if (!sched_proxy_exec()) + if (!sched_proxy_exec() || likely(!p->lock_nesting)) return; =20 if (unlikely(p->is_linked && p->blocked_cpu !=3D task_cpu(p))) { --=20 2.34.1