From nobody Mon Feb 9 10:26:39 2026 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2078.outbound.protection.outlook.com [40.107.92.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 899B314A62B for ; Sat, 25 Jan 2025 06:56:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.78 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737788178; cv=fail; b=Ux7q9xP5lgwZClUpKqbSTxRXzrWMiIvKJ/ilpoAG7DI3ajkI7yNJGuOIw5czDpLk1VhIC3ivoHgbmMDkrwxvSm1iwk7FpqKG2LXQw3vC0z5L5tXU3LddkAsmJO/640/PMPdw1LubNmgN+KUi89f7nWMBZEJPL+k416K9HN72E2k= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737788178; c=relaxed/simple; bh=ixbDO0fzoUSAUm5IJA3pBvvKEyxQ4qBF8gSgyp8B550=; h=From:To:Cc:Subject:Date:Message-ID:Content-Type:MIME-Version; b=evFtPrMfXcpcszpoq7QFzjOj/AXz2DYrfyOBWm6JXOGzwZf0y40Ft2GC5EsWqI+pP1J+a5F2DpX9Kwj67fAv3efwFqpSn4TjXSllQjzqztUceAKOcuVU/Hlh1gkNVJvJlEvhzf5j92/VcnqsxMlZ9InZRUVNDU6uuAnKpPuIXIg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JBScAZ/F; arc=fail smtp.client-ip=40.107.92.78 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JBScAZ/F" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ux7+ciA9Q+eOtLpAOTtmmbzEjBxuzIdLdl9UQQqxI3tmMypBNFimleqmJ50XfawnM5AjXEjxOYjDkQ+1kpk8o6XdRQZftbvLictNlbDtyi5NpmRWdooUKv64gaTyT5CDMpF3t4qAL/30BzeyhfjYegmprRPlB0EibovNU8CWHHn60k3IgrfFKFsJJeE6LQdYWjK8ATIJi3QDVBlj5C+y2d69dBDKRmAHhLAp78/wIQK0Sa7zEtfWh2cXT7xqadGxAdQtMfmNMo/p3H4/MPEpEZoNl+JzVLUHHQceGJ1XdScfP0cSQvpmENcJnEBEFqyQ25RpBIJnkMG7ctxPCT30gQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ekE+PaosHBo/qRAqE3KVJrt1A3Qj95MP+1r4A3M0+MU=; b=ehKjLdX6QRADa6T0+q2hkWlpMk1qlhPiu747tZ8ZEVptw6YkmQkLx2ii8W7d2n1S4lkTne2QPVw9N8VUH+oAyoF9EED/fmaI3iA4RgWMVIId6AjteBgBKNrsE6nCOCk09NEynoNXHDbJo0sBUasEGrrS+yj3rtcQoB/BM0MC/A/hZgsbIElSxxkpCbGxFEuXJcPlTy1N5kw2LqkUNwGRgTekSWQ8tAerhS3PsKAMPKviLu2hxHhI91trhuV5e6H3BsFWgtpN6M1/23MZpnM0v8YbC59fFC3IauIT7lvXL4nFdtRDGfEBXKu207BhX1pc0CFERX72udFUterfIusB4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ekE+PaosHBo/qRAqE3KVJrt1A3Qj95MP+1r4A3M0+MU=; b=JBScAZ/F28IknI/O9YivwxR3Uu94rHMx00meC8GxcsT4DmuHp0HkQibW9UqYlHTCKMxH/oYlf7zEo2vXjE8G4nJLrcn5zZFh4xVTni8+U5jFFYGDI0GaeGH6dtlZYwO7OapY6MYLA7PacZEFIbvdkqcv9g/cDUtcE/X4jEk7nGzEYp5K90ze1A3Xw6aBCrDHkQ3Pob+tSiF/BUzD9Hqh1mtPKVXNSM7BAuzC2iWPS4V/Rn0SVYp+LKEfiDZHZYWdwkEI629MrbrDnbzSpDUJCJ+bkq/82Bj6sr5hlOE4f2jjGEUZDs/295oHNsBPnpX7Kek+Tz70O1mAJCyY1E77MQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM4PR12MB6424.namprd12.prod.outlook.com (2603:10b6:8:be::16) by CY8PR12MB7636.namprd12.prod.outlook.com (2603:10b6:930:9f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.17; Sat, 25 Jan 2025 06:56:12 +0000 Received: from DM4PR12MB6424.namprd12.prod.outlook.com ([fe80::8133:5fd9:ff45:d793]) by DM4PR12MB6424.namprd12.prod.outlook.com ([fe80::8133:5fd9:ff45:d793%3]) with mapi id 15.20.8377.009; Sat, 25 Jan 2025 06:56:12 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: linux-kernel@vger.kernel.org Subject: [PATCH v3] sched_ext: Fix lock imbalance in dispatch_to_local_dsq() Date: Sat, 25 Jan 2025 07:56:08 +0100 Message-ID: <20250125065608.181754-1-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: FR4P281CA0244.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:f5::13) To DM4PR12MB6424.namprd12.prod.outlook.com (2603:10b6:8:be::16) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR12MB6424:EE_|CY8PR12MB7636:EE_ X-MS-Office365-Filtering-Correlation-Id: 31810ad0-1bc0-43a0-878c-08dd3d0d5aca X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?1lcb1QcoBA7XBrmN/z36nn+LdyaL2NzEKgd/8I5wniDee/iA8i6MjL9HNc8d?= =?us-ascii?Q?AlU8SL7q93IRG0Suessd/Lz+J85W/9IULT1vKf/Ll8PNm4MFCV1T7ywRMPfo?= =?us-ascii?Q?KhHzJHbtO/4oVPLbdpDK6Q5DQtjMcVnhEAIvN5flzlI0WpyNFJh7N2mha0iH?= =?us-ascii?Q?pOWBsWqskwdvSoUf0wzzXdt8xskxrnqgmuGysa9bdpTgybuIOeVDO54Ahlr3?= =?us-ascii?Q?MKGpvhU5ZiQtSl8ggy8573M7LVgCMapmPlL9ZUgNsg+GS4VktIgZMRu9nj9j?= =?us-ascii?Q?Xxj4LdL4oM7D73M59p8cFSM2tOSb9gj1z1YWczYfm9QZwHoE1czuCgKz+/XH?= =?us-ascii?Q?Pw3vAVvcOC91+JUqOLVdL+2ceD7TPRM75aSHa0Mllrik46QUQE9u1z1gSMpW?= =?us-ascii?Q?fBKxylZW/j4ieEjkBnjn/633viOaxWys3h/I1Om2WlbbQjvoAfWB2/La4vco?= =?us-ascii?Q?0OriUmeuaEGqKUKXOZK+yEbJ7QqHKR/0vC36PS2GTsNO6yc7ElQvR4QMcIZN?= =?us-ascii?Q?fFAWx/h7MOihSxSl3mgciIE+nIm2bjEjLJIH97LY79zEoo9VNSFAIs3ga2Bi?= =?us-ascii?Q?VZ8a9AUW1BwhkFpcGQhyWUwn0JoT66m8m9myz01pNdT0/v0kUNcpf6QCZ3MJ?= =?us-ascii?Q?+25mwmn2zQAnzGMjwYUbYxFjg+jFQ5iGBTdc5c2PRgyKz+LDn5OaeUMCyGUz?= =?us-ascii?Q?IXRezTrpEVrJhIkkxjRvJ7FMfTxHtb7jVMvNo1in10YB+SrD/UwbmdMBnLZh?= =?us-ascii?Q?DbrF8VnnyN7WnoA01kHGFy/3opRsFZgbfpRFZqk8WV4tNEJBjnEzBDdrCWGF?= =?us-ascii?Q?C0v5Iq1D+hSoi6kwAH2fk/ZfjzfaSceNLlZDQgW5hesNPrWls4rguLjK6MvV?= =?us-ascii?Q?FJvs0Z9Fd9XfQh9xuq1d0MiPDfaF8HZ/WMyHRLLme7MnaLMIv2o3knZizA0M?= =?us-ascii?Q?bB0LdyB2i5O0yFtpy1JEiXJOl2ux/gn6I2tBNO/D19KvagyNI+PlO9Foonwx?= =?us-ascii?Q?Xnv1PNAFJMi8fhllcVcdRSHfi5qdhUSCqJul/MTMvBmTH72bumbievb6XfV+?= =?us-ascii?Q?IAtQlHh1/K02/w/9DCOnda4sEkK0Qm0gwuiEZ5LwqaEnYKBFPU/jjR4n2aFh?= =?us-ascii?Q?D//UE1ZvyUGBSjfaT/GxSxpQuD5qaKqaWCD1MJUOLJi8tt4V5DUrP/WUaVv+?= =?us-ascii?Q?EBepsZv1qfx/uA4Ghb9r14+gdNd4VRttkpjvKeK9fWiVSsNr/Yp2WS1rriq6?= =?us-ascii?Q?bxsHJIFLdiawlh/TQJtue2DuFY83FwwhBY5uKaKDWfndUisZfwus903ZqpW1?= =?us-ascii?Q?k7R2gmensbpgiMgIRL1nlX9V12HKHDRek3jhaTfY0o9tFqzCvMIk+4LDnZ0Z?= =?us-ascii?Q?XswrPdxmEUFAtEN0At4WBoDFKG8d?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR12MB6424.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/evmUoeMfH1wHyyMpeD6K238gDSVrgMZRFRi7iLCkDZ3vyuq79Bu2wbczLjZ?= =?us-ascii?Q?HdlcYzpUZbb//lP+s/N/RnxkAEJKRw9VyuQHi7a4euiEP2f8UA+eSAcB6+g8?= =?us-ascii?Q?EI2YH85er45w9+dMU0zanO9M7qdA/1QtZD+WRWO60tNhahC+qguu5tpomZHQ?= =?us-ascii?Q?ZJEEFx0pcrAJQrlLn9aJqK/oRaMA9iHlQACJXq1HF9adxt3gWiA6mm/NezO5?= =?us-ascii?Q?kE+kuPeSV35zF5dy5SeD3WUTgiOptgptXGcr2uauDpZfI/PqPGM0K1WSEIZ3?= =?us-ascii?Q?baSCk0UFfK3TQTV02j+fIiHlUNw9j1O87PlKreCxW8ujRhbJpLfEFzQwb2JM?= =?us-ascii?Q?kczqBsinJyzDzsoeO2OYjPVdQOptHzC+quBmS8Psi0Cw01Yii1NE6xG9/1DR?= =?us-ascii?Q?RJRkZgHxKhUKrOZIMK6+sFh/Th2tgE+VCcqmsgjN/uXwuQAqrOYi5brsKOYs?= =?us-ascii?Q?dYQMOEuOYbbZw/yJ4YCQbzU8PQb4Rl+jHiSviNv3gW0CvzaDmcx5+ZCHFrrk?= =?us-ascii?Q?yDtrqHnD5ESSPbQksVMktBRkec3M8wrp5+mhta1RUP3WTuid9G6ME+/H/mlM?= =?us-ascii?Q?KZ5Km2tGxlMEyZo1DKuNKxiaUENtESGnZtdsl2IwYa3VDfsR5V1ejBo7l7/g?= =?us-ascii?Q?/zzSHN85mtD2xZRKMG5I6nltDcWbQ2MsoOaSO3w0U3SjeypV+SwM/25cwi5X?= =?us-ascii?Q?nZfMVyTaEv548U/rLizlzOcO3TQOk+yQWYfgSNebgHlCB2IRFbGnFqw9dWMn?= =?us-ascii?Q?jyBSlapUdRia/zsetXheJqpm3pHI81duE08aIlN7ujvougMEajqj0LJX+F/0?= =?us-ascii?Q?9lnLvw5tw7KRJeT+QhczsErmNHN4EWgl1h5XB3PQuZxob7IXqHKbIEAFXHpc?= =?us-ascii?Q?56bqp/Ap58x/baM7x4HmgNLsMW4JMTf9PaRmu/xWcTJRUPOgAZgLASE7P6oJ?= =?us-ascii?Q?TvgZOuh1+gfXduKTuXEQzYE4fjQTJ9KkEBhQfxmus5djTWEoY5Lyp7wGBFsy?= =?us-ascii?Q?l9D5g4iE8YPD5QXF3P+3VqJX9vwrZiPgo3U6eDsSrbLjuIRWY2JK58YDpX9i?= =?us-ascii?Q?ZFGO+cEUZCqzlRFQBuA7ejNnQO2lovJPe9vM1qmmJQwxrBpiR6k5AdrCje2M?= =?us-ascii?Q?3S2qtJZl8VHjcwtd+o4WpocMbayv6tzP/yheSFrfqWRvz2BzyQXVRUBo5DVK?= =?us-ascii?Q?ud9CBg2n8RBny+D+N9VQQbP4EXbhs8NfZO8SLZZVKqOgL6KUZ/DIwAOEdYSv?= =?us-ascii?Q?Pa65H2RZbdVVW1xBMX93Jx0BvdaOc97IpVxoZXyqGoOxS6p46nEf4Gzxl9JA?= =?us-ascii?Q?MBQLuesxRZlvEVPzrkrLi7he+9kNluV3NY23WSKXcgtnPhiYIVNgwOKcZ8sX?= =?us-ascii?Q?wxazJKAbgNES9m/2lC9stV3AGq93B6QQxIHoFYgyg1V9RocW9EsP9D/ZTGwn?= =?us-ascii?Q?uIPF+8RvNkdWX5djsJhdM1qhvOWy/61ahpms6xD+nvlFgCYfQS4jZieqSVzf?= =?us-ascii?Q?JV93vZ8bz+d5/1zCVRuCiYvQHif/sioe8NVQt7l4ouM4yfugczKBI3KzEG+9?= =?us-ascii?Q?yIno4E+hMUCZO07Oto36iI3On/q1IixIPC3+KjPQ?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 31810ad0-1bc0-43a0-878c-08dd3d0d5aca X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB6424.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2025 06:56:12.5039 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: S1HHjsZ+ZBbISGW4aR5QJqiuDSnWCIreGizpjWADlGsDbvkXe4FnYcpo+qkyd1NC7QMERZYGCU8o035bbUCVUw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7636 Content-Type: text/plain; charset="utf-8" While performing the rq locking dance in dispatch_to_local_dsq(), we may trigger the following lock imbalance condition, in particular when multiple tasks are rapidly changing CPU affinity (i.e., running a `stress-ng --race-sched 0`): [ 13.413579] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [ 13.413660] WARNING: bad unlock balance detected! [ 13.413729] 6.13.0-virtme #15 Not tainted [ 13.413792] ------------------------------------- [ 13.413859] kworker/1:1/80 is trying to release lock (&rq->__lock) at: [ 13.413954] [] dispatch_to_local_dsq+0x108/0x1a0 [ 13.414111] but there are no more locks to release! [ 13.414176] [ 13.414176] other info that might help us debug this: [ 13.414258] 1 lock held by kworker/1:1/80: [ 13.414318] #0: ffff8b66feb41698 (&rq->__lock){-.-.}-{2:2}, at: raw_spi= n_rq_lock_nested+0x20/0x90 [ 13.414612] [ 13.414612] stack backtrace: [ 13.415255] CPU: 1 UID: 0 PID: 80 Comm: kworker/1:1 Not tainted 6.13.0-v= irtme #15 [ 13.415505] Workqueue: 0x0 (events) [ 13.415567] Sched_ext: dsp_local_on (enabled+all), task: runnable_at=3D-= 2ms [ 13.415570] Call Trace: [ 13.415700] [ 13.415744] dump_stack_lvl+0x78/0xe0 [ 13.415806] ? dispatch_to_local_dsq+0x108/0x1a0 [ 13.415884] print_unlock_imbalance_bug+0x11b/0x130 [ 13.415965] ? dispatch_to_local_dsq+0x108/0x1a0 [ 13.416226] lock_release+0x231/0x2c0 [ 13.416326] _raw_spin_unlock+0x1b/0x40 [ 13.416422] dispatch_to_local_dsq+0x108/0x1a0 [ 13.416554] flush_dispatch_buf+0x199/0x1d0 [ 13.416652] balance_one+0x194/0x370 [ 13.416751] balance_scx+0x61/0x1e0 [ 13.416848] prev_balance+0x43/0xb0 [ 13.416947] __pick_next_task+0x6b/0x1b0 [ 13.417052] __schedule+0x20d/0x1740 This happens because dispatch_to_local_dsq() is racing with dispatch_dequeue() and, when the latter wins, we incorrectly assume that the task has been moved to dst_rq. Fix by properly tracking the currently locked rq in dispatch_to_local_dsq(). Fixes: 4d3ca89bdd31 ("sched_ext: Refactor consume_remote_task()") Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) ChangeLog v2 -> v3: - keep track of the currently locked rq in dispatch_to_local_dsq() ChangeLog v1 -> v2: - more comments to clarify the race with dequeue - rebase to tip diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 4493a3c419c9..83c9955e87b7 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -2253,9 +2253,11 @@ static void move_local_task_to_local_dsq(struct task= _struct *p, u64 enq_flags, * @dst_rq: rq to move the task into, locked on return * * Move @p which is currently on @src_rq to @dst_rq's local DSQ. + * + * Return the rq where @p has been moved. */ -static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_f= lags, - struct rq *src_rq, struct rq *dst_rq) +static struct rq *move_remote_task_to_local_dsq(struct task_struct *p, u64= enq_flags, + struct rq *src_rq, struct rq *dst_rq) { lockdep_assert_rq_held(src_rq); =20 @@ -2277,6 +2279,8 @@ static void move_remote_task_to_local_dsq(struct task= _struct *p, u64 enq_flags, dst_rq->scx.extra_enq_flags =3D enq_flags; activate_task(dst_rq, p, 0); dst_rq->scx.extra_enq_flags =3D 0; + + return dst_rq; } =20 /* @@ -2387,7 +2391,13 @@ static bool consume_remote_task(struct rq *this_rq, = struct task_struct *p, } } #else /* CONFIG_SMP */ -static inline void move_remote_task_to_local_dsq(struct task_struct *p, u6= 4 enq_flags, struct rq *src_rq, struct rq *dst_rq) { WARN_ON_ONCE(1); } +static inline struct rq * +move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags, + struct rq *src_rq, struct rq *dst_rq) +{ + WARN_ON_ONCE(1); + return src_rq; +} static inline bool task_can_run_on_remote_rq(struct task_struct *p, struct= rq *rq, bool trigger_error) { return false; } static inline bool consume_remote_task(struct rq *this_rq, struct task_str= uct *p, struct scx_dispatch_q *dsq, struct rq *task_rq) { return false; } #endif /* CONFIG_SMP */ @@ -2557,6 +2567,7 @@ static void dispatch_to_local_dsq(struct rq *rq, stru= ct scx_dispatch_q *dst_dsq, { struct rq *src_rq =3D task_rq(p); struct rq *dst_rq =3D container_of(dst_dsq, struct rq, scx.local_dsq); + struct rq *locked_rq =3D rq; =20 /* * We're synchronized against dequeue through DISPATCHING. As @p can't @@ -2593,12 +2604,16 @@ static void dispatch_to_local_dsq(struct rq *rq, st= ruct scx_dispatch_q *dst_dsq, atomic_long_set_release(&p->scx.ops_state, SCX_OPSS_NONE); =20 /* switch to @src_rq lock */ - if (rq !=3D src_rq) { - raw_spin_rq_unlock(rq); + if (locked_rq !=3D src_rq) { + raw_spin_rq_unlock(locked_rq); + locked_rq =3D src_rq; raw_spin_rq_lock(src_rq); } =20 - /* task_rq couldn't have changed if we're still the holding cpu */ + /* + * If p->scx.holding_cpu still matches the current CPU, task_rq(p) + * has not changed and we can safely move the task to @dst_rq. + */ if (likely(p->scx.holding_cpu =3D=3D raw_smp_processor_id()) && !WARN_ON_ONCE(src_rq !=3D task_rq(p))) { /* @@ -2610,8 +2625,8 @@ static void dispatch_to_local_dsq(struct rq *rq, stru= ct scx_dispatch_q *dst_dsq, p->scx.holding_cpu =3D -1; dispatch_enqueue(&dst_rq->scx.local_dsq, p, enq_flags); } else { - move_remote_task_to_local_dsq(p, enq_flags, - src_rq, dst_rq); + locked_rq =3D move_remote_task_to_local_dsq(p, enq_flags, + src_rq, dst_rq); } =20 /* if the destination CPU is idle, wake it up */ @@ -2620,8 +2635,8 @@ static void dispatch_to_local_dsq(struct rq *rq, stru= ct scx_dispatch_q *dst_dsq, } =20 /* switch back to @rq lock */ - if (rq !=3D dst_rq) { - raw_spin_rq_unlock(dst_rq); + if (locked_rq !=3D rq) { + raw_spin_rq_unlock(locked_rq); raw_spin_rq_lock(rq); } #else /* CONFIG_SMP */ --=20 2.48.1