From nobody Mon Feb 9 13:35:55 2026 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2055.outbound.protection.outlook.com [40.107.94.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B32561FFE for ; Sat, 25 Jan 2025 09:17:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.55 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737796626; cv=fail; b=cPSAA/pcSKSY76C+HQSdbNHB9e3gfsPvJd09Ehq5GEJub5SlB7GNoBO/tJ8KI9luy61hp+Wv+Fz4Wuers3+hlG1Ex3iGppq/DFZnYuxVZN6WjcXg+KQm6ZmqRAJVtBDJJoID11ZOSsBB8BuseUlRO30R+yIcKoa8TnJTd3pQvJs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737796626; c=relaxed/simple; bh=9fft6fn7/l1pbrUjNJ4P7wkRdK0u/L8GgPl52jDr6U4=; h=From:To:Cc:Subject:Date:Message-ID:Content-Type:MIME-Version; b=nY4WjbxvFXIc1UfIpk3awWS1DrAyXD7OM1VlORZzbQ9BU8/lzfFDHAV1yXP1DOnybSLi396o/+TT/KJYYXmoCfV6gsTK7J1IbHw1vkz/HmbpB6UDPppCnS+ttZ907ediERJF6uk+OscXPH6vsPk/VStJWdLIXinT7YXDpjtRSVQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=c6s4re7T; arc=fail smtp.client-ip=40.107.94.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="c6s4re7T" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=w5TooQVpgalfcm5XXOM5uG/3luf2JJf2GAbx6Zaexk0hJh8oBWdnaZM/jDx/5qsYQAMZJRlNfHO35NOV1E7fyqjhuW80VRpEsK1gmE2XU7v+CwvQOseuxkpjCYBcLheXK6Ugyr5ljKeFXNuN7t3caASWd5fgpFdqktEaJftHrooUMaQ0JD+PqTSx0YBB+N7j3xxH/zqZvMmjmSPji2LheSYREE7KIVzYzih7pchbWsQBxw3RUYpYi2y0qh2u5DtMsyWIPbUjKpFDfvTEEU3lSepRI72Ou1XW7S0Emz48VP5hbqwaoHUJdOmcB4/suEoJtTmk0I6pVtQhl9XPU5Pq5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ryh9qW3pyyrTZ6oLbwJzXx2U0QHSDkgrmy9u9Vbo7IY=; b=OfPMHNf0rsDI9docZKu4EmsS0xnQpujX+vX1dirfpEo8sPe9rEfzw2y89ChuSDdB9IWeIS8RvkMnmjuUO5ulCofHAeykG2ml1LLOgxTAjRTEKia/AarBCAWHDAcqEoGFxlZZpSawPZVUjDQjbVcDQBrRbP+JERmpt4ANkjKgftP+hfwI7jQH/sAKoNl7aGiX9niVA7NlKXzqLFXk+pZRAvTSMoAYaUSxSfX8uISyx33udv/A0ZWGyW28vMmrEqCfWONOQuaqLMuXQIWAke83YXzVbSsAK2oxPSvGCJ+3VpikX+T9U6gWrgHeQ5SqXhkzoljKxPa+NdtPx+I1z3Nshg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ryh9qW3pyyrTZ6oLbwJzXx2U0QHSDkgrmy9u9Vbo7IY=; b=c6s4re7TgofXJkPL/SJX4Y+KVUDEa37VajBHCh8TTVDmqcp85L/EsE393r3Hy6jTsvfXjQQEBXhozuTM9om+wlB/yf2yIeRB6IM9YTXZdWPurouNrocnL+uPEI6xr4LOLrTjndxBMu+kxBcykqIyQsabAUlFh/adbw/Hptv+rTGDFNW31PWeY1oleKfqRDci/GWvNJv0GUcZbEZLPqQOxZuZsDeypRd5FI1/j33/pTXpyNUMWVOkpaUviOeHz6zoRsG7UmrC1NbchN6bXd4n03ANaYtgtkBYpRxK9f425lhH6epzOeXPw7dbZw6fY7SVKTy76ydzCYxSXNCraitA1g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SA3PR12MB7830.namprd12.prod.outlook.com (2603:10b6:806:315::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8356.21; Sat, 25 Jan 2025 09:17:01 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%4]) with mapi id 15.20.8377.009; Sat, 25 Jan 2025 09:17:00 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: linux-kernel@vger.kernel.org Subject: [PATCH v4] sched_ext: Fix lock imbalance in dispatch_to_local_dsq() Date: Sat, 25 Jan 2025 10:16:57 +0100 Message-ID: <20250125091657.203445-1-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: FR0P281CA0144.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:96::19) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SA3PR12MB7830:EE_ X-MS-Office365-Filtering-Correlation-Id: 94da372d-f7c5-43b9-d2ca-08dd3d210659 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?QA13jwQAHY5YPIM6rejWn4Vk41y3bXkvbPAS450UP2cwMP6GMic2zuSbcFqU?= =?us-ascii?Q?a0gBQ3+I0HogyeR206JFFeFxcIUJb3PJTQQETEKPN6vNfgI8H0j/WoaSgARl?= =?us-ascii?Q?Riwgc2VQ/DbdLCjt1B4L6iH3ESQTp0z3h7iivGwdRbmwwya7UZvacCCcDaHX?= =?us-ascii?Q?Y5JjTWyJ9qEEehvIx/UTPtSRTCes9U3nIktT0u23S8AYDE7vksqO8/Q6c4Sv?= =?us-ascii?Q?kVBl7hr2XmtXspn6W06dOXNy3BlVSyp2tBa5TkTVg6F3HJ3zpaf04Z8WOIjh?= =?us-ascii?Q?HHyRusiQ2os930HSek7jZhsjbNT8sJNohrjx+FpEEVV4PCr3X51jycqrawQT?= =?us-ascii?Q?jfiEmBXyWJPxa106pp7hh0q+AUV1YUEK+OHIJSLKJa38SDpDftEd1bSuzasy?= =?us-ascii?Q?K/Qjr3Zptcyn+dsz/3UsblOUFm6BHN0pRZaAeaHBXpfxEjzwzyWCeLZEIXAY?= =?us-ascii?Q?qSXwA4DkXQRWRGE/zAQKON/xg+uOYpWpVmcMCIDlnr+UEJlOkbSWb4Dl88Lj?= =?us-ascii?Q?5IIgWMgPQGFWK4u5I5Bdqre3/RkRz2MOjF/a13u7unmrlO6XlrarH1dKnJlQ?= =?us-ascii?Q?intQOEcEP3JbOL0AZEp0e8nisl1PS9Y5T9x/45thZTDzKZXokp4vO/CfHeC+?= =?us-ascii?Q?hyMjy7j34oNVa2IDaXIbze/uRj2lIlSV9c5PVtYdjQKktmG9YMofT3086Ay0?= =?us-ascii?Q?CnPdzTo19i2xQB4xWlWunXpSe4AFyd+MP9W2tJBu681tUJD69L7b3W5S887N?= =?us-ascii?Q?RL18zZxRakA/P7r4bAvKBLK2cDHhT4JbXNsjn/xZ88SJPamBzwhrbptgSZ0f?= =?us-ascii?Q?D1BCl25Zxzw6gcg/wzBqO8UPn9wlfGquDDW0pRvnDz2SheIPMLryao6iXiJj?= =?us-ascii?Q?M1JXA2YS8UY819ZH4kJQculSSw0UJEItBkNYStB+hRE/olbpzSdMsq/12aNL?= =?us-ascii?Q?dJfCnXZx6wLkkxj+peCwaSqMKi+MN5KNK9OlY0X+JQKqyIScxkKmFs8ZRPN5?= =?us-ascii?Q?17V8fmTlDZXD7HcOoHXbBBv9a1LGOcsjj8Sq9Sb5sY6D9GmcV5O9+jvfLH6l?= =?us-ascii?Q?8y6rPYKmzHEsbC2zV8X4fGot4i+Tm8LGYgwuGwvX3dfBEZJu8Uymbhin8mmu?= =?us-ascii?Q?gVwRLSTnPpQEnq3c3wK1+Sr8ZN8QMOoso5ImGYLnaTWDdjbuV0k9RlIYR1qu?= =?us-ascii?Q?MTGxezoqGpPWR4flxQzhItv3LYbD/YUu9fgZDVH4l3XbDwRNx1+LjE2nQpd0?= =?us-ascii?Q?4+RheBEBDZvj1e+x4ajSGXDu9KEY08Ibie9QWcNbQZEMjvLubOeqqjQa1ho4?= =?us-ascii?Q?695UpyxoG+53moxI39LcYyu3Tn/VhLnoLrHtV5muqZa/3FCP7rf/lFxhOmqu?= =?us-ascii?Q?Za5DyLnl/1umm4x5z2vNePqpcrxT?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Yp80PnGy9wlxy6IuOTFswo0ctG6Kc7xSg891W6rZwJbnVM+Xv89OSX+DfRe4?= =?us-ascii?Q?/VMZBwBljQbnX4WWZYi5MympF9kDpO22aibdpg7lQSF/z2HO8V/bU3CH9Z/M?= =?us-ascii?Q?MYv8aH8LzCbIKcibKXZZD7m6bxhb6/qkVJVmx1eJ88rppbxd87x5Fv8/7NVg?= =?us-ascii?Q?Y9byGa0qOL8wF//Z2qe5TZ+giEfO4sT3cFpOd531fnuhn/7wlAZWt61F2OMM?= =?us-ascii?Q?UyTmFnmnzQeklHo1/7Up5zbnBdaYgCEDjhvbPUYgKJtBsNbSBBdeUTaUFAww?= =?us-ascii?Q?mWjQlNgFKrJBVyoAeN42BgYsZK0ehPdR8NNXvpQL+6COPVGwyfXcssNWuqap?= =?us-ascii?Q?DCMatgWuxoQc07Lf1G/5gPpQOHsdr/wS77J48imy3H/63cZtSZDQ7KHHp+0L?= =?us-ascii?Q?YsCxwjb9vbbeGZVHYDjRu3T5iBTxkDygy7aCqCI8vIDGu1WV339bteNM80RR?= =?us-ascii?Q?RSb52KvhP2maoD02Sw0BRxWegr4y01xJrJe0AyClyHgXn4EwSw1wNtN4wr74?= =?us-ascii?Q?Le9y/cq64uqjUcOBbcC26WlJRbwFlfuoOJdm7f+LNr+vxlheej0MXbE8vevs?= =?us-ascii?Q?k+hk7VIDaxZ6kZ/J4tI0FAmqRQ9/f5TK87TpSROKflXLERMpyGTMqOAG8IrL?= =?us-ascii?Q?b8FeDP1IIb+7uFTy9cm17nFbXuF/I4omusVQM2WbGCYCej9tU7LX7RCvURl2?= =?us-ascii?Q?euge4edmLw2xFDLvbuzYpD2xJDm+qMOpO4l+rfY6lcRZBTJeezdd0abXqF5x?= =?us-ascii?Q?+2798N2zxoHATaRh7wljhE2MpNHMyZFbVWcLU31hE3j6tL7bxaPfh3PE/nDt?= =?us-ascii?Q?fwe6nfXLyS/vOO3YyGgOfYiEi1ZZyPvYof5x8hkKDr/2lNnUpiEFjywvNpwq?= =?us-ascii?Q?VE+s+k3o2ffrPCaekoHbgyr8yr2dxY5YwA7n0VuZmgRmug9lJTmU7S3eU2lv?= =?us-ascii?Q?Uc0TmEz4B+iUat+GVguBonaEoqahuyLzAoRvosv7q49SWBsYV+NvWphXcjjO?= =?us-ascii?Q?EMWau0S4mUwwniGYdRhgC0BfWli/lVuj7RTfCdTrp3s770cB81pi27Iug/Gs?= =?us-ascii?Q?bZGxvi4PrgKabfiZp3TUypdhT5rtOC2H1tWqG2UVnk4DwRe45+gUGuTo6GuJ?= =?us-ascii?Q?hOKb+VBTmmxB8ZHfmXmIdhlLKyCD8IYZLap0Vrn/viE6xCLem7iFLbR2LAlj?= =?us-ascii?Q?3QQR7EADMOOBGP7ubJDroYIwshcmyjjbf1bgdmLYwC+X2Pznvh8+wgSn9FLt?= =?us-ascii?Q?lpxyCNpDm3NwtT/Lpy8KbJLGAAY5B83J+XAtTzAt2in6/N10v/6jyboEHz/f?= =?us-ascii?Q?4t5z2/iCY7uPb7ys37vOcezYd4g07aswExB+w4rBdjvmVmetpZokj/4ietr1?= =?us-ascii?Q?JffzpJC9p7spJdR+N9OxmzhC1J07uZMHHLsawlp0Vkwmge7v1YFTUZbHNa9a?= =?us-ascii?Q?tCul5XeIsL6lKQFqhyVwwkBWQF9tuYfE9zxl/1mkY5dIyRUS0hWj2WL1MCax?= =?us-ascii?Q?Cej5olsxV1trgS/j7JPHoJrM31DnpxzFtJOHDvYg1LCTML6hKCrdstfRHjqK?= =?us-ascii?Q?XAfjGoXPRWX00HX8sJV8dVvwMeXrdjzF8ljtWfXS?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 94da372d-f7c5-43b9-d2ca-08dd3d210659 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2025 09:17:00.8508 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wa676L9OzPkaEpONGKGsEXyCgtr6JGNgxc64QbsPSzi88nrhEmuxdwYpDQ6Lkh/jCONWbIGfSoyHUzlYnpJqDg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7830 Content-Type: text/plain; charset="utf-8" While performing the rq locking dance in dispatch_to_local_dsq(), we may trigger the following lock imbalance condition, in particular when multiple tasks are rapidly changing CPU affinity (i.e., running a `stress-ng --race-sched 0`): [ 13.413579] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [ 13.413660] WARNING: bad unlock balance detected! [ 13.413729] 6.13.0-virtme #15 Not tainted [ 13.413792] ------------------------------------- [ 13.413859] kworker/1:1/80 is trying to release lock (&rq->__lock) at: [ 13.413954] [] dispatch_to_local_dsq+0x108/0x1a0 [ 13.414111] but there are no more locks to release! [ 13.414176] [ 13.414176] other info that might help us debug this: [ 13.414258] 1 lock held by kworker/1:1/80: [ 13.414318] #0: ffff8b66feb41698 (&rq->__lock){-.-.}-{2:2}, at: raw_spi= n_rq_lock_nested+0x20/0x90 [ 13.414612] [ 13.414612] stack backtrace: [ 13.415255] CPU: 1 UID: 0 PID: 80 Comm: kworker/1:1 Not tainted 6.13.0-v= irtme #15 [ 13.415505] Workqueue: 0x0 (events) [ 13.415567] Sched_ext: dsp_local_on (enabled+all), task: runnable_at=3D-= 2ms [ 13.415570] Call Trace: [ 13.415700] [ 13.415744] dump_stack_lvl+0x78/0xe0 [ 13.415806] ? dispatch_to_local_dsq+0x108/0x1a0 [ 13.415884] print_unlock_imbalance_bug+0x11b/0x130 [ 13.415965] ? dispatch_to_local_dsq+0x108/0x1a0 [ 13.416226] lock_release+0x231/0x2c0 [ 13.416326] _raw_spin_unlock+0x1b/0x40 [ 13.416422] dispatch_to_local_dsq+0x108/0x1a0 [ 13.416554] flush_dispatch_buf+0x199/0x1d0 [ 13.416652] balance_one+0x194/0x370 [ 13.416751] balance_scx+0x61/0x1e0 [ 13.416848] prev_balance+0x43/0xb0 [ 13.416947] __pick_next_task+0x6b/0x1b0 [ 13.417052] __schedule+0x20d/0x1740 This happens because dispatch_to_local_dsq() is racing with dispatch_dequeue() and, when the latter wins, we incorrectly assume that the task has been moved to dst_rq. Fix by properly tracking the currently locked rq. Fixes: 4d3ca89bdd31 ("sched_ext: Refactor consume_remote_task()") Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 110 ++++++++++++++++++++++++++++----------------- 1 file changed, 68 insertions(+), 42 deletions(-) ChangeLog v3 -> v4: - small refactoring to fix an unused variable build warning on UP ChangeLog v2 -> v3: - keep track of the currently locked rq in dispatch_to_local_dsq() ChangeLog v1 -> v2: - more comments to clarify the race with dequeue - rebase to tip diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 4493a3c419c9..a290052478ed 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -2253,9 +2253,11 @@ static void move_local_task_to_local_dsq(struct task= _struct *p, u64 enq_flags, * @dst_rq: rq to move the task into, locked on return * * Move @p which is currently on @src_rq to @dst_rq's local DSQ. + * + * Return the rq where @p has been moved. */ -static void move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_f= lags, - struct rq *src_rq, struct rq *dst_rq) +static struct rq *move_remote_task_to_local_dsq(struct task_struct *p, u64= enq_flags, + struct rq *src_rq, struct rq *dst_rq) { lockdep_assert_rq_held(src_rq); =20 @@ -2277,6 +2279,8 @@ static void move_remote_task_to_local_dsq(struct task= _struct *p, u64 enq_flags, dst_rq->scx.extra_enq_flags =3D enq_flags; activate_task(dst_rq, p, 0); dst_rq->scx.extra_enq_flags =3D 0; + + return dst_rq; } =20 /* @@ -2387,7 +2391,13 @@ static bool consume_remote_task(struct rq *this_rq, = struct task_struct *p, } } #else /* CONFIG_SMP */ -static inline void move_remote_task_to_local_dsq(struct task_struct *p, u6= 4 enq_flags, struct rq *src_rq, struct rq *dst_rq) { WARN_ON_ONCE(1); } +static inline struct rq * +move_remote_task_to_local_dsq(struct task_struct *p, u64 enq_flags, + struct rq *src_rq, struct rq *dst_rq) +{ + WARN_ON_ONCE(1); + return src_rq; +} static inline bool task_can_run_on_remote_rq(struct task_struct *p, struct= rq *rq, bool trigger_error) { return false; } static inline bool consume_remote_task(struct rq *this_rq, struct task_str= uct *p, struct scx_dispatch_q *dsq, struct rq *task_rq) { return false; } #endif /* CONFIG_SMP */ @@ -2538,38 +2548,12 @@ static bool consume_global_dsq(struct rq *rq) return consume_dispatch_q(rq, global_dsqs[node]); } =20 -/** - * dispatch_to_local_dsq - Dispatch a task to a local dsq - * @rq: current rq which is locked - * @dst_dsq: destination DSQ - * @p: task to dispatch - * @enq_flags: %SCX_ENQ_* - * - * We're holding @rq lock and want to dispatch @p to @dst_dsq which is a l= ocal - * DSQ. This function performs all the synchronization dancing needed beca= use - * local DSQs are protected with rq locks. - * - * The caller must have exclusive ownership of @p (e.g. through - * %SCX_OPSS_DISPATCHING). - */ -static void dispatch_to_local_dsq(struct rq *rq, struct scx_dispatch_q *ds= t_dsq, - struct task_struct *p, u64 enq_flags) +#ifdef CONFIG_SMP +static void move_to_remote_dsq(struct rq *rq, struct rq *src_rq, struct rq= *dst_rq, + struct task_struct *p, u64 enq_flags) { - struct rq *src_rq =3D task_rq(p); - struct rq *dst_rq =3D container_of(dst_dsq, struct rq, scx.local_dsq); + struct rq *locked_rq =3D rq; =20 - /* - * We're synchronized against dequeue through DISPATCHING. As @p can't - * be dequeued, its task_rq and cpus_allowed are stable too. - * - * If dispatching to @rq that @p is already on, no lock dancing needed. - */ - if (rq =3D=3D src_rq && rq =3D=3D dst_rq) { - dispatch_enqueue(dst_dsq, p, enq_flags | SCX_ENQ_CLEAR_OPSS); - return; - } - -#ifdef CONFIG_SMP if (unlikely(!task_can_run_on_remote_rq(p, dst_rq, true))) { dispatch_enqueue(find_global_dsq(p), p, enq_flags | SCX_ENQ_CLEAR_OPSS); @@ -2593,12 +2577,16 @@ static void dispatch_to_local_dsq(struct rq *rq, st= ruct scx_dispatch_q *dst_dsq, atomic_long_set_release(&p->scx.ops_state, SCX_OPSS_NONE); =20 /* switch to @src_rq lock */ - if (rq !=3D src_rq) { - raw_spin_rq_unlock(rq); + if (locked_rq !=3D src_rq) { + raw_spin_rq_unlock(locked_rq); + locked_rq =3D src_rq; raw_spin_rq_lock(src_rq); } =20 - /* task_rq couldn't have changed if we're still the holding cpu */ + /* + * If p->scx.holding_cpu still matches the current CPU, task_rq(p) + * has not changed and we can safely move the task to @dst_rq. + */ if (likely(p->scx.holding_cpu =3D=3D raw_smp_processor_id()) && !WARN_ON_ONCE(src_rq !=3D task_rq(p))) { /* @@ -2610,8 +2598,8 @@ static void dispatch_to_local_dsq(struct rq *rq, stru= ct scx_dispatch_q *dst_dsq, p->scx.holding_cpu =3D -1; dispatch_enqueue(&dst_rq->scx.local_dsq, p, enq_flags); } else { - move_remote_task_to_local_dsq(p, enq_flags, - src_rq, dst_rq); + locked_rq =3D move_remote_task_to_local_dsq(p, enq_flags, + src_rq, dst_rq); } =20 /* if the destination CPU is idle, wake it up */ @@ -2620,13 +2608,51 @@ static void dispatch_to_local_dsq(struct rq *rq, st= ruct scx_dispatch_q *dst_dsq, } =20 /* switch back to @rq lock */ - if (rq !=3D dst_rq) { - raw_spin_rq_unlock(dst_rq); + if (locked_rq !=3D rq) { + raw_spin_rq_unlock(locked_rq); raw_spin_rq_lock(rq); } -#else /* CONFIG_SMP */ - BUG(); /* control can not reach here on UP */ +} +#else /* CONFIG_SMP */ +static void move_to_remote_dsq(struct rq *rq, struct rq *src_rq, struct rq= *dst_rq, + struct task_struct *p, u64 enq_flags) +{ + BUG(); /* this should never be called on UP */ +} #endif /* CONFIG_SMP */ + +/** + * dispatch_to_local_dsq - Dispatch a task to a local dsq + * @rq: current rq which is locked + * @dst_dsq: destination DSQ + * @p: task to dispatch + * @enq_flags: %SCX_ENQ_* + * + * We're holding @rq lock and want to dispatch @p to @dst_dsq which is a l= ocal + * DSQ. This function performs all the synchronization dancing needed beca= use + * local DSQs are protected with rq locks. + * + * The caller must have exclusive ownership of @p (e.g. through + * %SCX_OPSS_DISPATCHING). + */ +static void dispatch_to_local_dsq(struct rq *rq, struct scx_dispatch_q *ds= t_dsq, + struct task_struct *p, u64 enq_flags) +{ + struct rq *src_rq =3D task_rq(p); + struct rq *dst_rq =3D container_of(dst_dsq, struct rq, scx.local_dsq); + + /* + * We're synchronized against dequeue through DISPATCHING. As @p can't + * be dequeued, its task_rq and cpus_allowed are stable too. + * + * If dispatching to @rq that @p is already on, no lock dancing needed. + */ + if (rq =3D=3D src_rq && rq =3D=3D dst_rq) { + dispatch_enqueue(dst_dsq, p, enq_flags | SCX_ENQ_CLEAR_OPSS); + return; + } + + move_to_remote_dsq(rq, src_rq, dst_rq, p, enq_flags); } =20 /** --=20 2.48.1