From nobody Wed Apr 1 22:18:44 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=citrix.com); dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=2; a=rsa-sha256; t=1775061382; cv=pass; d=zohomail.com; s=zohoarc; b=Rp1sC8EDuA+qYxdNdqsyYIKkvl5SurPBFAl+E97G41A7ZrrPMjaKOt+6aQ3LrrO/AGi/jCRFPo/O3z7MbhQAGv5dsf4UjunxdRbrxqCmys/IJVMw5Lfq9AWVCI8LLfyOH4Roa5eTe+9vwhVXejglJKt3/yMK0434RqnE8rZcvYA= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1775061382; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Lt7N5PLZbSHwckz9JA+IcupceAjDI4eDaGLo9LKGbGU=; b=J5WagQWMOZryNNdrd3Ac350w8HJx3ViZngW7Byz776qSrI8ifJbPTol50Lq2+ZRLDpFRCmE/IRaX/OV2nKBHU1/GGPmNL97ivv2o1XO7coO5PeCGXBzR+DMRO5+APrIYZD5WrWLG6sMtGWJKDzVRlWLjbPNnR8c3Ni2prYbcyno= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; arc=pass (i=1 dmarc=pass fromdomain=citrix.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1775061382377415.8364039552449; Wed, 1 Apr 2026 09:36:22 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1270702.1559292 (Exim 4.92) (envelope-from ) id 1w7yXj-0007vs-2P; Wed, 01 Apr 2026 16:35:47 +0000 Received: by outflank-mailman (output) from mailman id 1270702.1559292; Wed, 01 Apr 2026 16:35:47 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1w7yXi-0007vl-Sf; Wed, 01 Apr 2026 16:35:46 +0000 Received: by outflank-mailman (input) for mailman id 1270702; Wed, 01 Apr 2026 16:35:45 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1w7yXh-0007iH-8B for xen-devel@lists.xenproject.org; Wed, 01 Apr 2026 16:35:45 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1w7yXg-00CBO7-Il for xen-devel@lists.xenproject.org; Wed, 01 Apr 2026 18:35:44 +0200 Received: from [10.42.69.12] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 69cd4956-2eae-0a2a0a5409dd-0a2a450c89f0-8 for ; Wed, 01 Apr 2026 18:35:44 +0200 Received: from [52.101.43.11] (helo=SJ2PR03CU001.outbound.protection.outlook.com) by tlsNG-d25034.mxtls.expurgate.net with ESMTPS (eXpurgate 4.56.0) (envelope-from ) id 69cd495d-f40c-0a2a450c0019-34652b0b80af-4 for ; Wed, 01 Apr 2026 18:35:44 +0200 Received: from CH8PR03MB8274.namprd03.prod.outlook.com (2603:10b6:610:2ba::5) by BY5PR03MB5112.namprd03.prod.outlook.com (2603:10b6:a03:1e9::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.17; Wed, 1 Apr 2026 16:35:42 +0000 Received: from CH8PR03MB8274.namprd03.prod.outlook.com ([fe80::ebe2:32c1:d2be:a096]) by CH8PR03MB8274.namprd03.prod.outlook.com ([fe80::ebe2:32c1:d2be:a096%7]) with mapi id 15.20.9769.016; Wed, 1 Apr 2026 16:35:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=selector1 header.d=citrix.com header.i="@citrix.com" header.h="From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=IyqCSfvsmZ0Xgo/dh4xVIi64fvzWiRWyLOIGH8y2AMkP+fqgE52qPvGpBwzA8isEcfPqY0eADphTaFkVBfUCwW4OVFj9w1BQ7PqAnmCsK2eLgsBygFSIMhRjOD52E46WH0HHmvjUwmQKOr99zdjsqhk3nal4BTB0j6SXXbu90lGGfLVt+r9hZxS+QFtFsXLHb/X0CgOgcpwuCQLAyefwOVWWmrmOfmTCi104FDbOpGqKd6YPLrjmIJFF/A8/N8/ZRCSxgpMB6T0FM+ub4A/v29oedXv54RRENownBDnwriTeCg67Z9oGXzl2OLfHL9UlpPsvgMQvJq3Q5wKdsYZsYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Lt7N5PLZbSHwckz9JA+IcupceAjDI4eDaGLo9LKGbGU=; b=byKQB08mKvQhj7K89rkbdXzUSB2X/Ge7J+nLgmG2NGJd4++tk5YtCfZHM24ulx+qo495Gh1axQKFHAzasSgQuuw6GkoXgd5iZ57chpNGLCozt34NmmJGxCaw9XfmkgXC/QQjQkChDAG8oCuw6z5CXBES26h4iF9t6MVX6je2dfZg56EYUbuL13Nv06mL1V0JS+6kd8THp1FmVALd2vQAOknCPmJSqZ6sWFUMJbjAD/vRZrJ/jdxI21itzee8VTjc2DpqpeogLqAOeBdyBBMPHIq3A4KPUPRedxAK9TjA+56jEGwcW5FWOn6RqqvFHvIef93s3RB8jhZsSQLa7C84Lw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Lt7N5PLZbSHwckz9JA+IcupceAjDI4eDaGLo9LKGbGU=; b=cL/xWMPJF1vGcsKlZaOZLWAgKpCQ5Is1jtgkcVrMVerbdge+CL3inYIZVfZHJrMvN+tbM56rVP1YG2tVyjpwlxXgQR93UCYF76m82iNPB1MQfU6UgMnWEvy+bBBWasEgNd56Cm6FMCwuVlQuXmgK6EMl2rHAYp1PstXroRr3bcI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com; From: Ross Lagerwall To: xen-devel@lists.xenproject.org Cc: Ross Lagerwall , Andrew Cooper , Anthony PERARD , Michal Orzel , Jan Beulich , Julien Grall , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Stefano Stabellini Subject: [PATCH v1 2/3] xen/smp: Rewrite on_selected_cpus() to be lockless Date: Wed, 1 Apr 2026 17:35:20 +0100 Message-ID: <20260401163521.3603665-3-ross.lagerwall@citrix.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260401163521.3603665-1-ross.lagerwall@citrix.com> References: <20260401163521.3603665-1-ross.lagerwall@citrix.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: LO3P123CA0022.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:388::12) To CH8PR03MB8274.namprd03.prod.outlook.com (2603:10b6:610:2ba::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH8PR03MB8274:EE_|BY5PR03MB5112:EE_ X-MS-Office365-Filtering-Correlation-Id: 83a11c8f-82f4-41be-f416-08de900cb749 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|366016|18002099003|22082099003|56012099003; X-Microsoft-Antispam-Message-Info: mhh8D7Q7XRRFtIsE2w5ExbasElK80qbf/Rw3M3NI7iCtp9cuMHUhp1k8+c91wV5Sf0QfyTfsAZHF8HjuNsl5Hi/BRw9Dbu86/MAXNEKJoDZ7aQornRGcHQRSYtY636DtjBQJ93PXkauBlGXGgc+iWc5z7Xx+IXsDpNZSOlQK5BwCPxXbcoFKnYnDmlpbEv4HdVXQPCccwjv68MqQRKMBmVshlDGith0yyHrdFIUTmPbURBTk+j32FXTEo5KQoIrP3MwRtUw983z9xe0I0usMgj7bitN8U+XHuR2MZOol+bWsai/4Iew0G3/lt8jpYACpTW056H62vUZqt67vM/SXJmV9QA0s9Cwj1cTCMkQNE9WCqoetNzCTCPZAqznwybhsp1ihczETVTozGxoqeEuALLqdVt8/fAsmUKQRQIQU1s9Vn1Gv674VzJbDd5QCUBdX7XytdY2QI9qlQ1UeLzqJRAyuWo1wqFEzi9d++46iFNRK+rNmk8Hskj4RkNzAM9GGNDV2azQXx96cxVy4XVhpmc3ikaPgH/Ipsj9knr7hebgB93YX2j2D12Jwm3qiy2LXMe6m2z0cJwdZyOkYbGchTimu2jO6r2vsrhCdYvNw5KkxJDL5U2297IGoOXOo7N06DngTorwwvxMgeqRijqpPnoQueXY3/kyIovM6cB6gVsgQtrkL3V54+DM3Qr0G1id/Dd3rZVfE1yaYbligLazzDdFRH+0Al2zq33UPgzvGD18= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH8PR03MB8274.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(1800799024)(366016)(18002099003)(22082099003)(56012099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?MsF+4+Zom55Tp3MFMKaI146ZBKSYcMyrVG1enfVb+kn0SvmCHbVF60DTk8Bl?= =?us-ascii?Q?8BLUbAqk4HzKMzfsG02zJ4vUar7Ore2Dd+9LT3ivf2qJlb67sFTyl5g+3T5F?= =?us-ascii?Q?XaDJbGaNXpKZKJA3V5FhGMnbcUcSv5uDzOzcXO+dxGF8DfEVFVyQi3BJ3X/6?= =?us-ascii?Q?mZOUrrzDeQihvxwi1awS0OVsFNUyFgCt8bvpCXzyV+GmXhX44j1NOlbWxaYL?= =?us-ascii?Q?ihCxlCwrH73K2MKLz5e5PlhIx5XmYojIylkixV3zNBkz36Wqrx6pvi1huajp?= =?us-ascii?Q?b/B3UJvAY8ge55LchKJTjCpmrsPaXvCBXqX/VNrVT9u6WXrj9i5M1BUQ/Pzz?= =?us-ascii?Q?GkbRZWfzNw7vDw5yDRZu7XFPPItQIm2hQa3bW30CdOhy58iKawnGnac3PvhZ?= =?us-ascii?Q?nC1iiFdQXqtiLS75E4R7oU0hOCp2pQ8E6SEdxaCnq+nfZx2mbiSBg6VFn76h?= =?us-ascii?Q?3mmbIewHtBdvKVKWjeKWoiaFpi8heSlpr+oikbPPMgtKoNBLwZB77ShZpuAP?= =?us-ascii?Q?TIzsdk7W7aON+TKG2WPjkkj/v4JbPP00WGT/Rnsfpp+0XEHIbRlxh/GLZQAZ?= =?us-ascii?Q?Kh4yP/qlh4s8LCCYrWFnRHzLaLkX3S18smP4lvU+mUtXI6/7N45VME3hXcRj?= =?us-ascii?Q?5oJEJY5lDWpoPFk4kx8L62avUJ5UbXbdmPyE3CyhZIHk2jfx8rZ4TblBojMF?= =?us-ascii?Q?fvmwj7Jc7sNKwnm/licuEhZB8a35DJvFWE9YEsipMeHBuofRrdNMn/Vqcrm0?= =?us-ascii?Q?2JN0VbD0h441IsSvgxCoyaR58Mr4dkeEx/XfswTZKaNweSW+nz9amGVQ3jDU?= =?us-ascii?Q?MiJIarlXtJ9lvyAmlpqahSI3XgdGUhqhihWhM9JZ3QYfZ+wQu3MwRDB+zjiD?= =?us-ascii?Q?UmTQQUzz0C2mrPuhz8KIWZcLMBVC08dRb+bfZ28pqQInFNwpJwszflE8JSrO?= =?us-ascii?Q?2hl+nsEfZzKeP4STPNJtrhJPGoD/doFiTEMU1YgiV0kJnlsqm7xfRMTlXSi/?= =?us-ascii?Q?6CMhqs81F3I/JvzH9bYFL/Ty/nPTEszduwbwzITaf79fXc9hFO/4XNjH9FDU?= =?us-ascii?Q?heEqbtus0YlTsVK4QKIORdAr0Hr0icb0MNJWGbZTf2+1VOI0TuRCdTseUSxk?= =?us-ascii?Q?HXHYMZc6vMkwD9m/ZFl+XsUrTR8C/l5rGfvSwh95QQgnW7YM1TbRv+tRDWu+?= =?us-ascii?Q?kDoEXhI6rFL/IjojXKE8YJSjF3Ulap1OtxYjh4XtJUxPH31EVYGnpgYNLElA?= =?us-ascii?Q?DlTfOPm6b+NkXC5JRGrfMJXG1Ktam7GIaEKBroNMJAqiaEWbGo7swkzJwulh?= =?us-ascii?Q?kCEAoTqYW49TM3rTVL5pEc29MR3mGxNEqICUAGJ/x/yfDcucEXYx9ogHJTZD?= =?us-ascii?Q?1JWTBYI7ESMRdvZDTKT/UVeBbCT6+NiR8vcPdX08MY5VtkJg1pndx3pQPi+q?= =?us-ascii?Q?OthqkSAQf4PmnQcCgsAZgwyrZl+FT36cGpxVe/kaDTjZJCmtAJh8250x9kYh?= =?us-ascii?Q?BhOQZj5DeVUcyzj1sjjHOwwCN5ikHaQqttwvZexLOKc9P06SokkBoJTtYx8x?= =?us-ascii?Q?zlyDkD+sr3OHqYaETgSdMGrbH2koqQHjmduwInNKBwyzqsnRg0KWEvfsFQrE?= =?us-ascii?Q?NKTKMfRLmsJrNKywHXvtb0nIJnUbjrofls1vkT7j8dznobWscpI3NdOhWEwO?= =?us-ascii?Q?t3/Rf3XX8EEx/O8v0c/YCoKlqw9UeeBk0gWKd9cVkTtakB/peR5T2AL+602t?= =?us-ascii?Q?HahatwnA65eSMZ5YSf0rTWmg8seAjmE=3D?= X-OriginatorOrg: citrix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 83a11c8f-82f4-41be-f416-08de900cb749 X-MS-Exchange-CrossTenant-AuthSource: CH8PR03MB8274.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Apr 2026 16:35:42.3955 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Y6E62ek+7GeQcY0iE44JSyWozi/wBFhxABKOnD8vhbxHw7dGRHDpSzFB5cJBwS9PNYxz686GKUGmNGSOfr7Cb09dB8qbWuTeJm28nq2hMRk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR03MB5112 X-purgate-ID: tlsNG-d25034/1775061344-A49BFA3D-A2293BCB/0/0 X-purgate-type: clean X-purgate-size: 4313 X-ZohoMail-DKIM: pass (identity @citrix.com) X-ZM-MESSAGEID: 1775061385050158500 Content-Type: text/plain; charset="utf-8" on_selected_cpus() holds a global lock even if the function is to be called on non-overlapping CPUs. This is a scalability bottleneck so to avoid that: 1. Remove the global lock. 2. Make call_data_struct per-CPU. 3. Track which CPUs are currently running on_selected_cpus() using a global CPU mask. This tells CPUs running the interrupt which per-CPU call_data_structs to look at. Since the call data is now per-CPU, skip waiting for CPUs to "check in" for async calls. Instead, delay it until the next time on_selected_cpus() is called by which point there should be nothing to wait for. Signed-off-by: Ross Lagerwall --- xen/common/smp.c | 101 +++++++++++++++++++++++++++++++---------------- 1 file changed, 66 insertions(+), 35 deletions(-) diff --git a/xen/common/smp.c b/xen/common/smp.c index a011f541f1ea..e592e8453fb3 100644 --- a/xen/common/smp.c +++ b/xen/common/smp.c @@ -24,13 +24,15 @@ /* * Structure and data for smp_call_function()/on_selected_cpus(). */ -static DEFINE_SPINLOCK(call_lock); -static struct call_data_struct { +struct call_data_struct { void (*func) (void *info); void *info; int wait; - cpumask_t selected; -} call_data; + cpumask_t selected __cacheline_aligned; +}; + +DEFINE_PER_CPU(struct call_data_struct, call_data); +static cpumask_t tasks; =20 void smp_call_function( void (*func) (void *info), @@ -50,55 +52,84 @@ void on_selected_cpus( void *info, int wait) { + struct call_data_struct *data; + unsigned int cpu =3D smp_processor_id(); + ASSERT(local_irq_is_enabled()); ASSERT(cpumask_subset(selected, &cpu_online_map)); =20 - spin_lock(&call_lock); + if ( cpumask_empty(selected) ) + return; + + data =3D &this_cpu(call_data); =20 - cpumask_copy(&call_data.selected, selected); + if ( !data->wait ) + { + /* Wait for any previous async call to complete */ + while ( !cpumask_empty(&data->selected) ) + cpu_relax(); + + cpumask_clear_cpu(cpu, &tasks); + } =20 - if ( cpumask_empty(&call_data.selected) ) - goto out; + data->func =3D func; + data->info =3D info; + data->wait =3D wait; =20 - call_data.func =3D func; - call_data.info =3D info; - call_data.wait =3D wait; + smp_wmb(); =20 - smp_send_call_function_mask(&call_data.selected); + cpumask_copy(&data->selected, selected); =20 - while ( !cpumask_empty(&call_data.selected) ) - cpu_relax(); + cpumask_set_cpu(cpu, &tasks); =20 -out: - spin_unlock(&call_lock); + smp_send_call_function_mask(&data->selected); + + if ( wait ) + { + while ( !cpumask_empty(&data->selected) ) + cpu_relax(); + + cpumask_clear_cpu(cpu, &tasks); + } } =20 void smp_call_function_interrupt(void) { - void (*func)(void *info) =3D call_data.func; - void *info =3D call_data.info; unsigned int cpu =3D smp_processor_id(); - - if ( !cpumask_test_cpu(cpu, &call_data.selected) ) - return; + unsigned int i; + struct call_data_struct *data; + void (*func)(void *info); + void *info; =20 irq_enter(); =20 - if ( unlikely(!func) ) - { - cpumask_clear_cpu(cpu, &call_data.selected); - } - else if ( call_data.wait ) - { - (*func)(info); - smp_mb(); - cpumask_clear_cpu(cpu, &call_data.selected); - } - else + for_each_cpu ( i, &tasks ) { - smp_mb(); - cpumask_clear_cpu(cpu, &call_data.selected); - (*func)(info); + data =3D &per_cpu(call_data, i); + + if ( !cpumask_test_cpu(cpu, &data->selected) ) + continue; + + smp_rmb(); + func =3D data->func; + info =3D data->info; + + if ( unlikely(!func) ) + { + cpumask_clear_cpu(cpu, &data->selected); + } + else if ( data->wait ) + { + (*func)(info); + smp_mb(); + cpumask_clear_cpu(cpu, &data->selected); + } + else + { + smp_mb(); + cpumask_clear_cpu(cpu, &data->selected); + (*func)(info); + } } =20 irq_exit(); --=20 2.53.0