From nobody Sat Jul 25 16:48:31 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11020135.outbound.protection.outlook.com [52.101.61.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A652C4D8DA1 for ; Wed, 15 Jul 2026 18:05:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.135 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138755; cv=fail; b=ZQ9Lh5baaE5lHPbXohxWaJ8+CFmPcJ9p+SKy8F+R/N4ehVbH9CiO3naaT/P9MQ5tkITnLsWQSR0hKa32wA8sGBp6vaWpmsmL3VB1xVEDKkeyhXKsttCo+mFsGEp4AMIv/ZAe5FASKp/Vy0qxvTCDrYV4gnntJhxbFGDr8CM9C1U= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138755; c=relaxed/simple; bh=9PAmADfEDC2QVjj1UpP1nSpFGF2fmM6e/8d2RNQCYpw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Jc8PEE2wbklgFSmYpUsBI0xJwaLXHQR0DEuBxZq3jzGnUjp00jU+TZ5utCQWOcQGS9yZqsWwCmlii0mA3gKSKU2ssfBcHJ/pCy7cd53eoXTKHwF8DuUI5ivqNmlPOVmMrfpq2fdV0DavOYv97ADNDZuA30KOlMDmDuFgCA7z0a4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=bm6EncYm; arc=fail smtp.client-ip=52.101.61.135 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="bm6EncYm" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=babKOyGQQMgbxo91XOOC/3fzWzL/e6dYwH4OmlIYcjd4h/s67zWFZtf/XkHoMVnzmKZ3XQSQ9LfT/T038TJ3TkO4MbsAY9TmbFhxUnR8kEpLpm0xTFQvDuz+LpY/m6W7+Yk6QpY6Wb6psxAw5IqOYnee8zLPG12RtAczSJelGUt+PWzw82v4OlQMj9obDaX4OpS/NXM3LUGjm5o8RCugRlfAUsbFy55SuWfMU/bNhUDy+sMkkOiDiJs2Cd2Wf/QVG4hBktRbvrtHldIrHM6qhq8L+8cENtDsqEeFiBosj8ljwoDk90FQ/8YPVcPj45Q68/4sxs356MiRoUXfV+yCew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=O95DtUyWMD06oTaLiZwi3Wfcr5BYuVUIdsRaM8li0Hc=; b=toMkkY8IA7+fyKfc5Ue3iVi0IK1A1gCKdgveP8pOCEUFpT+JQYDSrHPBNcOA1J+GSIw0WLCEecu1bpjSaFfvucDewJe8dj0ZocB+ExXCrTNdaviyWGngz/kwveoZC+5nTsbiammauj3U2KiwHg7twYff6anIb2QtOJkiWNioslU5zLoTYFSuiWcJqJKEJq5CmpxVCt0ukdEmxCht+AcJlqpyf1tLvJtoXjthw/F/TPVV6I9wa9TgpmTOcJqL30oHDW6SpJ2wuCK9crqoWXQFrBCer9aTm9I4FfUw/POzi/JVMiUlEsb79JlmGeBgAov/5jVTf6aJxuEZ+6xNkQ6MUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O95DtUyWMD06oTaLiZwi3Wfcr5BYuVUIdsRaM8li0Hc=; b=bm6EncYmjxMGikeRVTIW6ZzqHaAP8h8ZSmZr2Zkdyie3+fiWgymGI7rWyncl0VV2Z1pFph7T9HatiboOajO+wvFYnfTwauJSEQPPnIjbZvg0VaNd6T3vHCTMK2Aix0kOg3y8YodILfX9+Ggc4P6T+Rx04QM3EpquOjwX3HTBRno= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SJ0PR01MB6207.prod.exchangelabs.com (2603:10b6:a03:29d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.12; Wed, 15 Jul 2026 18:05:33 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:33 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/16] drivers: arch_numa: move percpu set up code to arch Date: Wed, 15 Jul 2026 11:04:03 -0700 Message-ID: <20260715180455.515692-2-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SJ0PR01MB6207:EE_ X-MS-Office365-Filtering-Correlation-Id: 77476de8-d454-4c8f-99ad-08dee29ba9df X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|23010399003|1800799024|366016|56012099006|11063799006|55112099003|18002099003|22082099003|10067099003|921020; X-Microsoft-Antispam-Message-Info: D2hNM/4PizZy/K7QwQIL5esPR4UJ0y8ULXnNrJ6lqj0UPwTAK2p4/nDyiKiXHYDsYy2YlIMCj3WgwHa2qn/o5TJ4tYhOGaZE0OZZbBI6TpwmRr8IXZYJaZmXgfJcoDbOdKukIzF3JpL6mHxc1d0ON3fk2x4nIqDSufX8cTIcIGrsy8e/8o1nX+LUXrFH7ONB8DRpP2HVUoQLXBwPJkZEl3NkVVvsf0gpLH587TrWxciKOOSJ16wOH4JB9rKjvdqu/dqThSPRDO0FuNR2oWcPWZZRIsYwgLmhE+YsiSBm+WUVaXE8102dumKXzPp3WDZVpKfmX48kQQY67r2MYvQ2yuVA9AGYLbbKVBhioM3eepT+L3k5uIG7zjL7jxmSOMsjsqIq064JRgGCT3vrtemdZth8+87ZzAlkRq0T5htlIYhBhSBlx6FjJ56iBoNkRoCRJztBzr5urGsZ1oMkHtiucMx7by1pWWqoexZWijmrtZUZzeVIgWQZvvlECJNC4tePtPX+hSRkMXkp/U3Sc0VtxBe+niNH5CvVxPMno3cm1LJJcu3+yQ4H7CrQwX/bpZe5Eb4S7d6Sfvs2UX2VDTzlCE4Nt1Sail2zTOGo0p1BrC9uFf5qft3AlI0iaFzH2UCtUrBNlNj5MQkj1Mxm3v6ts5z8XdUlACsnU0oO3L85nFNp7gHQmUIBOActflC35Eflb6IbSc5q8y5fGrawAnqKQg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(23010399003)(1800799024)(366016)(56012099006)(11063799006)(55112099003)(18002099003)(22082099003)(10067099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?S4MBncjgk2i5Y5IbCFgPPDjipknOA+BYcWeN5SDMMnHlh4og2D3OK23bN9MF?= =?us-ascii?Q?k7wBWAcxU4Q6wdgCkb11+kbN25Bd3Mb3e8MNEo4sskMXkhoy93FgKhSX/eyN?= =?us-ascii?Q?rALLy7EDxKlMq+jJe8f4FhvGED508PnGpql8QYYIUD8OahH66GyNxM8QlLO/?= =?us-ascii?Q?ijlcr0gXlPkm8fLOEtmgs2OtGE7iEfWES4ZAodiv4CLJ118hN8aaYqXrOI3q?= =?us-ascii?Q?m/j8lLjCgYic+gGTGj3T2pTBPVLss9K0RuucO6e3S+Wx9XA+e+wudtIbLgc7?= =?us-ascii?Q?nlTJvPoNADlV9xcihQrO1jahlYpbve2iC5lMFY7b4NyDufhTMbXjPMKtibTC?= =?us-ascii?Q?AaQG8Qr/3B9ia3VfvAOWN7pOK0yRpxdeM8tH4OS0DXcGSspvBNVEK0r/k9S1?= =?us-ascii?Q?v/aUGEJvs6W9ZYb9H0f5Oya3+DVV5NU4uQL0NqaSyJWPZWTdQpr2Usqtr7FE?= =?us-ascii?Q?Q/irTxGbiPyiUsZL4wJGhsY8feJzuOl4q1d3qSwEmprlg+7+nLJNHKjwcOS0?= =?us-ascii?Q?c4Sf7tsCmZPjtP9vNGCekuOfUzc/q3NfKWu2nzh7MG338ZzreUbzjDyJNZV1?= =?us-ascii?Q?rwiMwuu+4eJD0yNSfEtaP2WC6GJm4e/92Ug4KgITI044y3vkf51VTve97Y6A?= =?us-ascii?Q?GVCcUtQDIBHex5jU1olGarWSnMnEDOv3yoorvGn6mdG4VS88NOHKNS/UuT0D?= =?us-ascii?Q?d4tbrpxOHlMTkGd0Ysnu6ukeQ8/WYI9GlRG7AgOk5f8ZkOgTDRdiwhmTPuQi?= =?us-ascii?Q?v6coX3pyvwm5uLZe1KXZ/i/gFj7HS3UT8H07kIRPEugQsksRc+DeIKlQPoAd?= =?us-ascii?Q?tiZGlj3Kvaa2rubdToLKZC/B4l6Hho/meDjpn4QB51ivp+1m3GbGjetzT4CY?= =?us-ascii?Q?D4s+efTOrOT9syCrvl0B3bDz2ik5W5TU9YL4B4syp9EM1pBz3GszP0FI5ezG?= =?us-ascii?Q?fkjSipmDF2P1gYtL9iGbkwc9QOBiX/8KyjjCmCnXFcf1sGwig3sjPepB+qMD?= =?us-ascii?Q?UwmQOPgQO31vz3I+vrnGU0NRD9yjmgPBlUBae9Vn06/f0vjk9GKuqoBlfam8?= =?us-ascii?Q?UjopD6SyHC7C49T4Ntze61uJNtYB3/G5HS7xXmsYx/AH62JlHkIznhc4Ay+y?= =?us-ascii?Q?r3X0sezXH/ITb7ubtjchebYN4pCIqWwk6bmSj9GXLvLm7P0xKlrQF/xtyg8N?= =?us-ascii?Q?53POyZ1lq5hA/Ay39VEx9GSXJyPmW3NcpD9JOWo1jTXLVAxfctIrLCg4IzHy?= =?us-ascii?Q?milpKUZVFCG3y+Bw9GIkoBYUfue9JHJqrU/tNgsNF2n0q6bLQtoEfYI6d993?= =?us-ascii?Q?3o+/RP1QcqVNOz1PrTrM4h5Wf2YAI0SDq+8TLU4D+pepFvDgNiqBbSJXCNFW?= =?us-ascii?Q?P8PrLXZyadSbGMTkP/QI0EMov/E3IvoTfnCo16QQo7s4GMketzDwqbmIDVRv?= =?us-ascii?Q?UK1ozbYXJMxv/Xk5DmD1mfWUrAHIzaPMwJ734O9HlNH+/cnzY9d5dK3UfKPV?= =?us-ascii?Q?lO6StnTObUrg2MHefoF5qGZ0PO4XjrB6lUoSrlzXNYHKTBfQT1b7O+16oXOq?= =?us-ascii?Q?ctsi/PYBEIWu3JZTbYyZ0TLxc3FojkJ40I6KqOMrQpbFJP9W7X7x+LJVUrT5?= =?us-ascii?Q?nHpx76QoBhaReEHGAsE2akQw7HkVsc9uQrZU41fwSeU4p4ttZek+GmaPv5rL?= =?us-ascii?Q?nMdTHdgHwpNP7NCWjkTj2NtQSEjJeSE7ZOaWTKNUkDqexg5Pxs4GDpeG+F4k?= =?us-ascii?Q?9WQC2HSbZY1LoUM8Gh0xynpQCKrbOHA=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 77476de8-d454-4c8f-99ad-08dee29ba9df X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:33.1955 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: v+cyxQYtqolkkjEtSoewvPMy301RIW8VyQbWUztK0/zNolfj4iFIWFps6epSqn18Vmoi8l+z6HUQ3YtIPEppMcicrAd74SEy8LfY4TgHjXQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR01MB6207 Content-Type: text/plain; charset="utf-8" The following patches will need to modify it to set up local percpu area on arm64 so the generic code won't work anymore. ARM64 and RISCV are the only users, so move the code to their own kernel/smp.c file. We can't leave the generic code in arch_numa.c because both ARM64 and RISCV select CONFIG_GENERIC_ARCH_NUMA, so it will result in multiple definition problem. There is no functional change. Signed-off-by: Yang Shi --- arch/arm64/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++++++++ arch/riscv/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++++++++ drivers/base/arch_numa.c | 51 +--------------------------------------- 3 files changed, 103 insertions(+), 50 deletions(-) diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index d46022f72075..0dab4e9c97e4 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -819,6 +819,57 @@ void __init smp_prepare_cpus(unsigned int max_cpus) } } =20 +#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA +extern int cpu_to_node_map[NR_CPUS]; + +unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; +EXPORT_SYMBOL(__per_cpu_offset); + +int early_cpu_to_node(int cpu) +{ + return cpu_to_node_map[cpu]; +} + +static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) +{ + return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); +} + +void __init setup_per_cpu_areas(void) +{ + unsigned long delta; + unsigned int cpu; + int rc =3D -EINVAL; + + if (pcpu_chosen_fc !=3D PCPU_FC_PAGE) { + /* + * Always reserve area for module percpu variables. That's + * what the legacy allocator did. + */ + rc =3D pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE, + PERCPU_DYNAMIC_RESERVE, PAGE_SIZE, + pcpu_cpu_distance, + early_cpu_to_node); +#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK + if (rc < 0) + pr_warn("PERCPU: %s allocator failed (%d), falling back to page size\n", + pcpu_fc_names[pcpu_chosen_fc], rc); +#endif + } + +#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK + if (rc < 0) + rc =3D pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); +#endif + if (rc < 0) + panic("Failed to initialize percpu areas (err=3D%d).", rc); + + delta =3D (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start; + for_each_possible_cpu(cpu) + __per_cpu_offset[cpu] =3D delta + pcpu_unit_offsets[cpu]; +} +#endif + static const char *ipi_types[MAX_IPI] __tracepoint_string =3D { [IPI_RESCHEDULE] =3D "Rescheduling interrupts", [IPI_CALL_FUNC] =3D "Function call interrupts", diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index fa66f9c97d74..d420ff9dcf4f 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -364,3 +364,54 @@ void kgdb_roundup_cpus(void) } } #endif + +#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA +extern int cpu_to_node_map[NR_CPUS]; + +unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; +EXPORT_SYMBOL(__per_cpu_offset); + +int early_cpu_to_node(int cpu) +{ + return cpu_to_node_map[cpu]; +} + +static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) +{ + return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); +} + +void __init setup_per_cpu_areas(void) +{ + unsigned long delta; + unsigned int cpu; + int rc =3D -EINVAL; + + if (pcpu_chosen_fc !=3D PCPU_FC_PAGE) { + /* + * Always reserve area for module percpu variables. That's + * what the legacy allocator did. + */ + rc =3D pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE, + PERCPU_DYNAMIC_RESERVE, PAGE_SIZE, + pcpu_cpu_distance, + early_cpu_to_node); +#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK + if (rc < 0) + pr_warn("PERCPU: %s allocator failed (%d), falling back to page size\n", + pcpu_fc_names[pcpu_chosen_fc], rc); +#endif + } + +#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK + if (rc < 0) + rc =3D pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); +#endif + if (rc < 0) + panic("Failed to initialize percpu areas (err=3D%d).", rc); + + delta =3D (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start; + for_each_possible_cpu(cpu) + __per_cpu_offset[cpu] =3D delta + pcpu_unit_offsets[cpu]; +} +#endif diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c index c99f2ab105e5..b3b91ceed6a9 100644 --- a/drivers/base/arch_numa.c +++ b/drivers/base/arch_numa.c @@ -16,7 +16,7 @@ =20 #include =20 -static int cpu_to_node_map[NR_CPUS] =3D { [0 ... NR_CPUS-1] =3D NUMA_NO_NO= DE }; +int cpu_to_node_map[NR_CPUS] =3D { [0 ... NR_CPUS-1] =3D NUMA_NO_NODE }; =20 bool numa_off; =20 @@ -140,55 +140,6 @@ void __init early_map_cpu_to_node(unsigned int cpu, in= t nid) set_cpu_numa_node(cpu, nid); } =20 -#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA -unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; -EXPORT_SYMBOL(__per_cpu_offset); - -int early_cpu_to_node(int cpu) -{ - return cpu_to_node_map[cpu]; -} - -static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) -{ - return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); -} - -void __init setup_per_cpu_areas(void) -{ - unsigned long delta; - unsigned int cpu; - int rc =3D -EINVAL; - - if (pcpu_chosen_fc !=3D PCPU_FC_PAGE) { - /* - * Always reserve area for module percpu variables. That's - * what the legacy allocator did. - */ - rc =3D pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE, - PERCPU_DYNAMIC_RESERVE, PAGE_SIZE, - pcpu_cpu_distance, - early_cpu_to_node); -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK - if (rc < 0) - pr_warn("PERCPU: %s allocator failed (%d), falling back to page size\n", - pcpu_fc_names[pcpu_chosen_fc], rc); -#endif - } - -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK - if (rc < 0) - rc =3D pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); -#endif - if (rc < 0) - panic("Failed to initialize percpu areas (err=3D%d).", rc); - - delta =3D (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start; - for_each_possible_cpu(cpu) - __per_cpu_offset[cpu] =3D delta + pcpu_unit_offsets[cpu]; -} -#endif - /* * Initialize NODE_DATA for a node on the local memory */ --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11020135.outbound.protection.outlook.com [52.101.61.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 388254D2ED0 for ; Wed, 15 Jul 2026 18:05:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.135 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138770; cv=fail; b=MeZASL8TZq/vI1k5aLaHXgLwvO+Azq3RZad4gqJDTYS3zjWe/aaj/oIb8eho5+kY+rq00tO0VwtaMkTGrNYFIwrv0ChLwiprA/aryKEmjAdypZMd048Tqtt2v9TXZpnNI0coeLNZAZVyD+n9PRgMDHwwxXETtqfACxr392jsACc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138770; c=relaxed/simple; bh=AXg0lceVlkimxk9/1ICDfv4tEUQ5QlKBaWdPcf6OrR0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=qN4WKv2nDCJ9vumKgL8CcMbISY8UQ5HrDCH60iOh3zE4lVBnRSEL0xCEXz0vUqcVIDMJHFh6XbZhY3lMYKx9rvfnKfN6KTzcP3CShG7B9gfBvYcDH3sgFi/cxG8YmWax5OJVnqCUcDFXZ81hp8uIK1wB2Lo+qguxGKHb4Kfomx4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=vGEY361Q; arc=fail smtp.client-ip=52.101.61.135 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="vGEY361Q" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=eHnASWd8vyBC8va4caUr6TiSAo+5XhvERLlvBrjlJVz+l66uh8CSW2AJSrREIy0/24LJU6aHxws458zLMFm5p6yxpbQxPvqxw5KJkLWqmGlBw83+fSrybbjXBuQ1i32Dr6bMqVNzdflqdDLf6xiLI+T16BOybwIzZCxSlO6E/jrDBMIK+wWJZ3IzzlYWPGz7VdOjWAEcPpJaN5+dEOhXrjOg20/FwBt4tFQ84FTVFVWH/55KNRzzDVDlWlEhbSLjYpS1IitOVYgEvvsx541zdLELs5D2mZAT/xV+7c0t2e/Pidu4OE6ynxP9zjmx9ex0fCHDxGGZaYUXkJhbNxSVMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=e2t0BTW0L0KAqYei4Xv/1VSaBkyJwgz1/HwudPiGu9Y=; b=UazEXMSm+Vj8e7bgTyjtpQX1nCa19+AsVoCLWZbLVDINvDVPtTD4DJV5FJOliQ/fqu67OESeV0LNgEmtcwibQtw1UthRMJ5yYpN4xqdaJrEN62B2By9FDLb93oP+JCGapovy2Djl6CPQYnM7zmUrzOTN4VSewERpyhbODQK53eSZQy/jvkd9+ODS+H2j9YgOUesq8pH23uZ9VqsYcB8GV3TftUp5TH9P9+aqe5moLsUob4u+DKtVku5ZNL65nqWWh0faEdbZ8PXX3tMuaghwK6Q3tfM0waXR/cmoXB8F4H5PNIHFObJ/Tx/rJa+zXHTEzOXT140IFzlyb5K/wExpfw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=e2t0BTW0L0KAqYei4Xv/1VSaBkyJwgz1/HwudPiGu9Y=; b=vGEY361Q8Ha5nj1T+MJfsCVXK1y1S59gyrDxOuN/rEBIRfDt3rs4sm9ZWD9YXaKGMqYsTbadoZNyZiA6oQaVttEAskCNAMYVBmfbACdZNILjVUjgq1Y/FD2cAQADxrWzgR1wShjHiDoptFTPz8oU1/2aXU9I0uW7eFRxf/HRZuA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SJ0PR01MB6207.prod.exchangelabs.com (2603:10b6:a03:29d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.12; Wed, 15 Jul 2026 18:05:34 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:34 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/16] arm64: kconfig: make percpu related configs not depend on NUMA Date: Wed, 15 Jul 2026 11:04:04 -0700 Message-ID: <20260715180455.515692-3-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SJ0PR01MB6207:EE_ X-MS-Office365-Filtering-Correlation-Id: 9fb3fa7a-2e40-498c-3e96-08dee29baaaa X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|23010399003|1800799024|366016|56012099006|11063799006|55112099003|18002099003|22082099003|10067099003|921020; X-Microsoft-Antispam-Message-Info: bvw3/TQKLJ9gglh9ntO/D9ggaUYgfbwBQ7ejbyCdpdL5G7TFH/JQmNsZ5CwVi1dG15dPCUcrEGbAgKUv0iNEQ7ay+ZUfeOj/eMJyjR+RavckxdpTLb31Hn1Bi2G7myt/tF1lGGS8Nd5SEYkZBFuxepIOWWbPrFPmAd7MTwbw897B0eXLMEoWBi8VjxoQ9i3Ar8+QTk72uyC40wZ2MikV1B7RwsIBGib/rP3NLGsvs0owzJuEz6vt87b/vCntvzGNvHmRf17nm5HftNQsBnXlkuEuTkT3Hm4o7o1LAypmMbi2v5Xn1lWdTxK0/o0yYw0xYh74S3rgcnpdK4NWmlG+h+JbHSyii1FbTl2AcKeCa9MKfkKUJjyT8iqTRviRFkpyp0W4GKY1mF7emoeE2+vDJ3adfmZNlp7N+t2aDrDEtIf/VbOS2T8+sqs9hVB0HojVHhlDmltD39zBK6htAuqyQ0StJ6QenbFg9JlPeWCq9z9o0DgIPI5ZpO1oXF0I42tBxWfSZvHLObAKtYdS2xgQ3hZD7zv0CAXXZLdg0uHg2f8NzbopPaFhtw/HXhGlz1muXzVBxqjV0dvYWvL3IpHIt2axvZVx+eHkUqEQRWbVeZfy2wHSrbJV2mZIwX6XLoExKz76fH7x7hR/5wwArWY+xHsTJMC6KuY2nRtTwVxeqOUPYY+3QFNWS040AIc9lvma4VaracfLrjXMbn2uwsPZ/A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(23010399003)(1800799024)(366016)(56012099006)(11063799006)(55112099003)(18002099003)(22082099003)(10067099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?8sIQaHKMbPBNFzkR7JW9TdMpiyEs1UEylBb9Aza00mlUJSkM6WvuO7g42jam?= =?us-ascii?Q?Q9iR3vD5c5mPitO/eRSX6UAY7N8EwcVjdC/m6ira/SSFxL+jU1r3I7vzNIIC?= =?us-ascii?Q?KDCbX5NIa4v8XXz6XQxSOlakkXnKpvDBLDWVNRbjEIyHGwroiGe2pG+3A1iX?= =?us-ascii?Q?Hfunv/tRvgFrIF/PfEV0gX5DryQbRp9194RJNxnZ/byJUEUYVFqnpDAkoAKd?= =?us-ascii?Q?Z4VuUPGvd5DK0vS/wCP2qejZ92lhMZuIay+xt0GzZbtV2upt9cnMEXsmyJ10?= =?us-ascii?Q?xb/rfRv5UFzBp3dhdeV1tZMz+hqGKgAddSpPjXtPoMFAFP2YJ+MGFFTTJNiw?= =?us-ascii?Q?FzLIThYrCM4U+0CIpnNHzMlMZwpuztpLeSahuW9gK9EKOlA2v2f+hEl+CqXR?= =?us-ascii?Q?w/YcgpvUdk/Ddjqlr9hdgiSsfZi1eeNHC0R6u8mxNvJkBzfs5v3YaiwgW8KU?= =?us-ascii?Q?kexuET8bx4tAACTeuaNyGIxUJleN0qp5yFyOu+zgsxxERvKGd8CdykFdz2+8?= =?us-ascii?Q?OimmMd/VZadOyyIfESLx4d+8y1SixsB/dWNZWGysPEayhgR/R68nll2Ac4S/?= =?us-ascii?Q?Cs2VHLFLljO2ZaqYSVGNNCDS1cHnqG4F0v7QNx90uBMCdsCILAaDjcL2bbZR?= =?us-ascii?Q?XqWWq/vg9LPKJXOlJMECnnMoykZNVYFGikwVkyHUt5FQA5Px766OzBST6JZ2?= =?us-ascii?Q?Rr5z5h7jP/G2P8MC2zPiqQAKFCO4BNYsdtKEB02lNrdhyzMv9sAFFkVk5Mtp?= =?us-ascii?Q?CaqM91FwrG+KE2wCdQ+fXIQa7SS9vDWufIYazsDclT71tCcumb+iASLq0j4N?= =?us-ascii?Q?MVCyZpabJqE3DhGGEdC0tUBBDF0x21R27LCmEuS7Bs38aIy1zh74u7Y5tP4t?= =?us-ascii?Q?pMR07I4XW0231qfG1FCwN2lS1gggIhJ7hvU8uUB0XYxZoz8gX6NncWbjqT5Q?= =?us-ascii?Q?GPh6rBIzeBiURr6qXT66tnRiY2/Ugk/U4DPKFZydurQGiRpCYRJeaaHEt1cc?= =?us-ascii?Q?VMBvoxw1sq9ygCYqo7sm6YAfCm1JK1mVUyw8AmG0GucfOjkyxNkT1VnUNuNJ?= =?us-ascii?Q?IiWCuCSjK2tRRw0eShDFqx10koCj5eApi5tSw/m6BQvbm89BS8tKojjD5UPW?= =?us-ascii?Q?xjgr+uP78Y+vHyqNQKybDw5sAnQSh5ZaKvNC5GB0lMe3PDyyQ/aso+ku0qUw?= =?us-ascii?Q?XhyY8HDwwzQ/vg+UgzGE2Ioe8alfvspc2kFf7sIX9l+zDnOeRZGmj0pYSLBi?= =?us-ascii?Q?67xQxxwbUJQ8SgmcoVYBMokIOuqddA0D6l9NkBtQBPLPzxEPLvemO4O1PiXe?= =?us-ascii?Q?li+E/DA6hp4YHaUg4JDA131nzirTO1YLObpETzq7N5TlacR4lBKzrMoZDo7O?= =?us-ascii?Q?HjJS9DtfrdH0ALHgduog3B0MYnhx66jSmfiUqEctLHSD11UYoSR/7km4mwmt?= =?us-ascii?Q?2HREAr8htkDotTIRPsGxBvVTTZXF48SVaIzUiNJ3dqE31w0uzcMIGr4qhFDq?= =?us-ascii?Q?hKHNprJsUSklXvOt1Xak5a/9ua4of5D+Hap6IQnfvyquQIpQbi4pOb7xbpOE?= =?us-ascii?Q?13k+Cr1EMuB2i06bXJPGewVh6HdCtq89sPgXFGhl7qnqVXRoeDLl7uo3DR+u?= =?us-ascii?Q?eqgjuARFZul+ibNxEBtmjAGbTJDU1TEdpsA1X7kBDqSJgKiu66JxLe1adju4?= =?us-ascii?Q?+zBLQkTmz6D7WS0ttnw9Oc2Wvd1ILqhdkht3jzDGstzo7Mq4H624UwZ4Q36J?= =?us-ascii?Q?yzeSluiTCHv/fy+zHEqCSjFImlziLz8=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9fb3fa7a-2e40-498c-3e96-08dee29baaaa X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:34.5653 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 33VfcPOiSP9nmCRigjHinJoK+xKOcPHTTZBZls7rqS794WA/RKCvsz6/SBvre48/Uqn6E/5emdDe9pAR0NnhYDJJDgjUN/icj1xw448xJH4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR01MB6207 Content-Type: text/plain; charset="utf-8" The percpu set up code has been moved out of arch_numa.c, so it is unnecessary to have percpu related kernel configs depend on NUMA. Signed-off-by: Yang Shi --- arch/arm64/Kconfig | 6 +++--- arch/arm64/kernel/smp.c | 10 ++++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3afe0688919..2e1bd79a579c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -224,6 +224,7 @@ config ARM64 select MMU_GATHER_RCU_TABLE_FREE select HAVE_RSEQ select HAVE_RUST if RUSTC_SUPPORTS_ARM64 + select HAVE_SETUP_PER_CPU_AREA select HAVE_STACKPROTECTOR select HAVE_STATIC_CALL if CFI select HAVE_SYSCALL_TRACEPOINTS @@ -239,6 +240,8 @@ config ARM64 select LOCK_MM_AND_FIND_VMA select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE + select NEED_PER_CPU_EMBED_FIRST_CHUNK + select NEED_PER_CPU_PAGE_FIRST_CHUNK select NEED_SG_DMA_LENGTH select OF select OF_EARLY_FLATTREE @@ -1581,9 +1584,6 @@ config NUMA bool "NUMA Memory Allocation and Scheduler Support" select GENERIC_ARCH_NUMA select OF_NUMA - select HAVE_SETUP_PER_CPU_AREA - select NEED_PER_CPU_EMBED_FIRST_CHUNK - select NEED_PER_CPU_PAGE_FIRST_CHUNK select USE_PERCPU_NUMA_NODE_ID help Enable NUMA (Non-Uniform Memory Access) support. diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 0dab4e9c97e4..28ff889658d3 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -819,16 +819,18 @@ void __init smp_prepare_cpus(unsigned int max_cpus) } } =20 -#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA +#ifdef CONFIG_NUMA extern int cpu_to_node_map[NR_CPUS]; =20 -unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; -EXPORT_SYMBOL(__per_cpu_offset); - int early_cpu_to_node(int cpu) { return cpu_to_node_map[cpu]; } +#endif + +#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA +unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; +EXPORT_SYMBOL(__per_cpu_offset); =20 static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) { --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11020135.outbound.protection.outlook.com [52.101.61.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1E0C4BCAD1 for ; Wed, 15 Jul 2026 18:06:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.135 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138781; cv=fail; b=JM2VfYmPKOq9UtQ2VIKZedZaLMBN4Ie4Jc7L1cqWH6LtqjzX8Vn1e/yu7Ij9yhxdllZYLtg/3RDSDfX71oqN9qY295j45RZkJqGS4OQEm5NbLg0dl6S0h88gNND2MwnDFZxYS9EHoHz2AwgWHo+YdwHTEDBphpv0IK4CZSCkdYY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138781; c=relaxed/simple; bh=Tpif+BEBTHfcqx3brBjgPNB/79xLBN/qCdvmtn27j8w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=bpT7L6T4wmZB6mNoCkUHW6/3YeFzsvUrMCpdlVd0BrppIhZ9Xma5RkE6iCo9pg286xbN0owVH1YvolmTSojYTgenfO3lmY+lvn428l132QS/iEKOjiox86/fNfdbMCN8gx8ZwD/QbkOAwShFAYONBTqBivSzDoNj5VFgm6BcSqk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=O+do1APe; arc=fail smtp.client-ip=52.101.61.135 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="O+do1APe" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GkElIBLjHASkj+5/YAZEuXsjKxuIYebaJ5aGViP7phFHNNAOqxBmvuD/hr3uQZivCwdazu+DL0kekaCjyZ+vXOA7dpoPgu9OyY+b2o6Y3208+cU129MBnKZfQq/Gth/bwX9tTu2IvCIsZp/VyNRcNsA8T82ff2KEYR8RCWE11H/Io4sSHh0ihiBQMO7YSvS6Fvsgjjf6rqv4C/4xVbpVmw97svHtcluF26veStbUJ6tXS4sVs0EFQLDZGbikj5Dj27P6wsn0T8DCI2hwEjdp4wRTMyYaETVSVM+ooLwJU+/gdMw02vDnMWeYpDTLUKcqw+zmmHpiiF+/z7LPDM7JMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wYTcMpCGQoeSA3WN+ofCp+QWfkKB14/vhiJiLjROXnY=; b=USDGh2a0lHWqlqT0pb/oklHX0O5H/O5YQ1FB9snbLsdTiec33/r2bBiXV6Z4H7wbv/QV5mzUJvBa8cGlL/CNnWkBNgp51VsBEMQjrRK9tj9qDjeEAdiRArpvl+KBOZ8oxUs+pi1M/LsiHJXtRHHUO7NCs5859eCvahy3QLoZ4Me97UXxskp10r8ZNiUZcFothwFhufkK/rQIXrxv4DHFWh18AGGh+HmrJw9JlNMDbLahaH8h7l4Akxr0F9j5FdiqHZA44Bo6ilbdP1VgtPsB2/ncEcmM2BK/YPuvs2WUn45doxuGxwY3X3Azz2ZMoXvpCkiHVh/5nAzkTVL2StwcmQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wYTcMpCGQoeSA3WN+ofCp+QWfkKB14/vhiJiLjROXnY=; b=O+do1APelCiCAtIxgiNIxncFn5VXBVt9v88/DXjg8BOrBOxzFAdeZ6OojBQ3Kc4t1Ky+EXfN+bqVlTocJ0/w2pyc/2ddzLdF5ItB4oAvgOyQOCMhYY/3v3FnywgUAD44SDsHmXTVH4vKB4nq535r8TbkmxWqD8FSol9qToP3WkQ= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SJ0PR01MB6207.prod.exchangelabs.com (2603:10b6:a03:29d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.12; Wed, 15 Jul 2026 18:05:35 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:35 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/16] mm: pgalloc: introduce {pud|pmd}_populate_sync() Date: Wed, 15 Jul 2026 11:04:05 -0700 Message-ID: <20260715180455.515692-4-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SJ0PR01MB6207:EE_ X-MS-Office365-Filtering-Correlation-Id: 4ea69a7a-9e2b-4f2b-2cd1-08dee29bab6b X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|23010399003|1800799024|366016|56012099006|11063799006|6133799003|55112099003|18002099003|22082099003|10067099003|921020; X-Microsoft-Antispam-Message-Info: 2CvRLLDnfiUaGJoNy34blsDEsf5Ib3yBY1e+lF1cwU4OnMT+ybVtBVa2da+ZrwuozG4ZuFShZNigIsuzIvl04leg8vGd5VxzkWa6otxcf2t37gE8ZfNT1SYk5aUSevap39KuoEQrAlL41PUXFasbY6aB4Dki14WObyRqOm7Ajh46ejAerLLWwCkKeJf1xYzFJM3kYGD8agLERQqdLd1IhPtgMrlw/ChdVTZJJMDsbBMqTEPJfxLK2XFrgmPOjAvuHDcpY0Re2hSBn5kLIEJwsfx+zuKrIuIJhhEt4IH5tBwVPHwI1b/LQF1urSasPTnhpseUILi+uqO5Sm8sRGQLkkTBaHEgzwh23vQuaYMhXIP0kxTV0yjZPAoIVWDN+flRr7Y56IwsLEQlIemS52G+50Bm2iKYWm7V9FN32hyi1I1wO/+VB6Mkzc9c4IT0HBE06QlcK8yOZCAx40E5wr+WAHGOJ1CDBRHIu8C1xIu+CpVoB6F/esWMIt+BW6JaPZ6w43XZq4UnC9NRWN/AZezudHO/x+vbyZd8yXLm5sjZSa+qvsrYl95QN6ArjQlgBmVxiF5JUxrBwW92HSQ5ctI9RVZ5ZhtXSTHVKs7TDc+rKo9s3uz5hkxg4YRI39TOs10gt/IsfBTuiwbfnOzJ50+zXzo05mcR1UyrveVHbjSVW+sekqVdHaP8bdWjnHK2gd0s3jbHXbKEPtRGUC4zSCohjA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(23010399003)(1800799024)(366016)(56012099006)(11063799006)(6133799003)(55112099003)(18002099003)(22082099003)(10067099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FuR6of1q8nRdY/8D8xglkaqxPgVrZNimEfm0hx99Fa2w8BmdnmiaCz1zY6aq?= =?us-ascii?Q?uXJG73PljwoqHzXwDwiNlW95XHKx3n7jtVbU9HyE/LmD5boVqBNnHGZ6JJj+?= =?us-ascii?Q?yS9kn9R4gziQTfvVtcdsRtMMr3vvSJSvH1zLMkgpOZHUCT9aY2Hol20ZqWtF?= =?us-ascii?Q?zl63oftFQnpSXNIJ4m+VG+BCzOPf7QUmqBTllUu/qFSb/LqvKkr+D0oFz+zn?= =?us-ascii?Q?FD+/BOkBDPCwBR3EFSvb3A/L36Zm3XhmGENsM+Sd/wcQ28XiEOFk3l1omGOU?= =?us-ascii?Q?27zxcX/qzDlEifwPMKiAe0mK6N2qxNdJykizW0j96eEKCQBp7jtR5FQdXi9l?= =?us-ascii?Q?PHiO4eIapky6AtE5ooqSEUTyyke0Oq/rgXfEl8h3EJxy0uuBkG5jvuQhVToq?= =?us-ascii?Q?01L912kjLfYFVnqk/mX27cxkRorShPzX/om3V4HKNPtO6xXvjX16vcnKUB+Y?= =?us-ascii?Q?7elQGD/UqtyDHgLE8JsHfdF5SgmhW1AP8d8EPTG4OWIREF9tZ1Nu2Xd0U5mZ?= =?us-ascii?Q?e4pePN+/qxHqnTwclRB5VZaUEwkqvC2OU7RGaHFwIDNka5p009EmycVaodgn?= =?us-ascii?Q?QnVoUh9sU4+hkZ2r8wV99zcbNtXoP1KTD2QASQUB6rMs6BuOC3hK6Xb/vq4y?= =?us-ascii?Q?F1HDRG4gT6iz2vH9PZVqQCFBmV/0XtUr5u/TUiqcvVSlnk/eIDP/saeDknv+?= =?us-ascii?Q?qPeYh49OB35vBOCBRlwWT78ox//rrN1GchEBO1MOjggA45UY4tdh3Dp28Bzt?= =?us-ascii?Q?CdFXE6t0yimAjdeOmnOIn2O6Ur9qybHPQKo4byHts6FlS0gLytwS6/h5MKeo?= =?us-ascii?Q?P5jyb2l/FtcbxfoDRiCckw347j0lpzGC6+rNzz2ov5I4erDan25GYXWIzBSg?= =?us-ascii?Q?zbrpiyGxAz4Jpl0nUmG63vcqkgiG/Rv82JvAOW05Ta785qZSFdy3YNwzTEvS?= =?us-ascii?Q?3OsU3wFtzRID4JZ9+/WpLhhdD0OkxGR68wldW6aHz425/TNHMU0pKkFesEPR?= =?us-ascii?Q?Y5QFVaq2evgbksdh5NY2JW8rlEMcZfphlTZH5XehkGYdBMqqbcmwsgHw/RhK?= =?us-ascii?Q?HECc92xpTh9QUJKtidoVXoNoGp6bD22y93abSTzE7qSIgzyIKKM47mArA4Io?= =?us-ascii?Q?Pi35YEcmQi2nEyB9PJDkSH3PNET7oprrEWp9LSowW9JNYhUaagYr8AQ9Isuk?= =?us-ascii?Q?x8lcyR47/hWtyNZCij5yVXAAbZdoY98NGwJj7aFz8CTPoo4azmi+2Jk3GUyf?= =?us-ascii?Q?eDhFRyzO56Yay7QELeRd6iiw6dQ1tmrqduYYnfEErBV9RUHUNDXsSzwZbdxe?= =?us-ascii?Q?ZiFKmTiFQVM0WKTLlUIkTp0H3O1X/HTGOuUos0LE3kBB0S5W1Iw02VeO3xLR?= =?us-ascii?Q?fXdMJsBnwUL5b6nlKW+RVk2nXu8q7yQl5O9JW3P3IxPGs/NvwqGSyta40Gle?= =?us-ascii?Q?MsIJW9hwd6BsTNWlef9o5QvqU6PtT3djmmRnsoGqhNUhUYnNtZ0pwdsgsmpl?= =?us-ascii?Q?Djw7Et59b7CugF+7LEuJ3+7t42XndZrKd8Gqud0e34/QctX6aB3MFBSh60cP?= =?us-ascii?Q?RdT1ylPdQQq1ccXtlVZCWoeRIhAsVWhmzdnMT1PHPCRM3UEh8/1HBCE+vkVo?= =?us-ascii?Q?6ExBoxSzT7fpaB6phi4SIVt1FudbNZNqT2acRvOpPtlBBlnJdGCGbC/LiU6G?= =?us-ascii?Q?ehAEN6JRFPiioXUW/6PZ41dBQ7ojmJIWrElHbHqzy86Af1JjC6CH+nkBwrsU?= =?us-ascii?Q?WSMXi10azIld3Avog9OUwMp4tjiZ7dk=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4ea69a7a-9e2b-4f2b-2cd1-08dee29bab6b X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:35.8455 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yXOCk4yw0TTs918881rn5PTWrj8ZmfcPqacrEoxJOXsmxWWS2R81d+mKXmf54pqv2+j0S2hQ5at2hFCmeiI75l4B1JMp0aMM2t1pbrNCs+w= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR01MB6207 Content-Type: text/plain; charset="utf-8" pud_populate() and pmd_populate_kernel() are called by percpu and vmemmap to populate PUD and PMD. They need to sync up kernel page table when percpu page table support for ARM64 is added in the later patch. So introduce {pud|pmd}_populate_sync() and protect them with ARCH_HAS_{PUD|PMD}_POPULATE_SYNC kernel configs. ARM64 specific implementation will be defined in the later patch. We can change pud_populate() and pmd_populate_kernel() to take address as a new parameter, but this change is needed for all architectures and not all callsites of them need sync page table. So introducing new APIs for this usecase seems much more simple. This is a preparation patch, no functional change. Signed-off-by: Yang Shi --- include/linux/pgalloc.h | 13 +++++++++++++ mm/Kconfig | 6 ++++++ mm/percpu.c | 4 ++-- mm/sparse-vmemmap.c | 4 ++-- 4 files changed, 23 insertions(+), 4 deletions(-) diff --git a/include/linux/pgalloc.h b/include/linux/pgalloc.h index 9174fa59bbc5..062c64a5d4bd 100644 --- a/include/linux/pgalloc.h +++ b/include/linux/pgalloc.h @@ -26,4 +26,17 @@ arch_sync_kernel_mappings(addr, addr); \ } while (0) =20 +#ifndef CONFIG_ARCH_HAS_PUD_POPULATE_SYNC +#define pud_populate_sync(addr, pud, pmd) \ + do { \ + pud_populate(&init_mm, pud, pmd); \ + } while (0) +#endif + +#ifndef CONFIG_ARCH_HAS_PMD_POPULATE_SYNC +#define pmd_populate_sync(addr, pmd, pte) \ + do { \ + pmd_populate_kernel(&init_mm, pmd, pte); \ + } while (0) +#endif #endif /* _LINUX_PGALLOC_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 9e0ca4824905..13ea4b29ee7b 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1509,6 +1509,12 @@ config LAZY_MMU_MODE_KUNIT_TEST =20 If unsure, say N. =20 +config ARCH_HAS_PMD_POPULATE_SYNC + bool + +config ARCH_HAS_PUD_POPULATE_SYNC + bool + source "mm/damon/Kconfig" =20 endmenu diff --git a/mm/percpu.c b/mm/percpu.c index b0676b8054ed..45c30c6531c5 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3163,7 +3163,7 @@ void __init __weak pcpu_populate_pte(unsigned long ad= dr) pud =3D pud_offset(p4d, addr); if (pud_none(*pud)) { pmd =3D memblock_alloc_or_panic(PMD_TABLE_SIZE, PMD_TABLE_SIZE); - pud_populate(&init_mm, pud, pmd); + pud_populate_sync(addr, pud, pmd); } =20 pmd =3D pmd_offset(pud, addr); @@ -3171,7 +3171,7 @@ void __init __weak pcpu_populate_pte(unsigned long ad= dr) pte_t *new; =20 new =3D memblock_alloc_or_panic(PTE_TABLE_SIZE, PTE_TABLE_SIZE); - pmd_populate_kernel(&init_mm, pmd, new); + pmd_populate_sync(addr, pmd, new); } =20 return; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 99e2be39671b..463c2f8b611c 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -198,7 +198,7 @@ static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pu= d, unsigned long addr, in if (!p) return NULL; kernel_pte_init(p); - pmd_populate_kernel(&init_mm, pmd, p); + pmd_populate_sync(addr, pmd, p); } return pmd; } @@ -211,7 +211,7 @@ static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4= d, unsigned long addr, in if (!p) return NULL; pmd_init(p); - pud_populate(&init_mm, pud, p); + pud_populate_sync(addr, pud, p); } return pud; } --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022103.outbound.protection.outlook.com [52.101.53.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6248B4DA54F for ; Wed, 15 Jul 2026 18:06:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.103 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138779; cv=fail; b=B4c40cBM7g4CXi4u0DEdhNYwVfOZ0S0GK6iiNI/xQXqE5bgWeSlvW2LD9aQLENzGq5Byft8fb86PZz2tz7Urb82I0y7cs4IhXtqOA/kKJrmuJhRSeUc3q7QTOZMx0CcYN0OGo+klpJYAD3eAaMv+GdPtdmYWt8AnnHnJXC02RNs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138779; c=relaxed/simple; bh=upzXNcMN/1InmxBfUQ9aOjy0VM+TA3IQPKx1pilYKgI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=lnIQUx4YLLezMjKTVGS9fL5pW/Mj0qT/vGGMNX5leHTGlJoHcStG4VCxiCKzfHSxHR3WFFMBkEM+mSinrlMwjGO+SVSPj3CWqKNcpcgk6tEh9x3X3q+jC/3CFii+q7XLJwFhscSC75cQC2iqlgkeOqx8VNLqeWFdk5mgZtR4Q3w= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=puozKiFb; arc=fail smtp.client-ip=52.101.53.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="puozKiFb" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MhYqR0yPbAD8rZAeH48y+ZYzBM2hNcI/5uE83jaKLKZt0aqTG1+Y9vLYhC8FFAJ4yGWhXUJlN7YVxbBVExJZmwgPKfgrc10YzmEk2zqS/+pt1UAtYM4uqy6Laccwae72XfdzVnWHxiTdNc4KjJqV9uzWTeHxnmCF2VEov2OAYtHRCr0sC9gHXwwaFzPzgjxslGWYIF9YVLOzO5MMHtmSbrKdZAv23jFhi86ZsP0qXKYPf0TPdZ9QLtMTgepC8gWFMUhQ+oN56DcA65Pelxw3lLvd+QXguX+Ya/ToLCI1m409Ziy6S8OMyQJu9cCQErq/9SOQpd2sK50qiQoLn9amkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MTolXszXgyPGDtHoex5RWZvPvZ7X5m8x7jJcaafR+kY=; b=uhWztGDAy3dBGGY93w80TUOs5CoRgiQ3ithroMKRQPtzWAloXtb7d4jWyiez2UUy/0gS/nn8szIsCZ9GohKoGVn/1+IBNYM7LHKmMyvw2vIisdFncgb24yQ8MHK63ViKAHZtH68hUH1yFXdQIWOikyTosEp78A1wAF4thAPkuKcDTtiAJZQmZGThRvr4TRymRze0GXnDykrjokkjT5/jLzYujI3/CNf9V6hDpVMl7Fms0njRgTxd+XeJ5kwDA+//BScCk3YtdQHEd1lmWDJ7u9IXaMxFMkSwkYazAPH+okCaZN/m7d/+KSvD/8/HXzI8jdI/tH/uJO+4XMR+F26E7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MTolXszXgyPGDtHoex5RWZvPvZ7X5m8x7jJcaafR+kY=; b=puozKiFb7LxcGa41a3ve5ia590Tt7CAfcepkP7rEa/Ck7NJtsCK4aRCj8cmqeptsHTmEQphVvyGC+drAbThYiJNTJN7E5UEPpXUPSPyjFyYX3ezLC1P/d85AV1L4TErnSDgjooT7Ss6F0OGmwDFfCS18yhTGW4EWoLCaBcxKGcw= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:37 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:37 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/16] vmalloc: pass in pgd pointer for vmap{__vunmap}_range_noflush() Date: Wed, 15 Jul 2026 11:04:06 -0700 Message-ID: <20260715180455.515692-5-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: a5df8d04-8851-4821-a928-08dee29bac2e X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: JblgDHPCoQSET05ei9G377R+lh9TkpSIHhsqdro78esh86hVSilm1V306wq5MzOBMZ8CiPV2uk3nvB+FFBTkuxOIoKc1oyi3nOYozDeN7ybjbnx9sIqzyQnKj8ECWF0HcLtCxx4qxVUjhWp8IwdYiReA6OIlzJSafb1DtrZQvTcjNX54KCvZwOWcWQmLpiYcbukz5O4xJ7w01leYyl9heMs5cwia7PkSUOsXsJ8ic1jNJvHpaIYhN7omgG49v3dvMHWNmacz/gB4lC6PiKcsRynkI7gwqPDnhA1jnkJPCckyrzhT7vMNpVsaGMO8JZxg4AtkmGOS4WVkR0yznfOSu3yrFlazpwoENMyFa4MvngKLNDWs0lHeXlc4VapWzlTbBvdcA9zqlTzPP1yufHx40Z/y/YEiL8etbXhLPEB1eRouzYTbXq/6+tbQQZ2x3Y9+muiFdlwqpuAEnQ0blc1veaEahkICqy60oLF/4QQDaRVTe38q2muUcHiclc9Y66wF0zfyCZfti1dZA+zE0o3r4GbEVMYrXzfDIpTnWZxOOPLFhbgeTgIuoCiqUZk9penGfDU/8ctXWYi8zpEpMCfzMtvYvL0o+JkhKzf6KWpvI2vzp61M9t7f8mwhGSSSJathver20nnDCo5FFmGhRNtLFcVw0QUCHkSLF8SoZG4cWXKMiYYXA/EB2Fru95doXCAdsYciS7vWQj7HOS16t+bNhg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?xt2VrK+3YsctMYqVULRbvp3lGmuCUeGvnFHya3h6mH7zpv2tZ3VS2WaL5+1r?= =?us-ascii?Q?HZll+anz54SuxO74uq8xTJqMcesFuB467zJPIrm+FWVNh9o4hJ/fBU3nWvVW?= =?us-ascii?Q?eOwPd2sISYKuXTjtn2J7Jxn6208EwnjJ46/XeJtrCva8HsLq4rj5MLAnxo68?= =?us-ascii?Q?HwzDGtzFZWwHcAwTdFp5GbWV4GG93pODddrW8jm9YIqcLaQgEWU/wOU/ZgR1?= =?us-ascii?Q?xjlar90PhT+MniJTwETQYJu4yjv1jQ40um4GLzzXX4zSJVcagiS+3duocmMo?= =?us-ascii?Q?Aaq/A2Iya3wXvruLfGEfLi8m8ZVJ3HtdwhVcdPrbY++3fOPVIN8toXypfO1A?= =?us-ascii?Q?A1CGZIdEfbe/lwNrEWHOmC+GNCBpYoisookqxsTDqRfHx7WM6zjwG1E0zYH9?= =?us-ascii?Q?C/Thr1ugrhPJMQW3HpwvU4NFWChTzYH/a+obUjmZ4Nq3PJ8OkjHUvJZaGpk3?= =?us-ascii?Q?himWoKc014g6qBDN870/86TTdOllpauJ3TW+spROTm2z1qh0mxWeooyMThSu?= =?us-ascii?Q?/8R/tjSHh+fNMvk2qAzXsxQDbYvyl1PrUviDSSIhUZXPAVGM0h+JgdsWMIjK?= =?us-ascii?Q?HMCdpHQx1mf1i37wbAco39ch+38xJX/1jpFXonDjLY4ngbHjIwlEJJMpxncr?= =?us-ascii?Q?1aMRMhVlTCYd1K0y1mLo04WDQRc9tYpabhrBr5NbYy/mZ+p1qxUyvb2jvl+f?= =?us-ascii?Q?jB4CiRppYoWodn2a/1P4vgNwiEFd+1HXyPoVzHywq5zryGXbxptdUogyvhPD?= =?us-ascii?Q?/EVrjHs2uQC5Wvx8l2NMUZluMoa0CyNPPtAX5kStBeMSKl0A3ogqabFHLE10?= =?us-ascii?Q?FQgtsfXM0jIgBg9SChoVx+kRiMBcoAHP+6gJj5K8UR8Lb/MqJJFMf/30FvZg?= =?us-ascii?Q?vp78hgMBPila/DO+J7xWoqdeElf/qEjgP0wZIIXqdmmxp/3Pb0i772lOXjud?= =?us-ascii?Q?Z/0zmSY4JxFOXxfRo3UoPEwy8BA5iEU3nFtfrEe9tqdyK9ewlIy9VowtGjan?= =?us-ascii?Q?5arQaF79BF+FLYPP8ME9smk7DnyhQVCQ4ABp82jKRWwhD9HsA/e/PZxlpcp3?= =?us-ascii?Q?oUR5JbEEX//2AjAzgWtVl/xlNJWHBf16BtiexYJusk77KgLLlg/7gWdvXiMg?= =?us-ascii?Q?QJ8/7P93feo38b6I+Zygt6iCgEIEZ6Tl8a2ADzxrq/BpXj1kbkES3vf6f74O?= =?us-ascii?Q?NAYi3WkgZ9iYpph7XDww98yMOxfxWnbMal3HMum5gCaAfI2jnf2yc5AVWNjd?= =?us-ascii?Q?DQ6t6asDg4RlTkIGUHhvMxHvGM61aKnKFPXCfVbdXwN6bB1gpqmkXy4KlENS?= =?us-ascii?Q?89QsizwUIZyBflBmdLbnhtxLqpGXr6wlUdEIfQCSVrUA0NN6Omlup52FVqQU?= =?us-ascii?Q?ZvNIPjc/VXOYhAsbIoZ84ep4nWdSYQNqxxERese/rbJWJowtxoRZmeAEnbzh?= =?us-ascii?Q?WiHIhn9Qqv1+9M+m+HhG2XNlCQNY0xdmfk/5bdaXY5YOM8pPNUtZaLugPDpZ?= =?us-ascii?Q?tqjz5vDqCrFsRQ6ho4yV+F0Ws6lJQ76mSw1Sz/mlghdsnJuaEper7lt9o6dA?= =?us-ascii?Q?/yfb46F1S0vvNXxjXNwP4cjiIUfFrTtj/hBnoGVHOXVLlrPEyRDl/SN5pSCT?= =?us-ascii?Q?U2b8eOyNtPkqbVdsRwqkC/mx/SNINXcMQ9gLs5Im1Q9U363ViBE5h8Uh11Aj?= =?us-ascii?Q?cQ/pVCvZBMrACfsF5mb3X6pNu4O5ooM6elJtpVdqaoKLS/qbzSeam0Ft3qhP?= =?us-ascii?Q?aORmXQsGBgXDR6xCIRH/g7JI8eHF2o8=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: a5df8d04-8851-4821-a928-08dee29bac2e X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:37.1516 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8lgFsbSqtLBC4+IUnGaYQvIAFHsI3gVf8/WOgO3VqczV5+cKDLXm88GlMD0vYZ5b+pXZe2qHgOqbJ6cZXCdfuqXa1VIOmgpkvsqZjN02RUY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" vmap{__vunmap}_range_noflush() assume manipulate init_mm pgd. The following patch will map percpu local mapping into percpu page table by calling them, so the assumption will no longer stand. Make them take pgd pointer as an parameter. Also make vmap_range_noflush() non static, it will be called outside vmalloc in the following patch. There is no functional change. Signed-off-by: Yang Shi --- mm/internal.h | 5 ++++- mm/kmsan/hooks.c | 14 +++++++------- mm/vmalloc.c | 25 +++++++++++++------------ 3 files changed, 24 insertions(+), 20 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 181e79f1d6a2..9d9f08cb26a1 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1552,10 +1552,13 @@ void clear_vm_uninitialized_flag(struct vm_struct *= vm); int __must_check __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift); +int __must_check vmap_range_noflush(pgd_t *pgdir, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, + pgprot_t prot, unsigned int max_page_shift); =20 void vunmap_range_noflush(unsigned long start, unsigned long end); =20 -void __vunmap_range_noflush(unsigned long start, unsigned long end); +void __vunmap_range_noflush(pgd_t *pgdir, unsigned long start, unsigned lo= ng end); =20 static inline bool vma_is_single_threaded_private(struct vm_area_struct *v= ma) { diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c index 8f22d1f22981..e2a0faf344b9 100644 --- a/mm/kmsan/hooks.c +++ b/mm/kmsan/hooks.c @@ -135,8 +135,8 @@ static unsigned long vmalloc_origin(unsigned long addr) =20 void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end) { - __vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end)); - __vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end)); + __vunmap_range_noflush(init_mm.pgd, vmalloc_shadow(start), vmalloc_shadow= (end)); + __vunmap_range_noflush(init_mm.pgd, vmalloc_origin(start), vmalloc_origin= (end)); flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end)); flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end)); } @@ -181,7 +181,7 @@ int kmsan_ioremap_page_range(unsigned long start, unsig= ned long end, vmalloc_origin(start + off + PAGE_SIZE), prot, &origin, PAGE_SHIFT); if (mapped) { - __vunmap_range_noflush( + __vunmap_range_noflush(init_mm.pgd, vmalloc_shadow(start + off), vmalloc_shadow(start + off + PAGE_SIZE)); err =3D mapped; @@ -203,10 +203,10 @@ int kmsan_ioremap_page_range(unsigned long start, uns= igned long end, __free_pages(shadow, 1); if (origin) __free_pages(origin, 1); - __vunmap_range_noflush( + __vunmap_range_noflush(init_mm.pgd, vmalloc_shadow(start), vmalloc_shadow(start + clean * PAGE_SIZE)); - __vunmap_range_noflush( + __vunmap_range_noflush(init_mm.pgd, vmalloc_origin(start), vmalloc_origin(start + clean * PAGE_SIZE)); } @@ -233,8 +233,8 @@ void kmsan_iounmap_page_range(unsigned long start, unsi= gned long end) i++, v_shadow +=3D PAGE_SIZE, v_origin +=3D PAGE_SIZE) { shadow =3D kmsan_vmalloc_to_page_or_null((void *)v_shadow); origin =3D kmsan_vmalloc_to_page_or_null((void *)v_origin); - __vunmap_range_noflush(v_shadow, vmalloc_shadow(end)); - __vunmap_range_noflush(v_origin, vmalloc_origin(end)); + __vunmap_range_noflush(init_mm.pgd, v_shadow, vmalloc_shadow(end)); + __vunmap_range_noflush(init_mm.pgd, v_origin, vmalloc_origin(end)); if (shadow) __free_pages(shadow, 1); if (origin) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 1afca3568b9b..17c26e9796b2 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -295,9 +295,9 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long add= r, unsigned long end, return err; } =20 -static int vmap_range_noflush(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) +int vmap_range_noflush(pgd_t *pgdir, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -314,7 +314,7 @@ static int vmap_range_noflush(unsigned long addr, unsig= ned long end, BUG_ON(addr >=3D end); =20 start =3D addr; - pgd =3D pgd_offset_k(addr); + pgd =3D pgd_offset_pgd(pgdir, addr); do { next =3D pgd_addr_end(addr, end); err =3D vmap_p4d_range(pgd, addr, next, phys_addr, prot, @@ -334,8 +334,8 @@ int vmap_page_range(unsigned long addr, unsigned long e= nd, { int err; =20 - err =3D vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot), - ioremap_max_page_shift); + err =3D vmap_range_noflush(init_mm.pgd, addr, end, phys_addr, + pgprot_nx(prot), ioremap_max_page_shift); flush_cache_vmap(addr, end); if (!err) err =3D kmsan_ioremap_page_range(addr, end, phys_addr, prot, @@ -478,7 +478,7 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long = addr, unsigned long end, * * This is an internal function only. Do not use outside mm/. */ -void __vunmap_range_noflush(unsigned long start, unsigned long end) +void __vunmap_range_noflush(pgd_t *pgdir, unsigned long start, unsigned lo= ng end) { unsigned long next; pgd_t *pgd; @@ -486,7 +486,7 @@ void __vunmap_range_noflush(unsigned long start, unsign= ed long end) pgtbl_mod_mask mask =3D 0; =20 BUG_ON(addr >=3D end); - pgd =3D pgd_offset_k(addr); + pgd =3D pgd_offset_pgd(pgdir, addr); do { next =3D pgd_addr_end(addr, end); if (pgd_bad(*pgd)) @@ -503,7 +503,7 @@ void __vunmap_range_noflush(unsigned long start, unsign= ed long end) void vunmap_range_noflush(unsigned long start, unsigned long end) { kmsan_vunmap_range_noflush(start, end); - __vunmap_range_noflush(start, end); + __vunmap_range_noflush(init_mm.pgd, start, end); } =20 /** @@ -670,9 +670,10 @@ int __vmap_pages_range_noflush(unsigned long addr, uns= igned long end, for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { int err; =20 - err =3D vmap_range_noflush(addr, addr + (1UL << page_shift), - page_to_phys(pages[i]), prot, - page_shift); + err =3D vmap_range_noflush(init_mm.pgd, addr, + addr + (1UL << page_shift), + page_to_phys(pages[i]), prot, + page_shift); if (err) return err; =20 --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022074.outbound.protection.outlook.com [52.101.53.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 82BD54248A7 for ; Wed, 15 Jul 2026 18:05:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138758; cv=fail; b=fAyp+pX8/E7+cDGGw+DpGBnwhiwM9Ha2Qr5/mW1F+3a/xm1+kzZd5/C0DpPS8fi24UCQTr44kRJZ2Gi0CDI7IaIp4MXybXqPUzsFu4mUQZWW6RrcCDJGJZlUTIi62Dqob9cgh0OtI1HMFlcDfeZGbYjgRo9oPG7jOgqLTS3JMxs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138758; c=relaxed/simple; bh=LWEMFGRvVHxwOdg6aCJkQqGxAxQIhZWMYydy+n4e/Zg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=rnLCKHeFuxiyxXegS9kDDwKRnj36TRqgii0upN0Dp4huYcdY9rB3KPMaIM7QvRQUBMaDGN81xpaIloLAvaHr93FleEKhcGVqexfUHGhFS04wggi0eVcPCMtlZ2Jtjh9fjVatSvCWq3dVfBp0YoUn/WF7lr5D9myCU7WoFu427z8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=gnC65t6u; arc=fail smtp.client-ip=52.101.53.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="gnC65t6u" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lL2TPy9UaS1EfpYKmzJW2lRuhJS68W5O7pjR/b3oJ3pvaBVTUV5sKTYJPBHLQN9KIq2ZNgXVypfylXRd2mVAzg9Ko0M2U/1g6wczitmR3CfsSoTrk80UdxV1zu8+CnbHutoFAG2Il9kQIDtCeztXJbZ1z+H8jQtfSQDB9CTF4SNscGABm+Z4G4czMvUgYwNEeKhxpX9A2Vmjgxgw8B+AbGb8/dgKKDzPRIqszw9mizK8thFSGBAhCWUh544W+zm3b0sEA5xIU8dmvEWdhdXgNVNSMkuP92Vqxsc+8dcsS5a8lb7xMYLWIUcARu3KNnuFyzKv1PtNkogZLtqnc3WpWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=abzZe3fsbFI4wiw+EAoWz2guLXEYVn3YimvEHKe4VIU=; b=KihD8Jodx5wENBD87naoEB/Y1Q3FZZSJlW4Iawhj+kZSI2L3X6136Iyh7G+onNGifyAa+NafA7tEj7538NWVX4XRYVy4QSsKIU+wsmWen9SHkkhNkSYWkn1l2xa5KJkzdq1K86rlR8gV2CfEEEz63hUkF4RztC6i8F1C2EYid7XHq05WOqnbq0i7ReFgDT+8stUdlSDvp3Y37UENXIOc87mrp8+zHWDGaCjZUbQ5cUdUoJ2eFPy7Rn8rLHAccCspfdJrkaqE3K3Hq9sSF4wGFjFDwB9u9oCKuHu5LJZoCdychFdYM+K26kU4P+elR5Ik3CBNQ9a7BQkvjUc6YzLoBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=abzZe3fsbFI4wiw+EAoWz2guLXEYVn3YimvEHKe4VIU=; b=gnC65t6ut7hiUcS92BkoXjERrjqiow4/Vn1tJ+5gUQIomGksWSSvYI2CjDbDHD6/nDrpnz+VlIT+iPzyWrCnLPAMrXon40NQd2b80nZgG7z+kPBMw+VqHONd1yKsB7VjWeCZEXeSv3uTiG3R1zPFmL+tp32qv6zEX47o79fZXoc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:38 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:38 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/16] arm64: mm: enable percpu kernel page table Date: Wed, 15 Jul 2026 11:04:07 -0700 Message-ID: <20260715180455.515692-6-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: 404477ca-26c0-4daa-2154-08dee29bacf4 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: 3zR8Piin2QepN4JiBcYBCaZfSAp7AKMP2/ULuNK32f5FQCuoYgkBv/hvZ4FZ+oeUb+09tSL1CjbpnP69E1u+gcObjBdNHHyBX+vpffc9Vk76ZsOcBKOP7q2Vlo80Pwku6NlFQQrliy2sH5a/cdBbNo2QRcbFunZQM4O/9FBa9h1BQc7mDmrlvIrw9i0f9STHwOJwsO47MxE7bhUX/21NLQXZ1q6cO5w1yWv4xMKcBGL3ALo00FUvVfk1dsA2hdEqdNRN50RZjW2MiwyLB5Q668EnlqygEhCrzLRWadXGY6vu7m7hn8IjgxegvpPOKPNVDtMDyvzFOEvtC7otI66F9sxWulbCYr0sF/hKT8sUkpGUL0BWSG0vXkK1p9NVvb/sxuz5wfOAsh1vdGcg4iP39HpsWgG2yVLgLF64qCSkUbhqE3fqoMNx6Cr9vYrNB9U0WpPe+pH9y7oRuMYuMeyPjAjVkjFilzvJoOx6Hx2OLf1TUiQ9xq61pVoics951dgEaFHEzPKOsh8aCKpEeLdQHOpA496fxQEZPqPEyVwuX7IAhrz6dmZPlAPzZCBPsvgW3T0F2sawl5ZVgdoJjU7qhmskftoKkNpgsq1aaz8lkpUtNiaHxG8+LR9m0sg5FDbIn0B9UFN5O+X9iChVVC5Cxlo8cRBxsgjF4q3g0OLMs4Bg9tzsqeMSV9pJAtYf6BmHIGsRRFr+1biC2MPZtJxzXg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?f/2I/Z2wISLVtTovpvhIbrbBw6kmnm7BFw0KMNF12xXP0VK/DQhK8i7nGHZv?= =?us-ascii?Q?s+Y1hKRQAstkh3m2GkNIH8ERzB8ETED7HNmHLlSwYSjg4ZWq9MPwp1f0zhcP?= =?us-ascii?Q?0Dof7ltqmWdxBgg+x+m1wJg9vnZ7TdBW3QDkd/5qsGJrqCIZUtu7TpyhSiQY?= =?us-ascii?Q?eHIWpJbjfjXM8QVh1o37b+jxyvMQtD2UqT+P6G6n+Ha5kK1EAREHwob4eCc9?= =?us-ascii?Q?2WRlwtl9VZuwlPRCWVsvtU7dYgQO1u+RfkemJt9byNO4RqSRK7aI8Syj2JU5?= =?us-ascii?Q?q53KkJst/VhUAGi4Wne5lA4WmINhB36LnPTuvdZWctQhw+WyzEkwBVjs5jRm?= =?us-ascii?Q?lnLupysREmMW+cysZPqy2Afttu2pouP7qtFd+LPlzULhCS4XnEDScx21kz9g?= =?us-ascii?Q?bdFcfm4z6ptTkvABQoOa9T1X+xO6if4NwK0xLU0QeJVSe/nAo8dxYHgf5j1Z?= =?us-ascii?Q?E/xIkCvKe+mfvm6ex/gWGsF24ewOs+z8t5nqYKQPvDxEiyV0zF/B4L33zf6T?= =?us-ascii?Q?cUdcLgf5/oro/CKfwhZefahUDWs78JGjawvXOhz/goI2c2C/+tD4Lyh5+7fO?= =?us-ascii?Q?0/sOXRLRIEsa2G+XtSNHoIXiZFnBaiN8lu3QU3m2njHyaBx+uNl20xH379lD?= =?us-ascii?Q?SHVRmc8EzVawuomPMNuZB5B9lU+IwQjbf87iO+/JPt3Yxpg1fBtGQYFuLG92?= =?us-ascii?Q?uACr5uxVSzM/hgJWK3QmIAmVBi62UZRJJzelT4Yw8nbq45MCNUdK/IeIyQCP?= =?us-ascii?Q?fhD90QH4CC0x/vwg/z7LAtKIidYTgNPqL4jdwrKV/iMWl1Cn/0y6dBjUMNvg?= =?us-ascii?Q?bUHGfb9sxwqzs9jiFn1rghZzLWYgRCxzE93IivAUifrUbmXF3ZLbIB7a4riQ?= =?us-ascii?Q?lcMx7SjvOWTkDlJ/YVhfSaCXXyaRKnfLJYwRqFLD3P4bDosbD7OjLfkCWdPM?= =?us-ascii?Q?QrLlR0JUl74lGnjdhSp5qNMZvBkcZZUbXbXvBh6UtzvdlZuizHtvp7xv237s?= =?us-ascii?Q?xO1ZKBTRwT6+KcV07S0NzQktNvjJwO+/ifmqwihODdlWTmbsKO/bnOnUkQoF?= =?us-ascii?Q?wQxa4HA4utUfWt37btxEqM+A4k1wPEKXKwCruCvbPkHP0mofjO0AKSSuscOG?= =?us-ascii?Q?Xq5WwuiyXZ33NS/xQP3izM+EaWhstop+SxxDyiT9fi2gGBF1rfTh9Irs/7R8?= =?us-ascii?Q?qIeBjx98m8aJmikIMTlkP72OZyF+ZdE2eWk57pA9ZoN2US71Tr6hqgC1SQzb?= =?us-ascii?Q?xLLsdIZFAncpXIubLGdMbExl9/0Puy+xYZ0dnIVPOgaMK8wuyO0x3isjtLwK?= =?us-ascii?Q?jFFG3T/CZxzGY8ETQBu8SlT+b8/oRum2hdMCSwHlQqsB8U1CUpg7DzrJdHTa?= =?us-ascii?Q?sAfpqbNWg5nt3ix0P6mH/wYltDVMe3uOk6PytcYEBAR774nMnOaqVfAG4yh7?= =?us-ascii?Q?fDkCI3JNsgA4/no0SeQUr3JOVfYAdFFaRq7CT/n6uddVPC0CaZFgkaMbhxvD?= =?us-ascii?Q?oziUJUAkR4nMQl52QIatYtNyrTdONiJcfT/cqatjCSdukTH7GCDjiLV8GWM8?= =?us-ascii?Q?pIZP0fTEyoTz11uAADEJhvy3po+FX5rqibDeyERrs97hWisRoTTgEkDiy+Mv?= =?us-ascii?Q?5Fl691bXR5sjWYdASbq2G52rr/4Jus3qzf/n4BmDwrzddSM1Z9WvqCafeOcS?= =?us-ascii?Q?pVVgoDmcOOHpXBJxBUyRvHKP3c0Wf7wOKqArDs4djBErpMbTkBrYKOWHsFEW?= =?us-ascii?Q?QqlktyrGENcHBjxuTe25eZRoLT/G88I=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 404477ca-26c0-4daa-2154-08dee29bacf4 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:38.3836 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +6FOKOHYo5OGal9lq8y1VGpC/OZLMVh5T7TszMhqSjX+VsORxqEwU/HwUlruIcqy6ZKlfuGbG9o/roCcwfCZyTI0ebki35FuEg+mM1XL4Oo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Currently all cpus share the same kernel page table (swapper_pg_dir), this patch creates kernel page table for each cpu and each cpu uses its own kernel page table. The cpu 0 keeps using swapper_pg_dir. All the kernel page tables share the same content. So we don't have to duplicate the whole page table for all cpus, we just need to have different pgd page for each cpu. All kernel page table modification (split, creation, deletion, etc) actually still happens on swapper_pg_dir, the modification needs to be synchronized to other cpu's page tables when the top level is modified. The percpu page table can't be shared across cores, so clear cnp bit too even though CNP is supported. Some features may not work with it for now, for example, memory hotplug, KASAN, etc. They will be fixed in the following patches. Signed-off-by: Yang Shi --- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/mmu_context.h | 9 ++++- arch/arm64/include/asm/pgtable.h | 10 ++++++ arch/arm64/kernel/setup.c | 3 ++ arch/arm64/kernel/smp.c | 8 +++++ arch/arm64/mm/mmu.c | 53 ++++++++++++++++++++++++++++ 6 files changed, 83 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 5e1211c540ab..8ed3b5f3cf84 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -63,6 +63,7 @@ static inline bool arm64_kernel_unmapped_at_el0(void) extern void arm64_memblock_init(void); extern void paging_init(void); extern void bootmem_init(void); +extern void setup_percpu_pgd(void); extern void create_mapping_noalloc(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot); extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/= mmu_context.h index 803b68758152..0e991b98ed9e 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -27,6 +27,7 @@ #include =20 extern bool rodata_full; +extern pgd_t *percpu_pgd[NR_CPUS]; =20 static inline void contextidr_thread_switch(struct task_struct *next) { @@ -138,7 +139,13 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp); =20 static inline void cpu_enable_swapper_cnp(void) { - __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), true); + unsigned int cpu =3D smp_processor_id(); + pgd_t *ttbr1 =3D percpu_pgd[cpu]; + + if (cpu =3D=3D 0) + ttbr1 =3D lm_alias(swapper_pg_dir); + + __cpu_replace_ttbr1(ttbr1, false); } =20 static inline void cpu_replace_ttbr1(pgd_t *pgdp) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 27689c62bd25..13e069b01600 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1216,6 +1216,16 @@ p4d_t *p4d_offset_lockless_folded(pgd_t *pgdp, pgd_t= pgd, unsigned long addr) =20 #endif /* CONFIG_PGTABLE_LEVELS > 4 */ =20 +#if CONFIG_PGTABLE_LEVELS =3D=3D 2 +#define ARCH_PAGE_TABLE_SYNC_MASK PGTBL_PMD_MODIFIED +#elif CONFIG_PGTABLE_LEVELS =3D=3D 3 +#define ARCH_PAGE_TABLE_SYNC_MASK PGTBL_PUD_MODIFIED +#elif CONFIG_PGTABLE_LEVELS =3D=3D 4 +#define ARCH_PAGE_TABLE_SYNC_MASK PGTBL_P4D_MODIFIED +#elif CONFIG_PGTABLE_LEVELS =3D=3D 5 +#define ARCH_PAGE_TABLE_SYNC_MASK PGTBL_PGD_MODIFIED +#endif + #define pgd_ERROR(e) \ pr_err("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e)) =20 diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 23c05dc7a8f2..6d420ad59af4 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -360,6 +360,9 @@ void __init __no_sanitize_address setup_arch(char **cmd= line_p) smp_init_cpus(); smp_build_mpidr_hash(); =20 + /* Must be called after smp_init_cpus */ + setup_percpu_pgd(); + #ifdef CONFIG_ARM64_SW_TTBR0_PAN /* * Make sure init_thread_info.ttbr0 always generates translation diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 28ff889658d3..49524a7176b2 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -55,6 +55,8 @@ =20 #include =20 +extern void idmap_cpu_replace_ttbr1(phys_addr_t pgdir); + /* * as from 2.5, kernels no longer have an init_tasks structure * so we need some other way of telling a new secondary core @@ -198,6 +200,12 @@ asmlinkage notrace void secondary_start_kernel(void) struct mm_struct *mm =3D &init_mm; const struct cpu_operations *ops; unsigned int cpu =3D smp_processor_id(); + typedef void (ttbr_replace_func)(phys_addr_t); + ttbr_replace_func *replace_ttbr; + + phys_addr_t ttbr1 =3D phys_to_ttbr(virt_to_phys(percpu_pgd[cpu])); + replace_ttbr =3D (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + replace_ttbr(ttbr1); =20 /* * All kernel threads share the same mm context; grab a diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f2be501468ce..5a235491f268 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -67,6 +67,59 @@ long __section(".mmuoff.data.write") __early_cpu_boot_st= atus; static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); =20 +pgd_t *percpu_pgd[NR_CPUS] __ro_after_init; +bool percpu_pgd_setup_done __ro_after_init =3D false; + +void __init setup_percpu_pgd(void) +{ + int cpu; + + for_each_possible_cpu(cpu) { + void *addr; + + if (cpu =3D=3D 0) { + percpu_pgd[cpu] =3D swapper_pg_dir; + continue; + } + + addr =3D memblock_alloc(PAGE_SIZE, PAGE_SIZE); + if (!addr) + panic("Can't alloc percpu pgd\n"); + + memcpy(addr, (void *)swapper_pg_dir, PAGE_SIZE); + percpu_pgd[cpu] =3D (pgd_t *)addr; + } + + dsb(ishst); + + percpu_pgd_setup_done =3D true; +} + +void arch_sync_kernel_mappings(unsigned long start, unsigned long end) +{ + unsigned long addr, next; + int cpu; + pgd_t *pgdp =3D pgd_offset_k(start); + pgd_t pgd; + unsigned int index =3D pgd_index(start); + + BUG_ON(start > end); + + if (!percpu_pgd_setup_done) + return; + + addr =3D start; + do { + pgd =3D READ_ONCE(*pgdp); + next =3D pgd_addr_end(addr, end); + for_each_possible_cpu(cpu) { + if (cpu =3D=3D 0) + continue; + set_pgd(percpu_pgd[cpu] + index, pgd); + } + } while (pgdp++, index++, addr =3D next, addr !=3D end); +} + void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022103.outbound.protection.outlook.com [52.101.53.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D33424C0420 for ; Wed, 15 Jul 2026 18:05:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.103 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138766; cv=fail; b=Spa6cEnmHV7s0zEhtU9XfGyoDVG+feSK9yzq9Du+P22bQMfGr/qWlejHaJNFG2BnufxvRCx53ELjh6JdzsHy602I+Lh15OHsKQ6k+8401Y+vJaoekY/mJqglaWIAy8PgoOJowtcf+nnARRfFJXqKhW1jhHGJy1l8lMcBUgnHlF8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138766; c=relaxed/simple; bh=QTl7QpiXE+L2fPLNoPt/HPwOkQStX7lCJhX6qUVTU2w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=papngXcCRo832hTuCgnXNn1gKJk6gJLRb4MIRyhyOUAAtofdWfCa6T+3bm8EX4+AlJuNobq8WjFwysqatIDvA7ROyHbYMRxsh7ulB2115dMMUmV9RychBWi/XhtDeQ7pVm0OZ79HQtBnNSwnmqgaJCoFqkBai9nijQdsKkF8YVY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=F9kdg4g2; arc=fail smtp.client-ip=52.101.53.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="F9kdg4g2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MJ70x4nCZRvHASmk/uP0G2k/nt0QYNITtDZgCveE+Loxj/KBUQTMp4WUsKieKVqLHB8TAn8RgWGTj6B54HsIKBIa8jEkEarTYmbHf2rBUqWXacDy6THyRCLLW4XvXVSiqx8qX/cVFClNoATmnTW273uukyf6YcaT1MZhImAnyhzb0PiB2gRpRcVRAWlA/2flpFI2oqHN5+G/BVAJMgc294w7F5wQvn5IzpcDMsU3pIAgYpXskUyyw6smgmM6kzY+Xg+lcOJURnjS0whkvUkBZkpxrJ4J3Ric67/thuTGL3usLMAgFAANCfc3sNcZrroeU0WynZZRpTZ0bwcCATQXTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2sA1rTbZ2jNxhh/LL5obxZYJeX4N47sBqb4CadZn8bo=; b=ySbf2TF76KmwRd27Th5juVs3T9sEOmRGudjLJuzLSK4oxoKa8hoGhCdK5RFeJaniCB9VRO5wkfutw99ALxJLoTCsSBvvELH1z+0l89JJ56KCOFyzHAh74l7wNren9m3tsaMhjSedpqh3J5MsQgyFqv/W9Iu2+zKtasMCiNPzVazJJRlwzuDvn4o/zD8orDd6oYrwWU5HDNZIrCEZb0A4slXNYC1dWIKTWrgMNQ1G5KuyOglqaCvkryOtvdCpIovuhAHIgETTTyjgMsDzd6NNTSVXF9nPPQ/3MKSxQLhCVgNhygh+erBhETtanJbrB8ebpW2pVpj+sxCd99glfJfAIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2sA1rTbZ2jNxhh/LL5obxZYJeX4N47sBqb4CadZn8bo=; b=F9kdg4g2eVl4p/rztq/m7ZUcrPE+O9gqSfueh+zeldYyY1H97gqbZxQy9eqKEIfqgiLreaqtuLpy76zW6GuxIjqvq2A2PkQRG7kXsTzN6PKOiO4rBcOU83znp1a33dAuyUdCKT8DsNfLa7WT+WyZUrfzrkjFeqn3BCtfzAPqHy4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:41 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:39 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/16] arm64: mm: defined {pud|pmd}_populate_sync() Date: Wed, 15 Jul 2026 11:04:08 -0700 Message-ID: <20260715180455.515692-7-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: 441dc6b4-cded-4a35-d455-08dee29badb9 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|6133799003|921020; X-Microsoft-Antispam-Message-Info: l/+AtzMEL/FYa9qx4EpxdZ+JqNYhGv8crxd61afWUXL8uTcTyhloHLUgsjP+07GHLxJNd/Rj81HopEr8AJRyEyBZIEyniE/ZVI/JSgU2Uyji3oj/hhnDM0YPDW1SyLd/ic97rE8B3KLxs+rIu3c4PtyPwANSZHt3Qhxf8JuPLz0Syi5mKkb8T4ev1Y6EkMK41d2Xf/atsZOj9/oqARb81Tw187KjImmaZpenDQz6aKb2V72AjjjeH8+ZWRKoeJRRQWe+c2L6f2oz+ojC+KStniXklHhY9+Dg8hHVLpN/tDFDw7DwqsXUZ8OSjesKHjlbpjAuQOBcepbPDakCBU8wCZjLif0/0tmz9+8nfhsGwyKa8r1/b7HQZ4AbUb7EbFatN0H2QZcMscDWPseC9KHcWk6XOjN/Ak0XQ8SINaqVFlS9f0srLpZm7OfxuDWYyEedsNUlJS/S7MDPKQDcpETBfoPgSJkHbNf/uBhewrqotfXr2qbCKZBTiv0jZPJ+SJ/R47THEBaVhPDz5N40dsm7iqRQyWPR6xyXR2y9/oZs0W44YSPNLjW8exD2Xqw+m9LucDBxMSKc4Bc+EV8Z5qV0orIj6SqEJDs9kVScIplqTzWKzCHjsrWEQA3ItaoQ8SsaISTyTPF6gjIak2/s1BxsXfR6iyRwtnq4gQiYPrTVc6pJJvlDqWmRlz+3nm8rOYmP21G3rkfeZd8HTPZqKpeDgQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(6133799003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?dnTDVbdQmpwWiBbnXtMPjZnwdwyebhJ/lqH5JcOLgFgC31AbVfG45/RuSGSR?= =?us-ascii?Q?yh4KXmsIgWjtmovlgOaJxT81lOBUonzYHOUAxHfHVY36SLgJ5b55C+osWh4A?= =?us-ascii?Q?0jxHtDOWb/y6YXRq7nGDSPUuXPx6+Aly553GDnW7WAHI+MeJa0sw3v9xP/Iz?= =?us-ascii?Q?+FwYQO86gclyUMynAyP6t499jT8dzmLJZl6cHNRV1DtGE2rTASR5XSjJsRFW?= =?us-ascii?Q?V7ZgFy0MTanCZ5Epx2umwJxfLJ3dYAN95afBLGzFLgyg/gFK9NAKfXhMNubm?= =?us-ascii?Q?mQmH0UmIckD72LUjrFE5PoOnuRCczjs6PTKbCrhc8fHrmhuugXa3xlWRSc0f?= =?us-ascii?Q?N8fOTIXZfVt4uJkObCBFKr8YaQ4eum6z2OO4iQolS4YNX2R0s7y2R9ogwvbs?= =?us-ascii?Q?9Sz3b9nN2JgfIJHK8Rkd0PtIUmqsBn50DdBhnqJlXEo08q933PFClRevZh34?= =?us-ascii?Q?+5m+Xb2W1lqRcEJxr/6ubEXOKbuyRfHt7pCRrHDe4+ApN0YILHSS7tXTMarH?= =?us-ascii?Q?s08roBET3odu/bOS7dtkULVaYTPLXSpwKTz+s+ldJcDFMFZl2oN1lo3A4U6i?= =?us-ascii?Q?KCEb3qJe/R8d3zvi73NDQEEMep15v/1VhB7wJCz74c2MSmQC8dIOlzDQjIM3?= =?us-ascii?Q?rEsbyR/9hzv7tO1/Z8/x16opEYq4TzL6PUvjiMT6bUtWQOEnsctKWN1RHOLa?= =?us-ascii?Q?zI63KPd/78BRvCxdcqabxB3ZQVH1aqUAlJDixta6vrMrjDRJHvpAj+6AeYi1?= =?us-ascii?Q?g87ouk7SkwmVZIViDGjBdRz/QQIlN7dhq5HokbThwgyvo7WvcFWUWwgQ+eeL?= =?us-ascii?Q?+ky60Oov4NmZiJOiTEFZfhzpKUxSev4VcjjAg8ryiMuLom9oOgsfIrHJoS+j?= =?us-ascii?Q?TbMreuaOrVc704xGILbCDHOJaov3EZXMS2Wj4H1JQwshGiA4O7xqvDcwVWuR?= =?us-ascii?Q?vYtinPpAYIzz+beZHe9QstP9h6l48jyqUUPGDzxLrHFxtpSrCkguXrhosUFF?= =?us-ascii?Q?MvzTNFtJUhZZM2GuVyNVs81C8bvMvYdtRak0SvWXzNaB5Bs5jLaRm4lxf/A7?= =?us-ascii?Q?Y2IVvZ2orTnRnRLPIbZ6CM19LQ42onkNdS8our2rax8QsUs7P4Tt2sFmGl/m?= =?us-ascii?Q?AxeCJv7lWka3p33/2wR+hiL0bEmLhw5CmG1bIJUV0Lt0m6Pf4uhcKQtgdNN9?= =?us-ascii?Q?6E+dSGqV/uet0OyD2+Q7j+cPzOAgEMlk//B6QwlKAhokmqc0FvwW2FZmK9ps?= =?us-ascii?Q?FkTzC1t6mAdsXb+CTkQhGO1QOy1VopbdZUHXuVhPk5BA9nmFEt5PNVI/Sht+?= =?us-ascii?Q?QRTlF7KIBBqpakrmsFoaA42xZAh3Hn7y9eNTLgDetT8EIcVkWie0rwQKHaS/?= =?us-ascii?Q?/Y63Rhs3dXo8VyyKfc8e/llp1SvMcCuzc3DNXGt4lqkx/O7MEf0H8ztJWtUC?= =?us-ascii?Q?q9FwZQagGgqhX8/PyhOrEnIlk5cuPejrl4wYHx3n06zr+UbICDRGjAVyVoyq?= =?us-ascii?Q?cnUSbYlcnXKXDNDPDB+6tDsYtmMh5UbCXwTg7+jltjALCMpTYPQw+S23DPQg?= =?us-ascii?Q?5FRidDW9ERcGY3peGIs++J+74Pf79otkzawMpE2UEM1QOBCAX48eZ1dPQQxz?= =?us-ascii?Q?ry8l9oCfE340fMZGtnFonhL2Qx0uYuU5fqlmA1qgr3ckUV67XC3VHd0HfCX6?= =?us-ascii?Q?uNwHZ9dNi2unPcHxlZ5e0KKlwCNjQ3Vqf/CAprjoDcMLOWpjIAvrhugcHXEz?= =?us-ascii?Q?ZcDU488weG8UhCsl9TO3DwxZ3BogLpw=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 441dc6b4-cded-4a35-d455-08dee29badb9 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:39.7382 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uyCroOkm8eysBpenGAyCPehoXy9cgv8/ay7cORmfh9eB609lTSw3Rxkxe4jUU5aFmR7MFvsT3y0rJiDGlCCaknxxptD8HMPa9kWsMZqmwDk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Implement ARM64 specific {pud|pmd}_populate_sync(), which sync up page table. Signed-off-by: Yang Shi --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/pgalloc.h | 24 ++++++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 2e1bd79a579c..23cad9fc7046 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -41,9 +41,11 @@ config ARM64 select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_NONLEAF_PMD_YOUNG if ARM64_HAFT + select ARCH_HAS_PMD_POPULATE_SYNC select ARCH_HAS_PREEMPT_LAZY select ARCH_HAS_PTDUMP select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_PUD_POPULATE_SYNC select ARCH_HAS_HW_PTE_YOUNG select ARCH_HAS_SETUP_DMA_OPS select ARCH_HAS_SET_DIRECT_MAP diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index 1b4509d3382c..dd1ac84a1160 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -33,11 +33,27 @@ static inline void pud_populate(struct mm_struct *mm, p= ud_t *pudp, pmd_t *pmdp) pudval |=3D (mm =3D=3D &init_mm) ? PUD_TABLE_UXN : PUD_TABLE_PXN; __pud_populate(pudp, __pa(pmdp), pudval); } + +#ifdef CONFIG_ARCH_HAS_PUD_POPULATE_SYNC +static inline void pud_populate_sync(unsigned long addr, pud_t *pudp, pmd_= t *pmdp) +{ + pud_populate(&init_mm, pudp, pmdp); + if (ARCH_PAGE_TABLE_SYNC_MASK & PGTBL_PUD_MODIFIED) + arch_sync_kernel_mappings(addr, addr); +} +#endif #else static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t = prot) { BUILD_BUG(); } + +#ifdef CONFIG_ARCH_HAS_PUD_POPULATE_SYNC +static inline void pud_populate_sync(unsigned long addr, pud_t *pudp, pmd_= t *pmdp) +{ + BUILD_BUG(); +} +#endif #endif /* CONFIG_PGTABLE_LEVELS > 2 */ =20 #if CONFIG_PGTABLE_LEVELS > 3 @@ -121,4 +137,12 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtabl= e_t ptep) PMD_TYPE_TABLE | PMD_TABLE_AF | PMD_TABLE_PXN); } =20 +#ifdef CONFIG_ARCH_HAS_PMD_POPULATE_SYNC +static inline void pmd_populate_sync(unsigned long addr, pmd_t *pmdp, pte_= t *ptep) +{ + pmd_populate_kernel(&init_mm, pmdp, ptep); + if (ARCH_PAGE_TABLE_SYNC_MASK & PGTBL_PMD_MODIFIED) + arch_sync_kernel_mappings(addr, addr); +} +#endif #endif --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022074.outbound.protection.outlook.com [52.101.53.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DC5541A57E for ; Wed, 15 Jul 2026 18:05:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138773; cv=fail; b=i6m4EQaWHl52DxbsnCrqE2H+S3AUMSCAUftaOVm+q9jHNw/gcbupDIxWGLduIAwq/1mq49N7LcpwYQh7iaBcbxCh7IOLFK3uEdKHAFJn27yiQAu2K/IwQD7NDooCy8YhO60Py8pCBoAdr1Pxzc4LB+bdAsHGRN8SlOf27tv81jk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138773; c=relaxed/simple; bh=DYFQj7o4itcgZorIHAvnNvUIgxLTjsLUtLwqCBtFJQE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=lNSs6R8b3WUR7gpYJnFEiB6tUt+vrRvIktTFoSSBAZZp8YzdCsFaep6QpHS/0BXODdu2S8gsIvuoBCFvJNNgxVYvljg58J+7lWhIDi1OWjc9YgVZf2OMM/Lpf6TntCQ76wAXSbd2GPOuOBgok4qDsaTq1lmXGJZcqdF3cui1Fjo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=ErYp7ioY; arc=fail smtp.client-ip=52.101.53.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="ErYp7ioY" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rx65cVkxA7MptQmmJhZAV5bex9athqgAdRRG5ZIVIf2esZJjnZmsRVXoo8zV/ObmarlwYm5iSVFOXj3Eds2krLiQob/BFEUokK03hCAFufs1SBCV+FmtCNvePAoE9ZgTbVzYQ4VGxvN9UDAZjVFUGmbXZNvh/RHFDjgvQcNvajCMK0HUXvTnWyzfoywP1rvONpfVXjuz7NU85ErSE//yhyQKt1U+weaxNJbtQFWAXOB2mXcM2+V5Of6Rm1N1wMwq8Ipd9hSNO+sC8SqP9PUP3riiTb7/kIi859YwTIFZfEX9SM0Q91EveCc3OuuzzRhO9/652yvViGsqYx3b9P7tpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P5u3Ui48riKUdMAO/FmOhP2DfS5AnkZBgM9oG/Y3LyM=; b=vgJkfanLXWh/+pAroU7Jw3T2mG7UN24m8K9sIEJNKSppLzN/HXq3lru8/lgTpp4UYZoWim4l0vXY7I2u0GacgBy1ASfOWFv4+QEnLBzPD+lJicfCFmfPs5uBc0el8kihDSmVFtpKC1ohh7aQUk1NNkmT2t4joio27d2RwgLA7ZQd10KxKAq8yQ6CYoQHA/CDvPtz75AepyznZfylFASawgOlFpw4zWbE37NN1bTQV8PrvM4mXJIJ8dci2nu/46yK85q2w6aAcKpGRhTZ6xMKxS/gPCEMN9JMbkv7AbY5HgX01DM+Lp4f9wLNInzdK6mC1cl/iTf6oa4MCD9tzVGRRw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P5u3Ui48riKUdMAO/FmOhP2DfS5AnkZBgM9oG/Y3LyM=; b=ErYp7ioYxq+epTH8IZCyneSZopH58b2oo3+Eni3/XAIqnxid1AFZwnGbFtNkMuf0PhKjB0ZgJ7uHZo8ezVvYSIIw0JiWziyGMG2CihM/uV1UzrvA64uB5gRfPDh7wUaorAb/2VdPONlRp3oxMkxldJlyN2N1ogbsPNIdojJ1Fyo= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:42 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:42 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/16] arm64: mm: sync percpu page table for memory hotplug/unplug Date: Wed, 15 Jul 2026 11:04:09 -0700 Message-ID: <20260715180455.515692-8-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: a7f1a099-35e8-4495-fc6d-08dee29bae80 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: Byry+Fa+vQBaAwLGgOjBW5Bp7bkPRZWRpQ/+jjbuwktMlW7x9yyKLz7xkFGctt89SlvfGi4FZRoYCnscyCRh6/5EYaOSWKA+zCUrEGD0CZ9LKaQ0IbHQPhQhbIrY09Wy+E3/0QZTcA2GWz24PzXeaQeNsat8ATFiarx+u21dPbSU6I/yz/mLTKlnCH3v/Omuub3LtwVyYohecZEgLlnpCnsWhxHwvn5ffpvSd7qAFovSgsVoifWP16X28yeXdJgneE8sKGIOxqUuih9snPB4LpmkfakGXX5JqXNXhyDZipAqrdeN1/OQw0xWNqqAoRzrNyGn95sHd544uhyIQTF2XY2ffqtVy3/S6YomX4KEerbzfrBvAOTBchp7rR9Fw2cUzQtao/xqq07G+4fh3DfiGjgmvwpJFBhic9Pq3fADQc3I5z91mY2kYkrqUeHQRQJrobgaAa18jJPMuStH4o836BkQxGiDF81rouPHqnrC3La6Lq95hP/r499dTiI+hRLFly4vR9P3i5pna2+oBKgcu8yzmGgxrciPADZPhPeKMNGNnrDhTBR+vOMIjld0NrOfy4qQjAkLOgK+yxMpibeYLDCsPAkfqYK6AdgnlC4tESd3CAs6YvLTc0N6dt4T423+LPdORaYgBilCXEUMs47cRTpOGbIfuksZjXer7ukcDLMUMb6Eaqy2sE5W6l3Sk222GUG1oNyaLK8GiKUTsMSN3g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?fus4iSpsDqTMRF3snIBqfKxcvFC9irtMpadVcOGRKYCfqXxSnkWFzJba9j43?= =?us-ascii?Q?j5Bf2q1zCROn/XiWDB4KttS3UDL4ezRkDHpQFBKhwcwNCxrbE7clOe38+Fru?= =?us-ascii?Q?ujqsETPiFKpZ52gfZfxUvcoYfucM0QiXxLrwQsCXsoWT+ywMG3fitKln7ew9?= =?us-ascii?Q?1pjU3rlp7MRxYTL/rRB5ppCeSjRqxvYYvkBTx//vXGx/DFLcMn+Jyf9yU9Tl?= =?us-ascii?Q?i0HoiOgvM9Nu4NiKZ2ZIZF6QIGpkjC/7RKgKAxEszZY9ribhHexrOnUk8jfX?= =?us-ascii?Q?y/PYizAjag0jmlK19SjlAxuD7y4WU0oZ2wZL7xhHFk/60GzrNgbRydnng+Yt?= =?us-ascii?Q?AQHPjJzIbz2FmnCBBPbUQMhJhooaJHJmt+pghO2p8Pjrgk0euJCGh650WKOQ?= =?us-ascii?Q?nIXYnq8y1X0XyOq/kbXH+F0xkdQ/uBnuz8ub5r3WEaRFs2TzKdcKV0oRdiZP?= =?us-ascii?Q?BpfM+/plhIOXtQRJrsOJa/JAJD0CJJJ7Vfilc3g6JlL+EK3/8rcwzz+nLtmC?= =?us-ascii?Q?CxbPk2DC+8y6idG8B5CUeNHoDaevggYymWvz3TU5HeuAzbCy7ZB6MFBSKfSW?= =?us-ascii?Q?AnWToDtHp5P2QPpRKFT8fbd7Wq0mMk3Zy+ACGwacwwUwITixhxfkUOfU+nGA?= =?us-ascii?Q?dnxRt8PWhLS6Xc9Dl+UoKUDedOXV6lSnQCQLpm2VseKc3JTIMO4bO2ceEjk7?= =?us-ascii?Q?LaSEonZcbc+M2W3Eo/FEB3oZJgETl29Ustzv+eGP/PsGRbRduHmTutfswH/G?= =?us-ascii?Q?yboG0PtMHBQD0rf83gm0bCX15jNeGPCuagOadYclQrlgg1CzBhhY3R+5nDb6?= =?us-ascii?Q?3f7To+MSSPNftu5i9h/9t4Z6URyLddNXGLU39HnZ0QnyDLus1hLGS4RkDKni?= =?us-ascii?Q?tV8mCUUCoxwrmnl9wRHHwamtyGc6I9ChYLYPrBJB2FaI/2S3XJYjpaDeatGU?= =?us-ascii?Q?T2CvQVZINQmlYtUC0FfmGpkmSeG71ECjKjAbK46BTd0SEFXBOiIAHdDfknpD?= =?us-ascii?Q?GGt23+RUv3jCnlTq2qeRL3+7nSNsS1ZvDLeJNZeIThliIVKfxxVZPJWrKie9?= =?us-ascii?Q?2B/CHXip5z8wq3RsBZ+jY1bQGdJh9SAYuuD+0LK9ii/oSxG74/jAaheGN/lx?= =?us-ascii?Q?5PZpglE6Lc7IDHXPX3duvkKKVwcRk06qxT3rtI7YOMCz0QKbd+Mitw3zXWlC?= =?us-ascii?Q?tKghYLQLALX8HQPdIGjiLsy/vEZ80Kj6zFBqtBfrKcSSmF5AMWOLb/WOip9V?= =?us-ascii?Q?tDFisa6mv0TGcJNki9MkOCFiXsLLX/Q5cfTEo3R+ZKnDvo7VBnR4nZ+Hd4aX?= =?us-ascii?Q?l3TR5aoBqBxAYaTHNLXfN12naqRDUD5O0FCGntSvIEggr6JHr6DPygKH1oBs?= =?us-ascii?Q?8esoDs3MUb76g9KZNl3AV7JnN5KvFb1IZA2ndFz97Jvn7M/yyk7mfGhT4rvu?= =?us-ascii?Q?jap8BwMURxC2uPeGbQgip14r+FZY2OyEtgDinh1ho/wQEh0r5G+eqgzIykfX?= =?us-ascii?Q?B+V4eWRwnz1189wcDCh0cyECs5HEe5xMZZr2RIMng+YiJqQGST3dZYS42f9f?= =?us-ascii?Q?tp2PZkqKsWhDIxJ1zqxXJ3PJzY85F+LHAccMnl18UoH5IYsUNIjTm/15F9Vm?= =?us-ascii?Q?ssXJtqrYxnEzUSX64x8ZcVwha8GKQ+eCunlDXM6M4gIC4ltPjK6EDwIErp7f?= =?us-ascii?Q?6HCEYAC+Nu0C95VpggfYfDGdVBdiCVI3WgCGfBkPLI9jU2smqbX3XcRdvvR9?= =?us-ascii?Q?x47eQ7kFwBuZ71FfNXxi+1Rq7fMOHGw=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: a7f1a099-35e8-4495-fc6d-08dee29bae80 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:41.0561 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: d3TsiLXqp5z/1LGO/i0O/8fghtlkkn24kEHm4T71gO6uA7QEzCCvUNRqv2PUiRpSQh/udoYhRTsYIT9rpeN3ki/j9VKJ9003rDFnkAT50cw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Need to sync kernel percpu page table for memory hotplug/unplug, including both linear mapping and vmemmap. Signed-off-by: Yang Shi --- arch/arm64/mm/mmu.c | 85 ++++++++++++++++++++++++++++++--------------- 1 file changed, 57 insertions(+), 28 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 5a235491f268..e1290ccaa5be 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -214,7 +214,7 @@ static void init_clear_pgtable(void *table) } =20 static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) + phys_addr_t phys, pgprot_t prot, pgtbl_mod_mask *mask) { do { pte_t old_pte =3D __ptep_get(ptep); @@ -224,6 +224,7 @@ static void init_pte(pte_t *ptep, unsigned long addr, u= nsigned long end, * are deferred to the end of alloc_init_cont_pte(). */ __set_pte_nosync(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + *mask |=3D PGTBL_PTE_MODIFIED; =20 /* * After the PTE entry has been populated once, we @@ -251,7 +252,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned lo= ng addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(enum pgtable_level), - int flags) + int flags, pgtbl_mod_mask *mask) { unsigned long next; pmd_t pmd =3D READ_ONCE(*pmdp); @@ -272,6 +273,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned lo= ng addr, init_clear_pgtable(ptep); ptep +=3D pte_index(addr); __pmd_populate(pmdp, pte_phys, pmdval); + *mask |=3D PGTBL_PMD_MODIFIED; } else { BUG_ON(pmd_bad(pmd)); ptep =3D pte_set_fixmap_offset(pmdp, addr); @@ -288,7 +290,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned lo= ng addr, !pte_range_has_valid_noncont(ptep)) __prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); =20 - init_pte(ptep, addr, next, phys, __prot); + init_pte(ptep, addr, next, phys, __prot, mask); =20 ptep +=3D pte_index(next) - pte_index(addr); phys +=3D next - addr; @@ -306,7 +308,8 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned lo= ng addr, =20 static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(enum pgtable_level), int flags) + phys_addr_t (*pgtable_alloc)(enum pgtable_level), int flags, + pgtbl_mod_mask *mask) { unsigned long next; =20 @@ -319,7 +322,10 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, u= nsigned long end, if (((addr | next | phys) & ~PMD_MASK) =3D=3D 0 && (flags & NO_BLOCK_MAPPINGS) =3D=3D 0 && !pmd_table(old_pmd)) { - WARN_ON(!pmd_set_huge(pmdp, phys, prot)); + if (pmd_set_huge(pmdp, phys, prot)) + *mask |=3D PGTBL_PMD_MODIFIED; + else + WARN_ON(1); =20 /* * After the PMD entry has been populated once, we @@ -331,7 +337,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, un= signed long end, int ret; =20 ret =3D alloc_init_cont_pte(pmdp, addr, next, phys, prot, - pgtable_alloc, flags); + pgtable_alloc, flags, mask); if (ret) return ret; =20 @@ -359,7 +365,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned lo= ng addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(enum pgtable_level), - int flags) + int flags, pgtbl_mod_mask *mask) { int ret; unsigned long next; @@ -384,6 +390,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned lo= ng addr, init_clear_pgtable(pmdp); pmdp +=3D pmd_index(addr); __pud_populate(pudp, pmd_phys, pudval); + *mask |=3D PGTBL_PUD_MODIFIED; } else { BUG_ON(pud_bad(pud)); pmdp =3D pmd_set_fixmap_offset(pudp, addr); @@ -400,7 +407,8 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned lo= ng addr, !pmd_range_has_valid_noncont(pmdp)) __prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); =20 - ret =3D init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); + ret =3D init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags, + mask); if (ret) goto out; =20 @@ -417,7 +425,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned lo= ng addr, static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long e= nd, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(enum pgtable_level), - int flags) + int flags, pgtbl_mod_mask *mask) { int ret =3D 0; unsigned long next; @@ -438,6 +446,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long ad= dr, unsigned long end, init_clear_pgtable(pudp); pudp +=3D pud_index(addr); __p4d_populate(p4dp, pud_phys, p4dval); + *mask |=3D PGTBL_P4D_MODIFIED; } else { BUG_ON(p4d_bad(p4d)); pudp =3D pud_set_fixmap_offset(p4dp, addr); @@ -455,7 +464,10 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long a= ddr, unsigned long end, ((addr | next | phys) & ~PUD_MASK) =3D=3D 0 && (flags & NO_BLOCK_MAPPINGS) =3D=3D 0 && !pud_table(old_pud)) { - WARN_ON(!pud_set_huge(pudp, phys, prot)); + if (pud_set_huge(pudp, phys, prot)) + *mask |=3D PGTBL_PUD_MODIFIED; + else + WARN_ON(1); =20 /* * After the PUD entry has been populated once, we @@ -465,7 +477,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long ad= dr, unsigned long end, READ_ONCE(pud_val(*pudp)))); } else { ret =3D alloc_init_cont_pmd(pudp, addr, next, phys, prot, - pgtable_alloc, flags); + pgtable_alloc, flags, mask); if (ret) goto out; =20 @@ -484,7 +496,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long ad= dr, unsigned long end, static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long e= nd, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(enum pgtable_level), - int flags) + int flags, pgtbl_mod_mask *mask) { int ret; unsigned long next; @@ -505,6 +517,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long ad= dr, unsigned long end, init_clear_pgtable(p4dp); p4dp +=3D p4d_index(addr); __pgd_populate(pgdp, p4d_phys, pgdval); + *mask |=3D PGTBL_PGD_MODIFIED; } else { BUG_ON(pgd_bad(pgd)); p4dp =3D p4d_set_fixmap_offset(pgdp, addr); @@ -516,7 +529,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long ad= dr, unsigned long end, next =3D p4d_addr_end(addr, end); =20 ret =3D alloc_init_pud(p4dp, addr, next, phys, prot, - pgtable_alloc, flags); + pgtable_alloc, flags, mask); if (ret) goto out; =20 @@ -536,7 +549,7 @@ static int __create_pgd_mapping_locked(pgd_t *pgdir, ph= ys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(enum pgtable_level), - int flags) + int flags, pgtbl_mod_mask *mask) { int ret; unsigned long addr, end, next; @@ -556,7 +569,7 @@ static int __create_pgd_mapping_locked(pgd_t *pgdir, ph= ys_addr_t phys, do { next =3D pgd_addr_end(addr, end); ret =3D alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, - flags); + flags, mask); if (ret) return ret; phys +=3D next - addr; @@ -569,13 +582,13 @@ static int __create_pgd_mapping(pgd_t *pgdir, phys_ad= dr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(enum pgtable_level), - int flags) + int flags, pgtbl_mod_mask *mask) { int ret; =20 mutex_lock(&fixmap_lock); ret =3D __create_pgd_mapping_locked(pgdir, phys, virt, size, prot, - pgtable_alloc, flags); + pgtable_alloc, flags, mask); mutex_unlock(&fixmap_lock); =20 return ret; @@ -588,9 +601,10 @@ static void early_create_pgd_mapping(pgd_t *pgdir, phy= s_addr_t phys, int flags) { int ret; + pgtbl_mod_mask mask =3D 0; =20 ret =3D __create_pgd_mapping(pgdir, phys, virt, size, prot, pgtable_alloc, - flags); + flags, &mask); if (ret) panic("Failed to create page tables\n"); } @@ -1332,6 +1346,7 @@ static int __init __kpti_install_ng_mappings(void *__= unused) =20 if (!cpu) { int ret; + pgtbl_mod_mask mask =3D 0; =20 alloc =3D __get_free_pages(GFP_ATOMIC | __GFP_ZERO, order); kpti_ng_temp_pgd =3D (pgd_t *)(alloc + (levels - 1) * PAGE_SIZE); @@ -1355,7 +1370,7 @@ static int __init __kpti_install_ng_mappings(void *__= unused) // ret =3D __create_pgd_mapping_locked(kpti_ng_temp_pgd, __pa(alloc), KPTI_NG_TEMP_VA, PAGE_SIZE, PAGE_KERNEL, - kpti_ng_pgd_alloc, 0); + kpti_ng_pgd_alloc, 0, &mask); if (ret) panic("Failed to create page tables\n"); } @@ -1661,7 +1676,7 @@ static void unmap_hotplug_range(unsigned long addr, u= nsigned long end, =20 static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, unsigned long end, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, pgtbl_mod_mask *mask) { pte_t *ptep, pte; unsigned long i, start =3D addr; @@ -1677,6 +1692,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigne= d long addr, WARN_ON(!pte_none(pte)); } while (addr +=3D PAGE_SIZE, addr < end); =20 + *mask |=3D PGTBL_PTE_MODIFIED; if (!pgtable_range_aligned(start, end, floor, ceiling, PMD_MASK)) return; =20 @@ -1692,13 +1708,14 @@ static void free_empty_pte_table(pmd_t *pmdp, unsig= ned long addr, } =20 pmd_clear(pmdp); + *mask |=3D PGTBL_PMD_MODIFIED; __flush_tlb_kernel_pgtable(start); free_hotplug_pgtable_page(virt_to_page(ptep)); } =20 static void free_empty_pmd_table(pud_t *pudp, unsigned long addr, unsigned long end, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, pgtbl_mod_mask *mask) { pmd_t *pmdp, pmd; unsigned long i, next, start =3D addr; @@ -1711,7 +1728,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigne= d long addr, continue; =20 WARN_ON(!pmd_present(pmd) || !pmd_table(pmd)); - free_empty_pte_table(pmdp, addr, next, floor, ceiling); + free_empty_pte_table(pmdp, addr, next, floor, ceiling, mask); } while (addr =3D next, addr < end); =20 if (CONFIG_PGTABLE_LEVELS <=3D 2) @@ -1732,13 +1749,14 @@ static void free_empty_pmd_table(pud_t *pudp, unsig= ned long addr, } =20 pud_clear(pudp); + *mask |=3D PGTBL_PUD_MODIFIED; __flush_tlb_kernel_pgtable(start); free_hotplug_pgtable_page(virt_to_page(pmdp)); } =20 static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, unsigned long end, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, pgtbl_mod_mask *mask) { pud_t *pudp, pud; unsigned long i, next, start =3D addr; @@ -1751,7 +1769,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigne= d long addr, continue; =20 WARN_ON(!pud_present(pud) || !pud_table(pud)); - free_empty_pmd_table(pudp, addr, next, floor, ceiling); + free_empty_pmd_table(pudp, addr, next, floor, ceiling, mask); } while (addr =3D next, addr < end); =20 if (!pgtable_l4_enabled()) @@ -1772,13 +1790,14 @@ static void free_empty_pud_table(p4d_t *p4dp, unsig= ned long addr, } =20 p4d_clear(p4dp); + *mask |=3D PGTBL_P4D_MODIFIED; __flush_tlb_kernel_pgtable(start); free_hotplug_pgtable_page(virt_to_page(pudp)); } =20 static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, unsigned long end, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, pgtbl_mod_mask *mask) { p4d_t *p4dp, p4d; unsigned long i, next, start =3D addr; @@ -1791,7 +1810,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigne= d long addr, continue; =20 WARN_ON(!p4d_present(p4d)); - free_empty_pud_table(p4dp, addr, next, floor, ceiling); + free_empty_pud_table(p4dp, addr, next, floor, ceiling, mask); } while (addr =3D next, addr < end); =20 if (!pgtable_l5_enabled()) @@ -1812,6 +1831,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigne= d long addr, } =20 pgd_clear(pgdp); + *mask |=3D PGTBL_PGD_MODIFIED; __flush_tlb_kernel_pgtable(start); free_hotplug_pgtable_page(virt_to_page(p4dp)); } @@ -1821,6 +1841,8 @@ static void free_empty_tables(unsigned long addr, uns= igned long end, { unsigned long next; pgd_t *pgdp, pgd; + pgtbl_mod_mask mask =3D 0; + unsigned long start =3D addr; =20 do { next =3D pgd_addr_end(addr, end); @@ -1830,8 +1852,11 @@ static void free_empty_tables(unsigned long addr, un= signed long end, continue; =20 WARN_ON(!pgd_present(pgd)); - free_empty_p4d_table(pgdp, addr, next, floor, ceiling); + free_empty_p4d_table(pgdp, addr, next, floor, ceiling, &mask); } while (addr =3D next, addr < end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); } #endif =20 @@ -2039,6 +2064,7 @@ int arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params) { int ret, flags =3D NO_EXEC_MAPPINGS; + pgtbl_mod_mask mask =3D 0; =20 VM_BUG_ON(!mhp_range_allowed(start, size, true)); =20 @@ -2047,10 +2073,13 @@ int arch_add_memory(int nid, u64 start, u64 size, =20 ret =3D __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), size, params->pgprot, pgd_pgtable_alloc_init_mm, - flags); + flags, &mask); if (ret) goto err; =20 + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, start + size); + memblock_clear_nomap(start, size); =20 ret =3D __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11020121.outbound.protection.outlook.com [52.101.61.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0768C431A53 for ; Wed, 15 Jul 2026 18:06:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.121 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138780; cv=fail; b=HYV518GaO4WjyHHTVN2I0awIgkmlJcP1L7d5hg+8tlAKM8l9BTSBn0WuU/tDxFv5SMV0tjpZ71N0IH2TFo12+L/CEJkfaNarF7/IynC+Zu6kfyIKbkigY64OrSIFPTsYV8CWdvqrqkXD9Q981SMIKhufrfK9G1S11mOm9iUEiGk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138780; c=relaxed/simple; bh=TzVELhxPMfnL/OehTljwPF+BV7KmH6LALph2L3yva3M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=rrAzseh8047Qpw4YAAXHUIH8PlTTKre61USzeCC3Y1vGP7l9n8YYE+pOkzaBHb97R7286Qr1ivFfJ659ARIJkQiq/NJQ+SSS6V5tU/TkbeuHsxs04P4ok2QDjx4xvACNgf27VTC10TGqY+xsC2xECpsnucEmcJt1rjs/+Gs19II= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=BP2uW6Bm; arc=fail smtp.client-ip=52.101.61.121 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="BP2uW6Bm" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OKdAceXhiCgqsSSFz5htt62NP4YcvKijVthAUtPedP4DiKhHExKkai86muNYbDPD/3ddTkrpxEBmEcvic95UJDVMdBUhDt+5+Nv5+6YjM6Ad26jwm1BCaRjH9bE65V2AbIH1evNpMUrRCyKOtvHieqnQjZHD3YwBUROECX1tcpbmS019uAlxGR1nZK2GFQT+gSnn6sT4o+4O/F6spNrBblSgvlZHflmQOp4liWlpZhPiRU6cl0J6HsaRKngdNl668nBH8kgqsBl2MDdT4OG8r22LewkQ5GcWq36mo74iMoJDp2Sei7BvdguKpHnfod+KmdZeTdVH8O77aR/kobOX/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fKHtFe2JRdiFgYL/hT6/kLrYk/Tcs7+w/tnxoXje3+U=; b=oI/wgCuWi/tZlenEp4cITk2FbRdnjGN1uVHmJ5I2Tlmxx/tJ+5AGiaYmX/lsyguf4IWDKGwlvkfHOJORp/RGA3pC9jrtr8CjbTZfaR8/E7x93UUvNyTfngU+Cy9zcmpRneUEuAcF6UVjf/fCJESzrrvlV+SdGVHth3kSLJS1tepHqnO1zrgwam1zqxDaqHVYn1BNuY7PHjz+EbP9mLuF1RUGpAWFJ3SBSJkZW5k7RFIfo1GqBNPd5H1DUj4gHfUpVV5iuk6FYy7Xztu1ISK8n2DNjJHLT7TkPEvK14xekDi0h59vqARJ+3eDW//dy/zKg68Wk2roFRKdEFZDHOkpxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fKHtFe2JRdiFgYL/hT6/kLrYk/Tcs7+w/tnxoXje3+U=; b=BP2uW6Bmd6es7861eHkuUYqo/meHoQIG+ReTVAwAYQEn4O/yHIPYU55FUyktx7E5LbPqaHI04ZZBMe7qUKtcL1ywZg0PLwHAKUO+NqJvWvIRkbfDB2csK922EJfKHKX4oMSn8/CARCW0Yu+73RSUY8dCvw5Vhveq38OAakBsCXc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:42 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:42 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/16] arm64: kasan: sync up kasan shadow area page table Date: Wed, 15 Jul 2026 11:04:10 -0700 Message-ID: <20260715180455.515692-9-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: 9042f7e0-994b-4322-362d-08dee29baf4d X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: /xEU/x1rDAVUJnpNMu92877BYkqXJ0CGc6EUEQs4vYyZWl3cEJJvH1tjvjG76k4gFv+nP6lsjvzCSxqGubdodLVYcN7VdnRg02z29IjTSlBRlCFteIopvaY847eTeFFGRX5DDuHIjRL5EVSlaVr40qQ4HDJ4Qs+oGFAgaVIkdSkGLoEO4sx2pefsYbskSOZkLRPWUPszQf3sCVIDMiK4FWtXMkXDFby5BH+kk3w/QsafxfHVhl/RxfRlHZ49CtPfa8b+BsqcvNf0KDhuiFAgkbq04tA6L1jqGRNhFb8c2KDKrlBWqHohIb2O4GsOWLKdH2Ej7FeP3gUlmQb5930Di2UBxiBdVPiWP1RdmQibaxo3sxOgNMiSbwTG104UMsoXya6VPq0JGIojSixCV/kF2NDlOazGBcmXFvYcAHCIFxfW9VjfVMchXIiRLnJPkBMiCdwffZKVtcetDs604C4ujmg0gT5GPf/m1knsOoVN3Bkwx0wLKvaFMFMbWh5/lvhvhedcYA3AorXlek6hAEQkIVYx8hHVj2bGEOpb1/RukUJoLXnfIlR/ongFjqpC/bsZzPrv35N/Z9EJEWJEWIXjjeENnluMV0K6EusVbqyUuBe7Z4/FkoLnSSdeej9W7eKdRYZqBhzI8ywyp9JBTLvJ9OPGPDoosLkZ7dxf2SC9fptjpIvX+QJbe4O8ZlE39QviH2n2gooTZx1pQpSFXERPug== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Z9mGosFjWDo427XvGmSO9sM3LDL87roj1xKAOWiUO5z9c8eYPS/X6NQp4AAc?= =?us-ascii?Q?xB9lwfj3kn5XrvJZHAyBwA1psowLhTpFLeG5Maib4RXBMY4yKGuud76ResFT?= =?us-ascii?Q?LyP9s26JONjPeGtXGxnLgBY2doJAyD1dkoFulIpNE7iDwlLrOyZF8sI/t6T+?= =?us-ascii?Q?FCinKtKjGwCgwokHB5kC0l9x1NQaAJjqWgWcxK9CfAiIFlz1TTw8+suVu4yh?= =?us-ascii?Q?4fV4CzxMliydDIR+0RBw0lq/GJeVJSk/KypFL2BvoN8hSzQOehaFdMAG9hvZ?= =?us-ascii?Q?omonlDAFKJOsI3UZUbiEowgwQqnMf3LhEFqOxwoBGaTevewfxtfMn9drLrUc?= =?us-ascii?Q?27TENKAC1pvM3gdax1dEK429Ri0oC998mvtLd+YdfX5VcGAls/vq9h03FW5i?= =?us-ascii?Q?IB4FlMqnKjQ4dcHEkkqhpHEqh9xdl6QY6L4hV/kAb0n8QrOCTYZudbfquGeq?= =?us-ascii?Q?C6AnZRYFtqptRPISBVPYtGu3GdPTvQ4UM4VwifjRQPiqbTw+tRFY5DnuEEJ1?= =?us-ascii?Q?mnzGr4t0ISSG42d09el4w8qaOfjuWvp1Bi3KAOj6RMLKDaTeuk43rWW8VVK2?= =?us-ascii?Q?cH69cycJxd5NKIUg9gB1KBIyaytqHAlJAEyf6ICdq4gNv6SRySCLKdHEkdcY?= =?us-ascii?Q?UiNPXp/W/zw6EgI3FPRG7Xf58Ez7hol769QryfYwXMjNuxyc58o8TTGuzYZp?= =?us-ascii?Q?cq9tfpzJqGpBufRmbyknkJk+Ktoe2EkciNkFJATdPja6mMpy69pnr+4W/33Q?= =?us-ascii?Q?B+wTZNkgSFU7sEewhI5R0AkdG4GnvYfORFxj5nrXaRrBLg7Q9kT9I9LtzYgz?= =?us-ascii?Q?PHKACuwlC8LUaR6wpRJmZfjg66DxWinjNESUCnm3DATuczWUOJnVFHztc63T?= =?us-ascii?Q?LXjPz8gi1UFZ7mV1Y6IB88euDdpyUcvAYWLfUU/N8UWaB9oGEfO0+ik2In4b?= =?us-ascii?Q?D0nz6mt/9FNVZsH8mKtDITy6npTeApYS10nagmff+R9JNrTwr8WorwrIM7nE?= =?us-ascii?Q?UUVaJog57omtf34zS7tzzP1+6Ha+HTgTH9NaXAcE5Nnupx1iz2omSbqzmU6F?= =?us-ascii?Q?C6jo6lNKVDqIx2gKkdTHt4wspcYYhGRFdGo54lg570Dg77ObpDX1Any1EUCW?= =?us-ascii?Q?BXFv8oNRIzrlTpYrJGnaoWuEDQH03gB6qBFo59lBJ9OaXn/QpKgYmxrzuhiV?= =?us-ascii?Q?5Svt75LdpjOKOJfao2lANpDAxRo1zXiq48w9N0EI3cIBuRtKgWZGsWnm7gA7?= =?us-ascii?Q?jlyNqBmRtVFqUTwneITprfN7YGkGUxUaen5XsvolWWICq7q4qbgjjE4A9Jtl?= =?us-ascii?Q?rXhbKogvCNMoYI6Y7ZcVSYQBG1sS9fDP87I7t8SVK4vKEp74N/9yplGCLoiR?= =?us-ascii?Q?+/gqbh47C8pGhChOZwBvQnYGBZ/WkoViMKsD2JjYOPkf5GJybYWPGw9L6c22?= =?us-ascii?Q?taTAsrCL+LJJWhJxhs0Vc8wQTvOU5j5CdC/0Q57pSHTUfN0o9FNDGh/UyFMo?= =?us-ascii?Q?LEiXEaAckBlKDvh7Yj4Thj/lqjjVX7Jr350ClxDA+ayNGZa8N+8M6nlCpDZP?= =?us-ascii?Q?UEd11G78Uu+Iw48APiJAqJD0qFkRQg1u8mTk4L6Au5zjsiferbuZxzOev2IF?= =?us-ascii?Q?ywvDpn9fDJrjc9kZWvghtnMdaP1lE6LIfSKqVbepRtjgOLncZ3mwOvNDkyrI?= =?us-ascii?Q?s6Y9O4bNDIIz8fn+0lqQZclAkk/seuGqq6QZFm7Til8cD6vH3MtVDUJilM+A?= =?us-ascii?Q?fFpyKK7e4bXLb3oulrCRr02ypm8o0GU=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9042f7e0-994b-4322-362d-08dee29baf4d X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:42.3502 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wUrsDlyGSD/Xluqf7sFBryFFSvrsTOMI4RfjijX8FlGjKc3xIr8tZgLk3PqNWoTla+HLKTLtWMqeQ9ww+nlsQN9EVXqvgJdwwHZZlULfl0U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Allocating vmalloc and percpu needs to populate kasan shadow area page table, needs to sync up to percpu page table. Signed-off-by: Yang Shi --- arch/arm64/mm/kasan_init.c | 45 ++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 3fcad956fdf7..960020343760 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -60,13 +60,14 @@ static phys_addr_t __init kasan_alloc_raw_page(int node) } =20 static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int= node, - bool early) + bool early, pgtbl_mod_mask *mask) { if (pmd_none(READ_ONCE(*pmdp))) { phys_addr_t pte_phys =3D early ? __pa_symbol(kasan_early_shadow_pte) : kasan_alloc_zeroed_page(node); __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); + *mask |=3D PGTBL_PMD_MODIFIED; } =20 return early ? pte_offset_kimg(pmdp, addr) @@ -74,49 +75,53 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsi= gned long addr, int node, } =20 static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int= node, - bool early) + bool early, pgtbl_mod_mask *mask) { if (pud_none(READ_ONCE(*pudp))) { phys_addr_t pmd_phys =3D early ? __pa_symbol(kasan_early_shadow_pmd) : kasan_alloc_zeroed_page(node); __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); + *mask |=3D PGTBL_PUD_MODIFIED; } =20 return early ? pmd_offset_kimg(pudp, addr) : pmd_offset(pudp, addr); } =20 static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int= node, - bool early) + bool early, pgtbl_mod_mask *mask) { if (p4d_none(READ_ONCE(*p4dp))) { phys_addr_t pud_phys =3D early ? __pa_symbol(kasan_early_shadow_pud) : kasan_alloc_zeroed_page(node); __p4d_populate(p4dp, pud_phys, P4D_TYPE_TABLE); + *mask |=3D PGTBL_P4D_MODIFIED; } =20 return early ? pud_offset_kimg(p4dp, addr) : pud_offset(p4dp, addr); } =20 static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int= node, - bool early) + bool early, pgtbl_mod_mask *mask) { if (pgd_none(READ_ONCE(*pgdp))) { phys_addr_t p4d_phys =3D early ? __pa_symbol(kasan_early_shadow_p4d) : kasan_alloc_zeroed_page(node); __pgd_populate(pgdp, p4d_phys, PGD_TYPE_TABLE); + *mask |=3D PGTBL_PGD_MODIFIED; } =20 return early ? p4d_offset_kimg(pgdp, addr) : p4d_offset(pgdp, addr); } =20 static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, - unsigned long end, int node, bool early) + unsigned long end, int node, bool early, + pgtbl_mod_mask *mask) { unsigned long next; - pte_t *ptep =3D kasan_pte_offset(pmdp, addr, node, early); + pte_t *ptep =3D kasan_pte_offset(pmdp, addr, node, early, mask); =20 do { phys_addr_t page_phys =3D early ? @@ -130,38 +135,41 @@ static void __init kasan_pte_populate(pmd_t *pmdp, un= signed long addr, } =20 static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, - unsigned long end, int node, bool early) + unsigned long end, int node, bool early, + pgtbl_mod_mask *mask) { unsigned long next; - pmd_t *pmdp =3D kasan_pmd_offset(pudp, addr, node, early); + pmd_t *pmdp =3D kasan_pmd_offset(pudp, addr, node, early, mask); =20 do { next =3D pmd_addr_end(addr, end); - kasan_pte_populate(pmdp, addr, next, node, early); + kasan_pte_populate(pmdp, addr, next, node, early, mask); } while (pmdp++, addr =3D next, addr !=3D end && pmd_none(READ_ONCE(*pmdp= ))); } =20 static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr, - unsigned long end, int node, bool early) + unsigned long end, int node, bool early, + pgtbl_mod_mask *mask) { unsigned long next; - pud_t *pudp =3D kasan_pud_offset(p4dp, addr, node, early); + pud_t *pudp =3D kasan_pud_offset(p4dp, addr, node, early, mask); =20 do { next =3D pud_addr_end(addr, end); - kasan_pmd_populate(pudp, addr, next, node, early); + kasan_pmd_populate(pudp, addr, next, node, early, mask); } while (pudp++, addr =3D next, addr !=3D end && pud_none(READ_ONCE(*pudp= ))); } =20 static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, - unsigned long end, int node, bool early) + unsigned long end, int node, bool early, + pgtbl_mod_mask *mask) { unsigned long next; - p4d_t *p4dp =3D kasan_p4d_offset(pgdp, addr, node, early); + p4d_t *p4dp =3D kasan_p4d_offset(pgdp, addr, node, early, mask); =20 do { next =3D p4d_addr_end(addr, end); - kasan_pud_populate(p4dp, addr, next, node, early); + kasan_pud_populate(p4dp, addr, next, node, early, mask); } while (p4dp++, addr =3D next, addr !=3D end && p4d_none(READ_ONCE(*p4dp= ))); } =20 @@ -170,12 +178,17 @@ static void __init kasan_pgd_populate(unsigned long a= ddr, unsigned long end, { unsigned long next; pgd_t *pgdp; + pgtbl_mod_mask mask =3D 0; + unsigned long start =3D addr; =20 pgdp =3D pgd_offset_k(addr); do { next =3D pgd_addr_end(addr, end); - kasan_p4d_populate(pgdp, addr, next, node, early); + kasan_p4d_populate(pgdp, addr, next, node, early, &mask); } while (pgdp++, addr =3D next, addr !=3D end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); } =20 #if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4 --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022074.outbound.protection.outlook.com [52.101.53.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECFDA4B8DE8 for ; Wed, 15 Jul 2026 18:06:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138785; cv=fail; b=HAdUgNal2ygtZ5ihO3WQe1c93IDru4dTIYo8mJYJcYF+bQyJWxhfOT4UWdbJ9T9lUcxCLE/XO5L2sVZHlrTX6pXFvQL20701Lp20rnIZomoLcwYhv5tshYa9oLltHwxRzfZWZMrj1iRb4wkdvHglA0OFWPnd2KBzVKu2KqVUfck= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138785; c=relaxed/simple; bh=RzxinbwJEUBPaAkvJ860vCAzEJbjLB7F7XLSw1fD0TM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ob6ooJqJ9EfSDYg+7ry+u75wwqsoOMRRCuqnYUwaSmHXWq8wLVofpBb//BBGCc+zLI2Bg70K2tK7DGYQh4+Ymrfb5cxLFFnVneSL9KoLvSJomWgsVxg2RdBEkGQvNQcdFC8Kt7ELR00vMPIzSIIQOCM3mjH7XZvhLpjdKZ1CeLE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=EgrEr9BB; arc=fail smtp.client-ip=52.101.53.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="EgrEr9BB" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tUUoyg3iN1UgAZ+kb/P/e1Sjsex48PPTR6ERKqJn8t5CjHIMiuY5aFUZx5ch73y9fY4I+zsRgv3Byhckdo3VHZnwVZdflWwKv3rvQGVvHuLkJZ6BHFnb/wLNFnAvgBe0h2PYCC69Ntso969DavXJv4P885V21V4KJi/0sMzn7NxSDpLBlgBw0XHXr/HgkVj6ePh7DSWbaMU45LBLK+CvqgkjDxITLs+iAcYNwHO8oQ6LsCiqc+GI+vHCCdXW3CrZusuXFfVx4k34GHLKmD3teK4+0tL2FOx2/xIeGENXPGFKR6gyjpFn7x7kADjhT6j5XPWmrRO5W9vZ8iDqr5x8iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=T82x0+tqCac5WC2NVMzpold+Q4C9KwJqzoGnFI9+n5E=; b=G9Adg47TsPQAmnrI3F+4rEhmFBaHeorjElEtDVWX6vIQddWpD3n44EI2yUMiI6CBBbbjqGMkRZWSXQIg6X+o3kVROBZU/ID9dPkq9ARlmooswpLIsiQ/WTnl2fVrF1J10G+a8KA8sn3DLsklVG7Zfr0DjYinXBPsGbitAsNST6v68E6BEFuhn000d9ydISdKCB0xNYQmiuEFyzpqW/0yARWdOxf/hIt7d2KM+8VvSMiwXAeYARrcUDXA28lfYJ36QkLXV4p2vgCeU8NO9wa1Js+tPWWh5FTCiQ5/znHwJPMYUuJN277S6te8eBTItm2mZu77Ffe2DrbDuwxq4kZivA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T82x0+tqCac5WC2NVMzpold+Q4C9KwJqzoGnFI9+n5E=; b=EgrEr9BB4TUqXKZWfr2HxiLyHlxb4Dqchhq8WVCLOm4vnzqqJtuZkacYk+5kgNUPj/1ToDo8FIovl+ivMRZTv1HepOMMahChKp6iozsI/LaaFGBQHrFv7fMHZI5AVIKuLhzsUOcWapL6cNMQ7+QJYzcEkQr1CtR34aifmYNKpcY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:43 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:43 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/16] arm64: mm: define percpu virtual space area Date: Wed, 15 Jul 2026 11:04:11 -0700 Message-ID: <20260715180455.515692-10-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: f3f093f5-5e61-4ce0-77b3-08dee29bb01b X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|6133799003|921020; X-Microsoft-Antispam-Message-Info: Urdzy9ROeXGPhNuPLFw6vFTq8mdBBAo+2KSOcHQ9VwAOwFyJZ8SpkJNjGvUkjtfkldkqkKiYrFMLDx8s+15OeX3dYHPCk8fom4dyLuvSR+77RNVHBlfPga3UuoCxBv0w4sFeg0MfeBGdFXKXSqYhBLhFnCAvSyhOt1WNNhLbvGo42klamENjdc5dpWifzm0hm2jss90jVhzI11i1l4VAumYV/eMX9QK/NdIz4O3UGBiHjHglr0v/3jUIGq+1aexvkKWF2z6HRsSZ19fVA3sqdowgsKvrOWF5EewyaOuIqJVpbJEnWLF8zH+td3yAG+/jFPBVFK3j7AcSgCM0dZ8DYahVz89vv925CoRd4xD/ZfFET42WzlB7wGgt4Uk2CgQz191HO5qCE+HJb7M4f4mMvXYMFDfYUSAcNK2B/UvcKCuKzIXm5A7BJbcLT+egWCTdHgpeZUqn2n1tqDlK7Ob1Q4IFyBEhQyAwzkYEh2BTLzl/BBBxllEFkK1ehSwPzdCrVrO4AIj/QHJUkrqwTR3+A4Hvk0cjSIm5thFsooHWmO/7o71MWQwsdHVqvJ1iWbqHNVcQOupjyRKj0PXF6ScxGAsY9h1rxKPNJdheQLwoA0ux1QdeStIIQnGbnc5C3SP/pQmMMaPDyeGdD0MoTlm2byeDqmUEyqI4RSPnCAIh3G9l1/HORvoF4ttKhyliwc1DZ+sPMvq8XTELTy7D8/qmtQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(6133799003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/qUSn12XcCqHkJyVSLSpGtEfrAhmFaDFOY00PPrcSmdiaeE2sSEjZ1gAnFKh?= =?us-ascii?Q?RJ2guRDPMXrfoi8/wXxbHlfV1wpPvsaDJCOEonxtVOftrvFuNSXea9VfcAGe?= =?us-ascii?Q?+TcCf6uaJQnp3QXZIYUezf1MQSbtLpQqb7XNvYwOLX+60SAMvvObC0q/afN3?= =?us-ascii?Q?sqJrmmDRZumIzzaujlh0Vtrd61+bArabN3IbfeaH1RXUwd3D5863aaH+pjAw?= =?us-ascii?Q?FheOMjgOUYGYHPT7LO8LuRs+kgMPbHxzeLGOnjkV6c/kzlGA7lx9R/R9WXup?= =?us-ascii?Q?Leq1Saz+bRjGopFUNuGYFztJIP5g6PJqIGcYvfuJLJLsbTjYV9Yd2h6SiBv/?= =?us-ascii?Q?bpwa5TDch5xeZJHiecfaYg/3UdAzlPUvVUQsjoQhRBatHZMW5wHj/x/KrVFP?= =?us-ascii?Q?0tyVNtCQutH2woROOqY8DcEERdfv52gU8blJQ+WCQkUObgnIueXW+w1D34vj?= =?us-ascii?Q?IZL6wxqgnyE7taSDZsNX6ZKaoe8aqy+9qPsjwnaR3NkAYtHK3ILzW6FknqEn?= =?us-ascii?Q?XyToORVWDuNHmcPKPZt3LGrf8s5wSEfbbaK6xvqjg76bJsX0M5LwgAhM6Kbt?= =?us-ascii?Q?OCSwtRq13vDNkLe7NeDM5+z5+rqx9rd0AZaOqcsh9m7YLCR/yXqQz7jwM7v3?= =?us-ascii?Q?U1M3E7yNEgSUvoZ2vh2TEvAs/wHuPDGuRaHVx8O0o6keo1LJARPJNfWCtlzq?= =?us-ascii?Q?PCOOsc8vaB5ARyk/CRIhzJuLIggQJYwckQtoZz++SR9eQoKITnszvmt/SqWw?= =?us-ascii?Q?nj55YBpgBUXsVbL9SrgHjBHbtLHoChp41s6FTei079lbN0e+reNmajr8Phlf?= =?us-ascii?Q?UY7lUUAn9ayqegTmpR8uTLK1KoZfq4fGFgeNuRoqbn/Y8/OImHvPVyIJ3QxH?= =?us-ascii?Q?idu4EfzXImr9DwS/8dLlWJPiKAFBPMA2LjMHEQ+SNN0yn+ZlyWwqFwNWhPIk?= =?us-ascii?Q?kYWtYM4Pj+vr6ob5MC0Ejnddg3pfNwGTHwMiCJj2Fg51keyxNKbRJFXrYq+o?= =?us-ascii?Q?QRkPTVSeN9Jxt7G3Ag0+KPLbY7I/QyPRqiuXrepEFydJCu/tRH9iVyMH3ErM?= =?us-ascii?Q?hAVl3xOZnGrf0aNVCa3jyHFbbAE1esSB/U6rnoh/as5kNbaOTtoxkznV2Mu6?= =?us-ascii?Q?kVw0/SKMrB+Gpt4DNSQEB3RogbySrWVeLwIoI3Hsu76MuWd+bJ+bbo7PnlWl?= =?us-ascii?Q?hS2M4vYX8Dkwi+ExkL6uWd9Em1OSdfSTRo2CtDm2Akcd8mYcpCTb+xdbwbMx?= =?us-ascii?Q?QcWN/5nH/yGJMuBSUNwur5UoTMKfAF1GbSX04o2t77sMsucnLrqTNzvqHWJ0?= =?us-ascii?Q?f3hXCtj6SCecTKtwUuqh3BTLTrBc1msE50vy2sFp/i8Ob1iqWffH3AKOmn3H?= =?us-ascii?Q?Eu0VSHj39dUeD4A7Lj+lz3z0dwC67MIRfRDAzIOtpDPy4iMfYNquaQPFu8Or?= =?us-ascii?Q?D7rrnIyOpo5M6sUPeOH5KQS/lqne20RxT6YZ1AkIEXhFsc49bk32NY2DP8nw?= =?us-ascii?Q?gqAc/IRsblPHTrBVT88Msqt81fwmvAF4jEo7KXa8U7cagIUlAkjqU8d0OImO?= =?us-ascii?Q?71c7zHOLqJijr87OabOieZ9yQT0J2Ta8Va4Y3h7cH+fE9tbkn3RMpVTsZP/Y?= =?us-ascii?Q?wlAPouNzwgDV0Lbeowsjxnt4u816tAge4nZ4tUqA9ivPCsclJhvPPjzfaZHT?= =?us-ascii?Q?HLFtHgMI5WfQZ2y+AsWzJNfSyiEfviij7QAGLaBKW2BJXIZ5tl6y5msslbWN?= =?us-ascii?Q?VfPPXGL3GePp0Me1ciGwb2T/oHK4sEc=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: f3f093f5-5e61-4ce0-77b3-08dee29bb01b X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:43.7917 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zZ4ndr603TnkDWJAsr8nBerlwEIcJtVQcvhuwoh2ABbWup9g74P80h+fYvrEzXZetzWY2si/w9JkxhLI36doqdI4+q0/kyY7jdpqMKbRfC0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" The percpu allocator returns offset from percpu base address. The percpu base address is determined by the first chunk which is typically in the low address of vmalloc space, however percpu varriables are typically allocated from the high address of vmalloc space. So the offset could be quite big. It may be the whole size of vmalloc space. To support local percpu mapping in order to optimize this_cpu_*() ops, the percpu allocator needs to allocate memory from local percpu area too in the follow= ing patch and the offset to local percpu base address must be same because the offset returned by percpu allocator must be used to access both global percpu and local percpu. We can half vmalloc space to have either half dedicated to local percpu, but it wastes too much address space. So carve out dedicated global percpu and local percpu areas. Each area size is 2 * PGDIR_SIZE if page size is 4= K. It is 1TB with 4K page size, should be big enough for percpu. It has diffe= rent size for 16K and 64K page size respectively. The percpu areas are aligned to top level page table entry size in order to just need to sync percpu page table at pgd level to minimize page table sync overhead. The kernel virtual address space layout now looks like: +-----------------+ | Linear mapping | +-----------------+ | Modules | +-----------------+ | Vmalloc | +-----------------+ | Global Percpu | +-----------------+ | Local Percpu | +-----------------+ | Vmemap | +-----------------+ | PCI I/O | +-----------------+ | Fixed map | +-----------------+ Signed-off-by: Yang Shi --- arch/arm64/include/asm/pgtable.h | 27 ++++++++++++++++++++++++--- arch/arm64/mm/kasan_init.c | 2 +- arch/arm64/mm/mmu.c | 4 ++++ arch/arm64/mm/ptdump.c | 4 ++++ 4 files changed, 33 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 13e069b01600..83cd65610d7f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -18,14 +18,35 @@ * VMALLOC range. * * VMALLOC_START: beginning of the kernel vmalloc space - * VMALLOC_END: extends to the available space below vmemmap + * VMALLOC_END: extends to the space below global percpu area */ #define VMALLOC_START (MODULES_END) +#define VMALLOC_END (PERCPU_START - SZ_8M) + +/* + * PERCPU range + * + * PERCPU_START: beginning of global percpu area + * PERCPU_END: end of global percpu area + * LOCAL_PERCPU_START: beginning of local percpu area + * LOCAL_PERCPU_END: end of local percpu area, extend to the available + * space below vmemmap + */ +#if defined(CONFIG_ARM64_4K_PAGES) +#define PERCPU_SIZE (2 * PGDIR_SIZE) +#elif defined(CONFIG_ARM64_16K_PAGES) +#define PERCPU_SIZE (16 * PUD_SIZE) +#elif defined(CONFIG_ARM64_64K_PAGES) +#define PERCPU_SIZE (PGDIR_SIZE) +#endif +#define PERCPU_START (PERCPU_END - PERCPU_SIZE) +#define PERCPU_END (LOCAL_PERCPU_START) +#define LOCAL_PERCPU_START (LOCAL_PERCPU_END - PERCPU_SIZE) #if VA_BITS =3D=3D VA_BITS_MIN -#define VMALLOC_END (VMEMMAP_START - SZ_8M) +#define LOCAL_PERCPU_END (ALIGN_DOWN(VMEMMAP_START, PERCPU_SIZE)) #else #define VMEMMAP_UNUSED_NPAGES ((_PAGE_OFFSET(vabits_actual) - PAGE_OFFSET)= >> PAGE_SHIFT) -#define VMALLOC_END (VMEMMAP_START + VMEMMAP_UNUSED_NPAGES * sizeof(struc= t page) - SZ_8M) +#define LOCAL_PERCPU_END (ALIGN_DOWN((VMEMMAP_START + VMEMMAP_UNUSED_NPAGE= S * sizeof(struct page)), PERCPU_SIZE)) #endif =20 #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_S= HIFT)) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 960020343760..a8e272bfefe1 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -324,7 +324,7 @@ static void __init kasan_init_shadow(void) =20 mod_shadow_start =3D (u64)kasan_mem_to_shadow((void *)MODULES_VADDR); =20 - vmalloc_shadow_end =3D (u64)kasan_mem_to_shadow((void *)VMALLOC_END); + vmalloc_shadow_end =3D (u64)kasan_mem_to_shadow((void *)LOCAL_PERCPU_END); =20 /* * We are going to perform proper setup of shadow memory. diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e1290ccaa5be..f14ba512fd8f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -108,6 +108,10 @@ void arch_sync_kernel_mappings(unsigned long start, un= signed long end) if (!percpu_pgd_setup_done) return; =20 + /* Don't sync local percpu area page table */ + if (start >=3D LOCAL_PERCPU_START && end < LOCAL_PERCPU_END) + return; + addr =3D start; do { pgd =3D READ_ONCE(*pgdp); diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 1c20144700d7..dd293319823e 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -389,6 +389,10 @@ static int __init ptdump_init(void) { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, + { PERCPU_START, "Global percpu start" }, + { PERCPU_END, "Global percpu end" }, + { LOCAL_PERCPU_START, "Local percpu start" }, + { LOCAL_PERCPU_END, "Local percpu end" }, { vmemmap_start, "vmemmap start" }, { VMEMMAP_END, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022103.outbound.protection.outlook.com [52.101.53.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D7C54C77BA for ; Wed, 15 Jul 2026 18:06:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.103 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138789; cv=fail; b=SvIsq8EqlbG/dxwdcshPaM9Cb2zQQoBzOGQYFJdsjVA7tgrwVU6fACB9HK9/Opu9ZoedC+ajFC/NI7epFgNIFTN1SOLmjFwkwvHIw2ZqzCy4HjDtgLYFk5zDUv01iEF8hP8bwQIdq3+xTO9Pv3XM4WP0SrAyPkq4BJ2KZ3Nb9Zg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138789; c=relaxed/simple; bh=J51NlvkvFeQDDmX/6FAT7fzinR1KSDcbNYtBPo0f2RI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=US11f0C1e4woRBU8p26AYQ3OhIJtZjZ7XG0c+w5QrEMKIrjpXkDoQi2A6Hqzir/kcDpvSshQ4yLa5UrWxmjdtzJcLhmgGolMUNSHiJNoM8UIubBkd+RHRjD+KjxVEZAEuG9Vdor3bRBhezbqetF2mGqf7pwvN+93eUsy/lNWY8Y= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=vFq1UpQl; arc=fail smtp.client-ip=52.101.53.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="vFq1UpQl" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=opKWfv/BDktkF+XIkZLR+jqRzuh7kmvcPRq2gAwAkTrubPkK4Raz7TSAtrSiuGCKWDpyJv/jUbGA7gHh8hHKBs47YKOIPJvGuP+oUZ1Q4PjJjv4jsgWYe4Gywyk20hR0GYP3Aum7m2WFiO5niG8QIMECGMjwXMnWvg83a+4/NLerpTS1h6ZGRpmH6LP+ccgz3QHmQXtsrr8ChgsL/Ad42NLXQp0FQeew7lJ2p/3W/nu2tDGA582t7fUK6R0QCBGPcTHZ8BxuteHtJqgx5CCvDXwNWTJHxH54Eo7YSUG1eAYfSTzN6dVeRUQsQUOmcj+IlfWaxPyNxpY8ydp+foB6gA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sc4DDd+cQ6kv3hKNLVMRVQ+JpaHxehsf4M1/oCbtNBc=; b=k4HDgkgvKTOWSoMJ6o+/WA91xlFGqCzwsB4u5UZN6MX9sRlgiXeJ7JyDEWF0aRLqovcmmf7N7OWCn9v4OV6dvQwAsRp41DOYtp8HK6Pz/ILVSEi0xN3H6a23nuixN+uCmy7CF+pmY61spdCfPd6bXVgYXossXy7bbIZoaNU2ByfnZtdVByzOUyfSCykwddFz4Mch3Yq3tFqdbWO6y+wFTYXXaOukfM3s55hH5L3hv5OFO3VsHm5hOhWQ7Eg/h5BjY40nEa/jInr8/ASmEdtdo7Z0CJV4qIPAPDjiVD2Hx21EI3LK5HcW3O8yEeNB0rP0F+nVygedD/O27es7RjWQrA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sc4DDd+cQ6kv3hKNLVMRVQ+JpaHxehsf4M1/oCbtNBc=; b=vFq1UpQlYQ0XzqLTnBdhecGsj1sYSqHEBjRuDaT2sX4euyXMILT2l2bTwku0J6M6qt3i4OtHRDapoVJJFuzoj3R1yMonXDviq3Me9G/iVCBfcIwi0IIKTt+3iVIcQAbIgvpgc9xvRveB7XvvnE68TC2lfi8y/+Wf0QU8Oziu5Ao= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:45 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:45 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/16] mm: percpu: prepare to use dedicated percpu area Date: Wed, 15 Jul 2026 11:04:12 -0700 Message-ID: <20260715180455.515692-11-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: c18bd9bf-4020-4ab9-b56d-08dee29bb0ea X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|3023799007|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: lYTJ5qU1ar3ibmVjeEeClgHS5kqjfnNpGhRxS5MhZ5e19lN6aTin7P4smSSnpb4XelL2MKzFDz++nAHQSPO6VS4Wq2qhb1QxiADaWC62rTJ+tH+agEonvU3U2ArKAuwGPPdGe/g6NbJcjiIU3BU/pqpjBHoiI5WLVDAJksjJlXvA1Vq9eimhUGW3LROyjF9KVBDov/Fz2jEr3yK9zcvUBTQCmCOFzd1/XOeY0mgE6HUl15uoGCBVI3y0Oh/GJfo+5zWUmby2OEXfIXgOFqvDDt08r+rfiBcfdhorI4bp+V6+4f0ymu+j/KOn5srnTQvihA5J144AgZnPq8iU2Y+HhWBhmuplom7y044z80uZeqzn7it1bWdx6TMwsfMq4ApZT+4HXQaHehSXDlK2rmHGgsyzrUR0htCn6j5A4GJ/uqKbf++DQ/wHgL9ZN8Ipf5KukTz3i42s7znsxAwN6AB7YCsad0eDwHbeW0EKCEI0d9NEeANDpVDMFk0sTXu+uQndCLa2g2YuF0JVFQsVLpbm/qFqGeQepONR/jlvO2o4CbFeVkU4RnlVNhCx/fR5PsRvMmiDkIZF0TAxMSq6y1Af4v5bDIEs4VaYF32I5ANZAQL7rDpmCe16VHMP3QPiqiq8Z9OAMVPGLO00wOs+pvdtTBWLgGZPAe8Hd8rjQYn1QL07lxa9JuyFkyg82cQdCDWMkzSiLluZFl2A0tBeSYmI3w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(3023799007)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?XWgfVG1kkHYXJy3qYlSzFF445WgKrl4StXgYktOkEcXUFWnzwoizwZgZKJ+0?= =?us-ascii?Q?KyEYW+kLGH2A1T4tVTykzbZRO3eJBwjYrzoZtOu5pYecTnymnpyrQ+DilefO?= =?us-ascii?Q?jaA8fveW5M/gbYuGMx2h+gByxtGOcPXRzAJkMuIGnj2szB/Oo/tWweSYdSay?= =?us-ascii?Q?oA7VDPTALDw+q8D6a/LqVQFkLV7Snsoj6GfxmQDRWp/TqlcZlJyCoVjSucqx?= =?us-ascii?Q?jYZIsBxh3Z96HpIqqsOlM4hXHpvV79171p52Kv4TYb4ouEtw1m8NJlUkDDph?= =?us-ascii?Q?yLzFBFGzfEGcqSDw6H8PKFIsT1w560gE5trqKwYmFeGNjFD3IohwtVkd6XQF?= =?us-ascii?Q?8w3SroxMy0qVe2RfArFdxJxChVzVpwS1I5Q3uVLjgoI7NzC46eFDu33bI30N?= =?us-ascii?Q?XyFZ/E7cmNi/uO46Yml/9SfjJAVFBWz6zw/9FhBop8uLfkx6/8ujpKkOXmso?= =?us-ascii?Q?Ahz/valVc5TrPOecxIMYNuiz+TTjg1gUVdsUlUVH82vWCZ+HoNqzg/r10Pcl?= =?us-ascii?Q?WknRL/I2nD7VgXnggosoZgyXzASCNguYDi+/qAD3YEZcgX6HwB40XKnF3aFl?= =?us-ascii?Q?Q5TdQh+dfrIxlnkk6cIPXbmpv5DgLv9YojBogYjWNdCOqeRUtt5pPKE0qDxO?= =?us-ascii?Q?fTFvU5vvKl2ReYMnpWgo7LXuSCN7s02pWztBRJ2I75OqYwnSPATyTvUf+syL?= =?us-ascii?Q?ZRjMMC0Agmp+/aTpTOX2x9kgo2sjoH6QMdmv1EwuR//weDGv27CyDHV6Zshv?= =?us-ascii?Q?2Mt+CSkADp3byGGGJhxHW/6q8pDDbVkYCwZnga3jeeWbvf2g0PX8V2xKotm9?= =?us-ascii?Q?WV5Tidfem/8w7YiOQwCImEf6QWAkjR0qUxfrZ5Peh+c8afm6ALcjOpg3qx9H?= =?us-ascii?Q?XlmS4FWi5RpzSlJK3Gj0mlKnDMe2ysIwVrNaCEMtzngXkPQpGMa+Ax/tnf77?= =?us-ascii?Q?McLksGrDpPRthtRD3jhfFxdc0cEF9xHSMRD8fG4qtI/61eJ/+BcC+CaI52tj?= =?us-ascii?Q?pWRKcFK/qSwqxGonAYmo9r8ki+veO8HVZphUs5k85llG8ONcWstmfdvDY+Vg?= =?us-ascii?Q?ctDfWgl3/NrfbCr5fi66WBTTtMRaADYvhlZ3b7gi4hw/X/fcVZ+sxFs1uZpn?= =?us-ascii?Q?xhpR+qmGLDCtz5p6/Pk7NLiTvWAeg9LJjr6wVvL++7hM7lftvqYEDlrVP6KP?= =?us-ascii?Q?IVRp+5JcB6Vud3HCEwmbWw5+6nY8q1f1iVgjqc0ZjhrXiAzpa30LeZwdrER6?= =?us-ascii?Q?g+OKeiPDSW1O3caJOOwIDA+rCYS5b8ZytUDj90fiNEg9ola1DeDmzSadQ9is?= =?us-ascii?Q?SdOvtbbp21ANMr3HdvrwKFeLXzQlyjPir7/pEs3qCGCI6zR1BsyY/PbDu8fr?= =?us-ascii?Q?kjNDoihs21uaDQ0eUhluKmtxNHa1dAmSg5bRufg2TJixjKNkSOQP+vP/KB6W?= =?us-ascii?Q?68saiQCXRP8aOxY56O6F5e5dRfuEtip8+3tf0f9RQmYgJIXyREPEkJlAx9bb?= =?us-ascii?Q?Ce9DDU5WS+iS/JL4OfwxTPcJ9SwLKiYoxOwaHbt86IXSMR2ryIEhnSgaEsft?= =?us-ascii?Q?tJakY9nLZ/0vs94yJ+B1dEQ9blWOGdIrrpk59sok94sp0jC6SfKm938JLBV3?= =?us-ascii?Q?dJRnCPyYW1NTfgye4c/NX63eeJhAUeAu5bZ+cSJE1U1r8NhK6NIKAUgst5tZ?= =?us-ascii?Q?h6zDQeEJ8UBzNWq8Xz46LTtx4Wwon/2sk1Zx0NPHZGiGLKXepQC5Mj/34IqX?= =?us-ascii?Q?3glOWfqpN/fTARYwBKPC93NSMZ6XsqE=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: c18bd9bf-4020-4ab9-b56d-08dee29bb0ea X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:45.0305 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: E6BwwYx5TWqD3wOQM1Alni5OxCsi/gdZFSkguIaaL6r2K6SeZOYSCw4eCuUKnxaOWuPXpCF7LW4przjxbg6f/LfXPQguJqXP2BbpnPe8J+U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" The percpu variables are allocated from vmalloc area by default. The architectures which support local percpu map will allocate percpu variables from dedicated percpu area, for example, ARM64. Introduce a new kernel config, CONFIG_HAVE_LOCAL_PER_CPU_MAP. The architectures which support local percpu map will need to select it. If it is enabled, allocate percpu variables from the dedicated percpu area. But still treat percpu area address as vmalloc address. Signed-off-by: Yang Shi --- include/linux/mm.h | 11 +++++++++++ mm/Kconfig | 3 +++ mm/percpu.c | 8 ++++++++ mm/vmalloc.c | 23 ++++++++++++++++++++--- 4 files changed, 42 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 485df9c2dbdd..c8c3800fbd9f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1858,6 +1858,17 @@ unsigned long vmalloc_to_pfn(const void *addr); #ifdef CONFIG_MMU extern bool is_vmalloc_addr(const void *x); extern int is_vmalloc_or_module_addr(const void *x); +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +static inline bool is_percpu_addr(unsigned long addr) +{ + return addr >=3D PERCPU_START && addr < LOCAL_PERCPU_END; +} +#else +static inline bool is_percpu_addr(unsigned long addr) +{ + return false; +} +#endif #else static inline bool is_vmalloc_addr(const void *x) { diff --git a/mm/Kconfig b/mm/Kconfig index 13ea4b29ee7b..e1910c332638 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1061,6 +1061,9 @@ config NEED_PER_CPU_PAGE_FIRST_CHUNK config USE_PERCPU_NUMA_NODE_ID bool =20 +config HAVE_LOCAL_PER_CPU_MAP + bool + config HAVE_SETUP_PER_CPU_AREA bool =20 diff --git a/mm/percpu.c b/mm/percpu.c index 45c30c6531c5..b27828602c32 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3243,9 +3243,17 @@ int __init pcpu_page_first_chunk(size_t reserved_siz= e, pcpu_fc_cpu_to_node_fn_t } =20 /* allocate vm area, map the pages and copy static data */ +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP + vm.flags =3D VM_ALLOC; + vm.addr =3D (void *)ALIGN(PERCPU_START, PAGE_SIZE); + vm.size =3D num_possible_cpus() * ai->unit_size; + vm_area_add_early(&vm); + kasan_populate_early_vm_area_shadow(vm.addr, vm.size); +#else vm.flags =3D VM_ALLOC; vm.size =3D num_possible_cpus() * ai->unit_size; vm_area_register_early(&vm, PAGE_SIZE); +#endif =20 for (unit =3D 0; unit < num_possible_cpus(); unit++) { unsigned long unit_addr =3D diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 17c26e9796b2..9ec401918736 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -80,6 +80,9 @@ bool is_vmalloc_addr(const void *x) { unsigned long addr =3D (unsigned long)kasan_reset_tag(x); =20 + if (is_percpu_addr(addr)) + return true; + return addr >=3D VMALLOC_START && addr < VMALLOC_END; } EXPORT_SYMBOL(is_vmalloc_addr); @@ -4926,9 +4929,15 @@ pvm_find_va_enclose_addr(unsigned long addr) static unsigned long pvm_determine_end_from_reverse(struct vmap_area **va, unsigned long align) { - unsigned long vmalloc_end =3D VMALLOC_END & ~(align - 1); + unsigned long vmalloc_end; unsigned long addr; =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP + vmalloc_end =3D PERCPU_END & ~(align - 1); +#else + vmalloc_end =3D VMALLOC_END & ~(align - 1); +#endif + if (likely(*va)) { list_for_each_entry_from_reverse((*va), &free_vmap_area_list, list) { @@ -4969,14 +4978,22 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned= long *offsets, const size_t *sizes, int nr_vms, size_t align) { - const unsigned long vmalloc_start =3D ALIGN(VMALLOC_START, align); - const unsigned long vmalloc_end =3D VMALLOC_END & ~(align - 1); + unsigned long vmalloc_start; + unsigned long vmalloc_end; struct vmap_area **vas, *va; struct vm_struct **vms; int area, area2, last_area, term_area; unsigned long base, start, size, end, last_end, orig_start, orig_end; bool purged =3D false; =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP + vmalloc_start =3D ALIGN(PERCPU_START, align); + vmalloc_end =3D PERCPU_END & ~(align - 1); +#else + vmalloc_start =3D ALIGN(VMALLOC_START, align); + vmalloc_end =3D VMALLOC_END & ~(align - 1); +#endif + /* verify parameters and allocate data structures */ BUG_ON(offset_in_page(align) || !is_power_of_2(align)); for (last_area =3D 0, area =3D 0; area < nr_vms; area++) { --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11020121.outbound.protection.outlook.com [52.101.61.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EAC284A3402 for ; Wed, 15 Jul 2026 18:06:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.121 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138793; cv=fail; b=fnRMmt9dIyTbvlzdxikHKv29OD1hXc0zTz1YhLMesePnsjaXHlSHDfP/EfeR5YjdWZcsplvQzmzkY50Lwg36bbfMbXgNjIRao4YsJjwBF8Pko8HTgT+pcxcUL1eZlFWLAGo+Msjp30Zrr1O8jnJcAfM7TK9V85smEkZXjZ/Yp9g= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138793; c=relaxed/simple; bh=VDf/oq0KB6Xu3xCrvh/brKIzDwi7o75AVbN7l3Ms4l8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=YoBn/sNo6qbQhVOJXdekr9EMAqutRiI1XMPVcBk3AfnaEylYubDWP3LnQ7Vi1HNCvKxM+4g5qDKQzOUQiZoArilaaNa2+WkZ2Lzomr0NsKgUbmbaNRRCFarkcMZcsFl9TE3XPftdj1q+F30wWz6e8XnLeI+y90wQ9QnTLdFJldQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=CLqhFtx7; arc=fail smtp.client-ip=52.101.61.121 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="CLqhFtx7" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zKgCLVjlW8P2yKcp7hbPDAHKkrzmWUSgyzH9g/oFmjlsNTCjq7O+9RFBTxL7CmHNmNZg29Vkd1sNVXoH5jdleBRE80CEpv3GQhcuEZgQw/MDEtXA27SIVF+HvichkBtMxab+QKzq6DwDvkT2F4s0NrxqPLakCII8J3KW22FyPuWIrZx/SBRk2le5vJwNl3G9SZH4IN1JE4/jCQrvSTut7Gv2X3/ggMaEcvPrpHPoZyNE1VqmdYGRLwYS/rf+ZoAomYD17glAhcnUDRQnacwhJtAwJhI9VHP28JmHh7xrOtYJgXXPowTdolxYZEZ9hYKa7ieTzm+/0Qqj6CY4P/hbdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iqPy7+ky7gV23SiopGjOcwW6j11A2L5Q4P86X1V4LrI=; b=CG+n0K++vVZ5XahGwPzWipPSzVX47L7Ef7Gn+sJkJhBMsNLxqQEf4Q9Wvq2ITECtQpUOaJuj66somk81wPfIG0DsXfD212gYy15DEpK8MDjoUqUaEFelBI2cuh9nf8OHtRp9xrh8Xozzs3LbD6vyyihoJ0FHIjOopOQoKPC+yo93f6ptku9zKtJaf+sZB1kgfLfTuOQVVFWGd71ude8PxFZIS1JzOVmoc1qaU6oy5HediYvVmHV06fTagvgjaMUZFih4covIR5HItkELFR8/wAfs67xk2/OlO8xLZvB2SeeOBrQk4j/yTKYLkdzMl5kEKCInu20C5y2p8yAl00VRXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iqPy7+ky7gV23SiopGjOcwW6j11A2L5Q4P86X1V4LrI=; b=CLqhFtx75v0vMX1IIBg/c30Dj/q4aYEvbQn97iO77SYhiKmnb/F4D7uCkc023jtwjeDFOZ3OdGO2eazPhuh5qzKlM5ShwVB1nguEFdlCc9xp5w+EK+J/9SDA4CDMaqLmkhlqi+702cZr/z5AxooM+XKMmPtYtbKm2D1eGHqUwbM= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:46 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:46 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 11/16] arm64: mm: map local percpu first chunk Date: Wed, 15 Jul 2026 11:04:13 -0700 Message-ID: <20260715180455.515692-12-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: 7cbf1262-dbc3-465c-e4b1-08dee29bb1a8 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|6133799003|921020; X-Microsoft-Antispam-Message-Info: 7K1iSLciUox6EMUZnEbPa7x9VSSUSC0MtsKvsjkFlOjdOz38fASuMcDPBBA85prS/fw6p82V30zr6KDNi1ej8o/CF9nKYCurJS9lHA1fsXwjQ9CsYs74WK9Ej3NhDWy+fNgOE24DCcB0TuaA0wEFaRNhaeg4jbMywU2dTow8aIAbrZuEiq61CZn2HqymV8IhXj3qJ24utVEroRi4Vyou3V7efKe/gY+o6Cuorhlxc+QdPCXYc0VnIBOwYfXFGhI7lfk22alPAUlwI9C9rIxIYiYEffVPyViiSAf4FwxwhuILs6fIohbhrdmB6Jt513JFj5U/K5vDjk9RjLup+groQIApm3AyQpy9PeJpI/OQ3p5BVH4FaWnLXBGvmo9JTK/5/HyDqaJkK41Nr2MmtxRo60Zh+zc0B+HwlQYTcLyoo9EvcxTu4c3iO0P0sBynoSxU2dxs9uIbcRHg2hmSJfG5h2iKgHWzrt6hhdern9LK3XI7i3XbVuMGgXfYJ6ou5b1XTyWoMU56iak3R8pG7jN8B0vVcrJ7ZEXX8PGkCz6/NJaV8okl7lXYutpbXAONYZQS3MDe/Fkk+/vY4AMmZ32SwMbcseQBowXkl6klsDhBxcl3XELt3gEI6daSyyi5ETV5axzPs4nB4r7kyXDXUQFSMFZEsiTd2zqY1UyvMiJw4vkOhhT6bdnjxZhK4ENMYqg+QLbsw5rtfDXK61DplLWepA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(6133799003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?MamGYGrNabTudSSFf9xtlMv6vcX1/cs6BifFNRy+qth2qWS9j7+1sQ+USQ8/?= =?us-ascii?Q?1HKkpn3qx7bbk4xRkB6Cbv+bRW6vb+I5ooE5HqQslyz1ZB76/LNXZMLxn5GO?= =?us-ascii?Q?ov52ky4nLUl+bu064X5dn1Y8s8/XBiXjj7+QAbhCgFOVjgXpKJUF2eUp2HX5?= =?us-ascii?Q?Drvm4iGMOBSD4b43nxVZF76noCixi0rsuSXXHiiAt2gNi1xVggTGMR8/0BwE?= =?us-ascii?Q?zGI7AbtgDx4lcK70Sm1T2ITWUnXHRzoW7dfSMleRuc+Lp+jhQdyLScvX0f+m?= =?us-ascii?Q?4AIeFNJbhjg8SDHMRHEE+lkoYJVp7FLY22Ne54iYhX+Omq5tcpydWE370Pw0?= =?us-ascii?Q?xjwHDuGwWpfrIFy3puoAZW+0ZN49fow5z9z1Y4NCR1W1wR+w9XXA7XNxa1QP?= =?us-ascii?Q?UOg4iUreuzLT5XgSORntlNU0dkP/XBlzOxsN0XjxPB/EzOmg2UI28EvKcEx+?= =?us-ascii?Q?yQfKb0x1CuSqjQ6uiYYPMYY5DY1hLzGfp6GQrmzHqRzkWO/z8Y+ZPlimSMPG?= =?us-ascii?Q?pmM7uJaZMIi9LwzMafOp8nqVV56b28mMPYD07M+eBCdQf/2UQCSIgHC16cHT?= =?us-ascii?Q?aYtEHa1YGjAIysV+8H8Qs9QdoacTuDzzKlnaP6bbVDblXojtvZF3dNA7MHJs?= =?us-ascii?Q?rnlOv/hhfuVVy4EY/bqMoOjzYC1VpigbG9hClSZTBood0YGGARJrTM4kUkCY?= =?us-ascii?Q?T4Gk1XKoqvI4L3bGltEw1ytZMmRp5uyT4nYA1qAM27NMieJhvtu2YWhVef1O?= =?us-ascii?Q?aUbvB6V31jSdmoTeKszNvAmCXHwVnWGy+p+HlcdZeTwEm/Jn6t1NHrmR4R8c?= =?us-ascii?Q?tHhP2tXavW3vlyAKrVfA3onpnP26lVEBZiKAu+R1SxLbvEqVyIRAVcirM/9d?= =?us-ascii?Q?ja4QPMkERUr1lysrMpxO0xRLxPpnDdrXO2GsrlThgNdsm0Koheop5yLBFleZ?= =?us-ascii?Q?1jsfIDbKEIWgpjj7ZHRIhPnjvHCTS/bjf/oO9rCx25K+cdJQXYUm3y2+eEPK?= =?us-ascii?Q?rLVyLl4TWUq4Yo7qgjvu/fRfI1ju6vf3OARVNxKaQOcUdB6ge8M7fT1n3bJV?= =?us-ascii?Q?urftIrvq1Uho9mqBcEEm1kJlG01qgZkYZhS09eZq6BIRrbtKjrqRPh5VDCzl?= =?us-ascii?Q?bCGuoRr/xDboXuMNEpsL2g/arMEUnNQezmsaKNqLv/EY9Zc8p4jeyUtpF82e?= =?us-ascii?Q?mdKV3TxPumg21PEBKCoINKvLS6qD/7o8tvSVWdjLVNiwCz6uLNpIVAjLWJiv?= =?us-ascii?Q?jIjLojNcP0brc+Kgv5tWEL7kZ9a54OfhiOkHctAQLMHEFIM4jLDg5ZfhiA9L?= =?us-ascii?Q?fSY0Ls/Yf4tf3qzW45KAKxA50aEmzxz+UPlxMcx0jcXS5yeUfo8rw2GLPcnt?= =?us-ascii?Q?XJfmRABo1y7YCsa5hEZpIqAT7WHEs9DxQkNlk7UKBQWYLIYWsQmSst8EK51I?= =?us-ascii?Q?6jD9mOSMAnQRXA2yH1n7+kF/jn7IF/CC+czFqAOVaI7mS/LOAp9D0qAVfa6l?= =?us-ascii?Q?g6UAl81fCHdcqaAeLOVg6fQbXm1CJBbul/GiQmpZ7Hchs10Py1lA7/by/O75?= =?us-ascii?Q?fJszHsRo+TpYq85NHwnPRSvJ1N53yYVk2ZEiJaQKmb3b7MfzBwjGYd+HFjKQ?= =?us-ascii?Q?0VtSEasZFz2JmMwbj7iUFWe8nSg8JURvxACjdAdRF0FTosdwUzKfnNhOGtFS?= =?us-ascii?Q?4m64eGf9th5T3ppciMNaP9WqtG+3gPGvJzqb9LcY2GybIpleRLxwNo05GRyV?= =?us-ascii?Q?FFdGjzmhX7qjFdIREdlIYPussCWcuyc=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7cbf1262-dbc3-465c-e4b1-08dee29bb1a8 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:46.2964 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jMz8Ssj0ZoCn56/42crDsbanlCFBdkkwZkBUb2jvZVjR5wy5P55xaO5V4U/BpzHX4dr0nzYjOG4TKnL8yi5mDF7nBfGQnmrQ8AjecmmYsJM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Allocate local percpu area and map to percpu page table for the first chunk. It doesn't work for PCPU_FC_EMBED because the percpu base adddress may be in linear mapping space in this case, it will result in returning huge offset for percpu allocator. So percpu local map just can work with PCPU_FC_PAGE which allocates percpu variables from vmalloc area or the dedicated percpu area. So unselect NEED_PER_CPU_EMBED_FIRST_CHUNK if the architectures support percpu local map. Signed-off-by: Yang Shi --- arch/arm64/Kconfig | 1 - arch/arm64/include/asm/mmu.h | 4 ++++ arch/arm64/kernel/smp.c | 25 ++----------------------- arch/arm64/mm/mmu.c | 23 +++++++++++++++++++++++ mm/percpu-internal.h | 11 +++++++++++ mm/percpu.c | 16 ++++++++++++++++ 6 files changed, 56 insertions(+), 24 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 23cad9fc7046..df55e6bdf81f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -242,7 +242,6 @@ config ARM64 select LOCK_MM_AND_FIND_VMA select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE - select NEED_PER_CPU_EMBED_FIRST_CHUNK select NEED_PER_CPU_PAGE_FIRST_CHUNK select NEED_SG_DMA_LENGTH select OF diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 8ed3b5f3cf84..aa0818f7ec58 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -73,6 +73,10 @@ extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *= size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); extern int split_kernel_leaf_mapping(unsigned long start, unsigned long en= d); extern void linear_map_maybe_split_to_ptes(void); +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +extern void map_local_percpu_first_chunk(unsigned int cpu, unsigned long v= irt, + struct page **pages, unsigned int nr); +#endif =20 /* * This check is triggered during the early boot before the cpufeature diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 49524a7176b2..4c112bab8356 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -840,36 +840,15 @@ int early_cpu_to_node(int cpu) unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; EXPORT_SYMBOL(__per_cpu_offset); =20 -static int __init pcpu_cpu_distance(unsigned int from, unsigned int to) -{ - return node_distance(early_cpu_to_node(from), early_cpu_to_node(to)); -} - void __init setup_per_cpu_areas(void) { unsigned long delta; unsigned int cpu; int rc =3D -EINVAL; =20 - if (pcpu_chosen_fc !=3D PCPU_FC_PAGE) { - /* - * Always reserve area for module percpu variables. That's - * what the legacy allocator did. - */ - rc =3D pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE, - PERCPU_DYNAMIC_RESERVE, PAGE_SIZE, - pcpu_cpu_distance, - early_cpu_to_node); -#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK - if (rc < 0) - pr_warn("PERCPU: %s allocator failed (%d), falling back to page size\n", - pcpu_fc_names[pcpu_chosen_fc], rc); -#endif - } - #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK - if (rc < 0) - rc =3D pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); + /* PCPU page table just can support PCPU_FC_PAGE */ + rc =3D pcpu_page_first_chunk(PERCPU_MODULE_RESERVE, early_cpu_to_node); #endif if (rc < 0) panic("Failed to initialize percpu areas (err=3D%d).", rc); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f14ba512fd8f..ee4ab031ebf8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1080,6 +1080,29 @@ void __init linear_map_maybe_split_to_ptes(void) } } =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +void __init map_local_percpu_first_chunk(unsigned int cpu, unsigned long v= irt, + struct page **pages, unsigned int nr) +{ + int i, ret; + pgtbl_mod_mask mask =3D 0; + + arch_enter_lazy_mmu_mode(); + + for (i =3D 0; i < nr; i++) { + phys_addr_t phys =3D page_to_phys(pages[i]); + ret =3D __create_pgd_mapping(percpu_pgd[cpu], phys, virt, PAGE_SIZE, PAG= E_KERNEL, + early_pgtable_alloc, NO_EXEC_MAPPINGS, &mask); + if (ret) + panic("Failed to create local percpu first chunk\n"); + + virt +=3D PAGE_SIZE; + } + + arch_leave_lazy_mmu_mode(); +} +#endif + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 8cbe039bf847..6d0542fb9e49 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -162,6 +162,17 @@ static inline size_t pcpu_obj_full_size(size_t size) return size * num_possible_cpus() + extra_size; } =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +extern void __init map_local_percpu_first_chunk(unsigned int cpu, unsigned= long virt, + struct page **pages, unsigned int nr); +#else +static inline void __init map_local_percpu_first_chunk(unsigned int cpu, u= nsigned long virt, + struct page **pages, unsigned int nr) +{ + return; +} +#endif + #ifdef CONFIG_PERCPU_STATS =20 #include diff --git a/mm/percpu.c b/mm/percpu.c index b27828602c32..df2daa57309b 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3194,6 +3194,7 @@ void __init __weak pcpu_populate_pte(unsigned long ad= dr) int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node= _fn_t cpu_to_nd_fn) { static struct vm_struct vm; + static struct vm_struct pcpu_vm; struct pcpu_alloc_info *ai; char psize_str[16]; int unit_pages; @@ -3249,6 +3250,12 @@ int __init pcpu_page_first_chunk(size_t reserved_siz= e, pcpu_fc_cpu_to_node_fn_t vm.size =3D num_possible_cpus() * ai->unit_size; vm_area_add_early(&vm); kasan_populate_early_vm_area_shadow(vm.addr, vm.size); + + pcpu_vm.flags =3D VM_ALLOC; + pcpu_vm.addr =3D (void *)ALIGN(LOCAL_PERCPU_START, PAGE_SIZE); + pcpu_vm.size =3D ai->unit_size; + vm_area_add_early(&pcpu_vm); + kasan_populate_early_vm_area_shadow(pcpu_vm.addr, pcpu_vm.size); #else vm.flags =3D VM_ALLOC; vm.size =3D num_possible_cpus() * ai->unit_size; @@ -3256,6 +3263,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size= , pcpu_fc_cpu_to_node_fn_t #endif =20 for (unit =3D 0; unit < num_possible_cpus(); unit++) { + unsigned int cpu =3D ai->groups[0].cpu_map[unit]; unsigned long unit_addr =3D (unsigned long)vm.addr + unit * ai->unit_size; =20 @@ -3272,6 +3280,14 @@ int __init pcpu_page_first_chunk(size_t reserved_siz= e, pcpu_fc_cpu_to_node_fn_t =20 /* copy static data */ memcpy((void *)unit_addr, __per_cpu_start, ai->static_size); + + /* + * Map percpu data to PERCPU map. + * + * PCPU_FC_EMBED can't support it. + */ + map_local_percpu_first_chunk(cpu, (unsigned long)pcpu_vm.addr, + &pages[unit * unit_pages], unit_pages); } =20 /* we're ready, commit */ --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022074.outbound.protection.outlook.com [52.101.53.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E531400985 for ; Wed, 15 Jul 2026 18:06:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138793; cv=fail; b=LY4xgyCyNsQcRnp0h6xO+IUo1BKzVPLU2otVZCpMRq4ApRSc2lM6kWTBljhVoPh/InY2GC0Kb9CeBIdQ/iFFtfAIcoWLWg3WVTumZuDjocJ4FDaks0W8CNEZfIxRYTgd9NVULrW+UCDCceWvTRx9oJJU4cirt+kWF2oPO4Yv3dM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138793; c=relaxed/simple; bh=WWv0ohiXz0NklG3tad9kSf0bOjy960i6d2ElZN/U1iA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=UFSJonQTMf2imGf3/bnAKxQ39wGUK6txKrdiKn40IVX/MJRr5m54OyYEI0DyppPbpiK7p6LEz9yxNH0xFzPzDVc9YzSWurSpSKA2S1dNz5/xp4LiPPX2iPsgqANzy1h8/B2s2my/Qkop5xrYQX8pdRqXajGc2TNhWMnlPftkTgY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=YSKkkzSf; arc=fail smtp.client-ip=52.101.53.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="YSKkkzSf" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=imqjSA2O/7J8zCHW9HlNEKE1iKO56hq4DKZsJLJFHfeWtbI0tkz2kWhxQ6p6qce4ws5RKINVsqomPypT+GGrD5JDcBLGhppHuUxlLxVEb+5bixLQK35NN4gfMluPtRujg1xp4iEy631bUamGJzyQFosqAso4u+JmVXajXbpjtrgPnEtZD/pO5g749oWDQWEMQZEBYqPRT2YZCQxocbWBMq519I1G+qxd4uFz695k5+ylmg1bUKOrBNvmIJYQkd5tRk4RNVh0+ESynnfj4i4LD13QFgINi4A42W10wW4Mx/G6pkEVG0DbR97ka110N+QCgDdY80vThGVGJI00JOiZyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NSx8H0FV10BFt7xj9//KnorYKlq18iN0Y6e4bLh9hRI=; b=iIZpMNw/urwrvqF54LKqAjtEPmT38aOZF5zW4+cyftn1HdGgyQK7qh6bNP1SeCluYBl/p/hTHJ5mB9UPtcVAayU4FH5n084NhxIDO6bSig4V4zd/uaxS2wWcHXn7kgW6OFnERXlcHZYuGaNaty+iiPRGYvDskWWi8045iZNctfMH9Vr6Vy6mFmAiqnp1qwyn0Pm2+WSoRhVnQBLeGYa7NpHGqLHIGF7+LpCcOyv7H35n9htPI9a+1noXYQlc+qyjqadom86gaUH9mDSkVJ5TJD9JE0wOpIpLGGvYcanaRktM4NFOXarAmpQnZcMIE30d/LUH7NA8jRBpChR8ylzyEg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NSx8H0FV10BFt7xj9//KnorYKlq18iN0Y6e4bLh9hRI=; b=YSKkkzSf3hVvz/kZlV7mDj83y6TX00EIzAQwTncxH6XIPMTCs/w3AZiMbvhLVCiexOrIhqB0A+gFJxlqUyyr1wnls4Zpb+bjzoKL7pForu98ApJ5+WyJUvP30E/Hp3ZiW1kn2wRlRZfdNkeMJbD2YS921GRpirG8UiJr9vVO/KI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:47 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:47 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 12/16] mm: percpu: set up first chunk and reserve chunk Date: Wed, 15 Jul 2026 11:04:14 -0700 Message-ID: <20260715180455.515692-13-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: 73c4d2b4-ed2b-4738-7700-08dee29bb270 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|6133799003|921020; X-Microsoft-Antispam-Message-Info: k2soA6jyWHcdkU4PAHPi1NePaYptG0SfUM3+iOUeRQxTukQo5I3FlrYr27o2pJfChwid8fHkcpek4VTjZDvMkdYRG9mLFeFQugDOYwGGaE4WX54TneXdvACCUx4IBfQhjxhJYynuPUabsQ/uwQg47TevjuGKNcT0TXxtl6tv4qXJxXKSE8bEEq3lRzQiKWRAQcX6m3aKVNwODKk2KUg+I1TeexvmWh//Gr1jLy9z753SHkoWYtxwXJ3bcrWYdo60VLAg2c4jkVIsmGnC1GwcOeTAS9IMUbOw2bSbRQWOIUQIMHSuZccUyjFn9Q2qeE5IEl9RFdlA/LFqkRQrxNdCnZuyHK2Pq7ddUoas7dw+PHIfcL5x4/V9Ppqm8AepPCJQq1S/lfMqBTNDRWhcT/ZzxU94gQnnwkd+TyrxS1NM4pq7/P0P1hzHJkz1izR0PTiKuFnCWDe6Uqa72NMd50vbbu97Wr+bunaJtrqLeoy8mTV1IAeq9eR/lQuUESrDtQhcqpIUucGe/xuPrTeklbOTpII1s/qn/VYTHBymHzZWi4l2KKvWfgRoIl98GxgMuYojeVL6zk2C3z9cASzELNCd7tvAe0xtUP2RqjQb+FgwOFd0tOwiifXM8Pwk/ZBLSg84yGtXTAqS8QIBScKUWUQ0O7WfWKdadVwXA+99PF6ZeP1ORlu295WZjCX1BheM4wUgHHTyR4tTNkdcruQsnIpoCg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(6133799003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kSARMe0KKhBVY++DJJOaSzBXkIL0NRshRLRVSVLKuYlTX7qoV1Lm7FuIjErn?= =?us-ascii?Q?i/8PUM664yZ6+rBOhIHW/mi3hTT3j6P99eExZTJiRsU7+C4BPmRJOfqYrgH1?= =?us-ascii?Q?MAW2QSqqn9dIgT3QrViH0vzMphdjmMr/fNXRgBLCagwvPvHYTDhtThzORwKs?= =?us-ascii?Q?Dikh39TY0MAwZFWV4bRawWGgZ90tle4hXD/nwRe4RGkEffTf3plzFi5l0Upe?= =?us-ascii?Q?A0u0I6KqywwCI5z2be3aJh5LTW4uIrscjvu7MqYU9qppfg9fxTgjPWXYgb6U?= =?us-ascii?Q?ryjS4PIykdXIM2+T+HSIoFP/uzYUgBqFWj6M05oVgK4pGRoiC2NpcjZN2+IR?= =?us-ascii?Q?GMC23DIPzMCuRDVNmONsQvlJ/Rce6MYz7M/FdOAomIagcAOa8nzw41GouaQ6?= =?us-ascii?Q?R95zVbsJ/M3GGCd5953qZvO24PsC6Y3hNDx4zReMW6fqJigHyKumApKj9UhA?= =?us-ascii?Q?VRaonC1hs01WwMk9JUI9090CHprvB0fDI+RKU6GEp33ZOPaQDq8/YluYcYmm?= =?us-ascii?Q?U3y8KMbXCACvtHfy0YDBomlmDzt60S3QiYALHLZ1GjduU2S2wrOYxVhaLS5/?= =?us-ascii?Q?+QnE56uC50yG4Kkxga0ZWprcKjqf4ql+ofotujIfkzHBDZw+V75GSh4O1Oen?= =?us-ascii?Q?h2K3fea8FFWP6W60sjWOQapSkjN5m1CZMnx7aOasMOW6pZQ3DKdpDVbx400m?= =?us-ascii?Q?Vm3uJOg6FVx+ykov4cGEGCFRYQkkDFsPqg++JEP4ef6dadt1Psg3shOXW2TY?= =?us-ascii?Q?ynfEkl1iQGIxIIgD6MRTjHwl1n6b6aDAfmsBEDMyGzjDrwt3baKcSi8oTMTl?= =?us-ascii?Q?detWVzxtI+onKJI1FaY6Jnu8wwmWaeYtKwqYMY1DLfh5IaLcYuZNDkv03wnF?= =?us-ascii?Q?FlWuxBQoIcjvfFHQvh+p6zR4w9n95ePq0melQFrL6//0UvcpQrsgTgViaZoi?= =?us-ascii?Q?je4lEFhpv7eBdPEaZlMi/sViijKRphb4Ib5vEv8XmRiAa8sS07bHWRmgwbpV?= =?us-ascii?Q?cZqVDA1VB+tGylyLcOo6hMABkWXqlf0lWzjf/mriq5Z2j2OtOJvyT5IbUQNB?= =?us-ascii?Q?+yUZm24wPh1cogh24rbk+EQjabMeoJNi5eISivJ/RJk3QayRXMM1rl/+Nwp0?= =?us-ascii?Q?cR/Y7gLGTYWjpCSLK0mWtQKr/eVNKBwGgZsL9fUgcoPLEy2yz+dk9s2Ia5Nm?= =?us-ascii?Q?Xbff4LEtivnZZ+JOgJkSQS4rsPT+1yVZJWYFUUVs+hwKv5S6kDjvHcHuuH5A?= =?us-ascii?Q?JQOULVi081TXvQMixDLdQ3+Jx68PxC6MTZz1bDnyWu+YgHZFR4bJVrEx5le6?= =?us-ascii?Q?rfwIPusVIq5aPBj/EEvn+kp/oY4CHf+7IoC1vdp+Fx/5qpJvUB3RnmsrPpzA?= =?us-ascii?Q?UNXvMEmpWf7n45+qC+y28v6ZZ2ReWdqfzOs8iVJQURPWjr0fiux0dEMbiJxI?= =?us-ascii?Q?lTE0BmD43iwnvQ89tglrQaQjv4KdYAqUXGcVYWrLFjQtgVfuINvC6jO4bdpF?= =?us-ascii?Q?G2eNYOeANmsgN4sLJdQseJ83Dxx/rpBuK6htksPqbgKBvASuhLProyWkvOtf?= =?us-ascii?Q?atxIuZLogIFmJWIvrOEK85T7zpIs930PFEfWXeK9Y8fx5XFnau4dD7nYLiwD?= =?us-ascii?Q?y1MUFCErha2mpl+bv1O50NbYn7Rv1fwuCjqhpVnHY5nfE/yNV7KTM6YdtT42?= =?us-ascii?Q?Ni3GOsxq6YmHXZN+N888w03J1JEH8WvN43ONR4YsfgGGS2vCEWxB0Hhbq+8R?= =?us-ascii?Q?YVbuz/XasdFEX1rbvRdCaLOpYRi1OIY=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 73c4d2b4-ed2b-4738-7700-08dee29bb270 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:47.5930 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: J4utBpbZ1+OW5y3XxjqCoQL1NTX7pSYRn7jbSxiNy7UpROBMhFH4yaeOvFWue2JvisjzgbMNdi57qg56vupwd7dtlvf7FjXBqA1c5QNFG0o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Set up the first chunk and reserve chunk with local percpu map. Signed-off-by: Yang Shi --- include/linux/percpu.h | 2 +- mm/percpu-internal.h | 2 ++ mm/percpu.c | 27 ++++++++++++++++++--------- 3 files changed, 21 insertions(+), 10 deletions(-) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 4f36b8585995..2f0e190aca29 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -114,7 +114,7 @@ extern struct pcpu_alloc_info * __init pcpu_alloc_alloc= _info(int nr_groups, extern void __init pcpu_free_alloc_info(struct pcpu_alloc_info *ai); =20 extern void __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai, - void *base_addr); + void *base_addr, void *local_base); =20 extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_= size, size_t atom_size, diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 6d0542fb9e49..ec2e55a2f836 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -63,6 +63,8 @@ struct pcpu_chunk { * chunk_md. */ void *base_addr ____cacheline_aligned_in_smp; + /* percpu local base address of the chunk */ + void *local_base; =20 unsigned long *alloc_map; /* allocation map */ struct pcpu_block_md *md_blocks; /* metadata blocks */ diff --git a/mm/percpu.c b/mm/percpu.c index df2daa57309b..15303611ecaf 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1338,6 +1338,7 @@ static void pcpu_init_md_blocks(struct pcpu_chunk *ch= unk) /** * pcpu_alloc_first_chunk - creates chunks that serve the first chunk * @tmp_addr: the start of the region served + * @local_tmp: the start of the local percpu region served * @map_size: size of the region served * * This is responsible for creating the chunks that serve the first chunk.= The @@ -1349,15 +1350,16 @@ static void pcpu_init_md_blocks(struct pcpu_chunk *= chunk) * Chunk serving the region at @tmp_addr of @map_size. */ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp= _addr, - int map_size) + unsigned long local_tmp, int map_size) { struct pcpu_chunk *chunk; - unsigned long aligned_addr; + unsigned long aligned_addr, aligned_local; int start_offset, offset_bits, region_size, region_bits; size_t alloc_size; =20 /* region calculations */ aligned_addr =3D tmp_addr & PAGE_MASK; + aligned_local =3D local_tmp & PAGE_MASK; =20 start_offset =3D tmp_addr - aligned_addr; region_size =3D ALIGN(start_offset + map_size, PAGE_SIZE); @@ -1370,6 +1372,7 @@ static struct pcpu_chunk * __init pcpu_alloc_first_ch= unk(unsigned long tmp_addr, INIT_LIST_HEAD(&chunk->list); =20 chunk->base_addr =3D (void *)aligned_addr; + chunk->local_base =3D (void *)aligned_local; chunk->start_offset =3D start_offset; chunk->end_offset =3D region_size - chunk->start_offset - map_size; =20 @@ -2562,7 +2565,7 @@ static void pcpu_dump_alloc_info(const char *lvl, * and available for dynamic allocation like any other chunk. */ void __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai, - void *base_addr) + void *base_addr, void *local_base) { size_t size_sum =3D ai->static_size + ai->reserved_size + ai->dyn_size; size_t static_size, dyn_size; @@ -2572,7 +2575,7 @@ void __init pcpu_setup_first_chunk(const struct pcpu_= alloc_info *ai, unsigned int cpu; int *unit_map; int group, unit, i; - unsigned long tmp_addr; + unsigned long tmp_addr, local_tmp =3D 0; size_t alloc_size; =20 #define PCPU_SETUP_BUG_ON(cond) do { \ @@ -2713,11 +2716,15 @@ void __init pcpu_setup_first_chunk(const struct pcp= u_alloc_info *ai, * chunk. */ tmp_addr =3D (unsigned long)base_addr + static_size; + if (local_base) + local_tmp =3D (unsigned long)local_base + static_size; if (ai->reserved_size) - pcpu_reserved_chunk =3D pcpu_alloc_first_chunk(tmp_addr, + pcpu_reserved_chunk =3D pcpu_alloc_first_chunk(tmp_addr, local_tmp, ai->reserved_size); tmp_addr =3D (unsigned long)base_addr + static_size + ai->reserved_size; - pcpu_first_chunk =3D pcpu_alloc_first_chunk(tmp_addr, dyn_size); + if (local_base) + local_tmp =3D (unsigned long)local_base + static_size + ai->reserved_siz= e; + pcpu_first_chunk =3D pcpu_alloc_first_chunk(tmp_addr, local_tmp, dyn_size= ); =20 pcpu_nr_empty_pop_pages =3D pcpu_first_chunk->nr_empty_pop_pages; pcpu_chunk_relocate(pcpu_first_chunk, -1); @@ -3108,7 +3115,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_siz= e, size_t dyn_size, PFN_DOWN(size_sum), ai->static_size, ai->reserved_size, ai->dyn_size, ai->unit_size); =20 - pcpu_setup_first_chunk(ai, base); + pcpu_setup_first_chunk(ai, base, NULL); goto out_free; =20 out_free_areas: @@ -3260,6 +3267,8 @@ int __init pcpu_page_first_chunk(size_t reserved_size= , pcpu_fc_cpu_to_node_fn_t vm.flags =3D VM_ALLOC; vm.size =3D num_possible_cpus() * ai->unit_size; vm_area_register_early(&vm, PAGE_SIZE); + + pcpu_vm.addr =3D NULL; #endif =20 for (unit =3D 0; unit < num_possible_cpus(); unit++) { @@ -3295,7 +3304,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size= , pcpu_fc_cpu_to_node_fn_t unit_pages, psize_str, ai->static_size, ai->reserved_size, ai->dyn_size); =20 - pcpu_setup_first_chunk(ai, vm.addr); + pcpu_setup_first_chunk(ai, vm.addr, pcpu_vm.addr); goto out_free_ar; =20 enomem: @@ -3377,7 +3386,7 @@ void __init setup_per_cpu_areas(void) ai->groups[0].nr_units =3D 1; ai->groups[0].cpu_map[0] =3D 0; =20 - pcpu_setup_first_chunk(ai, fc); + pcpu_setup_first_chunk(ai, fc, NULL); pcpu_free_alloc_info(ai); } =20 --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022103.outbound.protection.outlook.com [52.101.53.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B499B4D90A6 for ; Wed, 15 Jul 2026 18:06:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.103 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138799; cv=fail; b=TcArws/kI/R03V1cVIqS7V+Wu8ugVTTcj5nJVYLBpnDB/R+PeQ+RPiB1tTlJ+RbwTNxm1A7BZ3PfD0SG2qbtw4YcoxOVCKNZmTxn4RT0u60cW5iN6wEwA3vaLbXlle+1oOoqlNuYibpgveX0az4HTJN+8e3ql0FrluuULkf0SJE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138799; c=relaxed/simple; bh=lkXGAl0zozci4vBGLUKvsd1XTQtS+56eXquI39Y4n5c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=E3/h58p4j6ZRLUpxWuby0g1d8HS25hkZTc2nAuLekYJjkJFknctAnqifMqPDBgEdZgQi3jU3Or2PE2EEFxW52GuJs7eeg3lTu4dkJzUr2vRnXYStZibuGoTrapExQAvh54t0h7G5Qaqeu3wELY20R+FpJxnfKMAzmyQZi+b/7W0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=i5k1ZoJN; arc=fail smtp.client-ip=52.101.53.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="i5k1ZoJN" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gpLzghd/1svgM+rZwR9kzPcSz+ZVTh4TGvRZU3jKuhbZomeKHH/LBVhbsnWjF6GYK7sf1JParV+14euWMEYblQ5Hs/90hrVqyW3UtA4xtM59C+HT+HSP+JNgN97QpCZ0N3NuSpfLitlnQHoziaDIOcUT38sTFEvN5prO4VRFQl2rLa3PYlp4tzw3QtKzBc4PSi4Js6vudnAmTDYWhp9OEtHk6TIA/NZuuMYkP5ixaWPnHEXfU3d07Xxqt6R86UF7unBak4PJHgSxBO7lL9rflEGCQITi+JM1tVqVoSp3LFL6+sQib7+Jntxx/937wQrRvgB+G87T6abfBpNL5k6m3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WOgFm9qhOLb+wxgXOI82bcxMevK7QfzP0zeD96Jih54=; b=YvMd4yS9v60bXWRK+Vv5PPnF1MtCHXdMwGuTB96YgT3kBAGPTSxVU7ttObLTlrywsKBCoEPGSXMlz8zaoqCUeDFlr0MnbH9oVKynHuqYhwamobElHbeyQ5IcuyvDl7Oe4++z8E3JIbxD+LsQ9wW53XA+6WioQp3smEEvzCQQNake/0878G2aI1d/g1Mvu3qtq+44Shluz2J++s3fTDU/oydRgylNvJeMiV5mpe9N7bQR1pLoTxNjTe5t/YZPhuQrHwPcxouVJsBDUBs6pZR2ERRJQNqT1FfASSru44pSNOVkUdkaK/NnTFOQxisOad6JkrXe8QrvCxQhH0eO0+LT6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WOgFm9qhOLb+wxgXOI82bcxMevK7QfzP0zeD96Jih54=; b=i5k1ZoJNNbidHuxd9Adl2ansOnBTAHdB/K4arTm9npbffwCJP3pKSSziZGyqO0PKPoNzrZ/KLmsOEuKiUlY4YGk5d+efnkJiMWy+NQQ+8ZbOAcESrSk/JXRyf9Cnaww3jox5fVomdBIdsDyu2ZJGZmFSxVxAC6tjJjNv7gZXBYI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:49 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:48 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 13/16] arm64: mm: introduce __per_cpu_local_off Date: Wed, 15 Jul 2026 11:04:15 -0700 Message-ID: <20260715180455.515692-14-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: e21ce412-e5f0-4652-c497-08dee29bb33b X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: MdJ9UsjtkFBgleoxW644LKL+3oVFm4lssz2WzXGXFa6+QEh4JzSiejElE/MiufYnHIxIUI98ARsFEBAzKdE/QrG+xRm3f9apK/LOYxMdcpR/ZuBiCyzz4v0cFvbwOvb/oaRXpvX+0wNnOUn/77E/iFVRpOHjCmdYlW7t5GiY30t1vuzaaddPcs/WZyAc+ifZCLv51sns2Z3tKXl0QMN75f/72nE/y0SwMcGSMJK3TDwDGEMzGYlGEDkxzsFdwqPysrRruzN14BI4GbZquJnoJMtsQiA1Mm5VBTTafI6Ku0l3L8C80yoqYwCbKIb0HRP/UhD7CEDtUmUb1As2ztVtd9dEqtriOZf6UwhidAsYAkBdG+2BmHY9m0dsmP3MDsBLYLOpVLimojADeN7ccuJ4h87YepNVqIP1qE35/GklZSlzCluid8ayeUXmVcDxVLn3tIa9iHeF+hQY3r8VguDnedGq/uavJ8vRBGX06Fqd0o5r0IwMWObYFU83+6dIyyd5wYOKsFt9mkMmpLxiEdYm7F46XiHn8dMkd+df5nc6p3tLmuSdK5ZMDoD1esVebdQFFsdxqXnA+6xCe+iBdSbmR759j8d4r5vk6pdsU8Fq+Y3KumZPzAvntYkoakVVAwgWrKzTzeKBPhEVs+Y8ETWBEZtSAzvcQKNZAIHu/kN6A9XMY8oANTfgC9yd2n7eQw+UqEzAksPL1OgjoC0lppSVzg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+t/BFO+ONFdzWcUepU9k+YbMQ7AZF7pgsU2yALztcMPGEJJ7BfQrlAjrVbiY?= =?us-ascii?Q?Iy8kDuxduZBgIEToNv5lBp345iLx/ra04PoYnXFpRP4rzIPB1Z0xm8mqidXg?= =?us-ascii?Q?tsCzXdThg+4+wrY9agjVHBMzsg6vQl7KxmsaMqC5U5z0Ox9uLz1ooiGUFNX7?= =?us-ascii?Q?nQRtuKYYpHu9DLwt7kHeoKDpECVLiRupnP5ux3QQP1sr4rTzqN2Bsjiex7Kd?= =?us-ascii?Q?Px63l+/J1n3y1XtemFD7Q/tOimdYfI9hCSryXCIXSvTCafCo8egov2YKXGTy?= =?us-ascii?Q?H/QLO5ra4yObRSy3hzoYKeCMbhjQAXyX/9qjNWtPyl22Ts0PM5G/o+Lnjdyi?= =?us-ascii?Q?Yj/DpdRMyHZYjK8asA//pqZ4sj2LLi7oAAu6LtiKxXAef7SjDyrlMJmxIe4d?= =?us-ascii?Q?PeyMMIM8mUpn3B+y0rum77a3xRwKi3D1byzYURuwVWwJWRvuQk3Z6Sm4NPqU?= =?us-ascii?Q?BaVLIcV5O77EjI/11eDRz4G0gjwewtNAoN48d2zdY79nu8/lXE6PO63V0Xa8?= =?us-ascii?Q?1j1u15v2rcMUxb0mP8dX0uUrZhJ2Gjo5MplnJTConIRFdy7D1kweCYOsTRUX?= =?us-ascii?Q?r5W6CrW4oco5A2rwF/vW0QDr9dbOVhcNHdZYN0mFjWXVkoOHgbemqBloEpTr?= =?us-ascii?Q?FLKuxIsQWpukrdtyou8ccQNDPBj5yXchSpet5EYxgRGlrHTdoRx3ycl13Xha?= =?us-ascii?Q?248vkipvghonUT3tFzyEBhAc9uvDl8wOa1VqUjfOA35C+s/9VhnLbJ5bDXSi?= =?us-ascii?Q?S9EkHEqlZd0zDSGLza5PZkDaAS1lbPXhcjoM4a/DfNhg4rGAVSI8k/iuyp5t?= =?us-ascii?Q?SiZHkk7booFHa3446SKlhzG0fFgcy7T2+kZD9V1iXCdR/qqhjSjm6ihA3W8g?= =?us-ascii?Q?TD1dcqNaAl0mHUr26bzPt5eRQm1uvDC7eDm3PRr/Cys6r9WjtY/IfEneKetv?= =?us-ascii?Q?bg3e7R1S1oepvHM4AoZgVvYVlyplJ4GAX+xK/2yNOT5gFZ97mcSzHUY+wQ/1?= =?us-ascii?Q?gjyiDyk+gHBQRzyoWZ4F0gGmfKezckUj2b5jXO80hdaKZRMoecoKxuV1znpW?= =?us-ascii?Q?wY948Tgf308UB+Jtnx/j60cm1ihSyAeJUxbrZw1UW78BfVyGOVrQyYm9qhpq?= =?us-ascii?Q?fDXP6vFXJ3VTXHN4AQ62s6N+3qV3bHgnYSvx4Rutlj/xuYHGtBpa4/Jgg5KT?= =?us-ascii?Q?dBKKSLU1HSjfJzGQ80VKKgEZrVO8If+Ec5rfj4sUS6gddmCqiv8iDxxgW4Nn?= =?us-ascii?Q?Q1RWMnPRN+m6w2xV5sjnjY5EDSATA8j4CuXW0/yV/jUXFI+4VbCErW3QyHeb?= =?us-ascii?Q?R8+WR9UpvTOl/pdB0lqZKhOE0XhJlzLhCptGzKg8iv1eruFCaL/i/SW2CmuS?= =?us-ascii?Q?nvIa7tcOi07/J0QZBKjxVHcL0Id24PwQ5gnMYBpTQl/amUpPb7bvazlLuqlD?= =?us-ascii?Q?wqe0K1bUzwlBRmwhX7rd/PWPanBdyPYA8pr1lIVorluKpHtdukypkpotZLFi?= =?us-ascii?Q?AR9+AXqeWPcdvG8Gc2hfuqMd8KwboqcxAS6c19euIj+CGiHMe5eyzGQt1glk?= =?us-ascii?Q?w7W1Ryj5LMQEBNwNJtPr6qzTVKITrZTio1uSuzoLu4JlXqwCsTjTREV1m3Pp?= =?us-ascii?Q?th+vP3VtGXsD+l0RH3BB5Pc8GaIZhR0iSESokWpS9L2+cBdpCK4HOJQPbxBU?= =?us-ascii?Q?M1hwy/IITz68Cq7f3koyCpLB0Z6U3c9x0snlZHQUqUJW9tPrQcFtY2+oBEwn?= =?us-ascii?Q?n1ZElq1G6Civb9gKE4yVO6qFqyyB/ag=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: e21ce412-e5f0-4652-c497-08dee29bb33b X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:48.9304 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: c6hCLL4v7z+ydJltN+7y+gZpwzXg0QA+t4kcK9OKlpD9CusUjD5C8sdyhkA/9oavAN6e1dkQfaCr7O6x5szZ7P/SIh2D+AyIQWRQUPUuYi0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" this_cpu_*() ops will use it to get the local percpu address. It has the same value for all CPUs. Also introduce pcpu_local_base, which is the base address of local percpu map. Signed-off-by: Yang Shi --- arch/arm64/kernel/smp.c | 4 ++++ include/linux/percpu.h | 2 ++ mm/percpu.c | 3 +++ 3 files changed, 9 insertions(+) diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 4c112bab8356..4a811379c191 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -839,6 +839,9 @@ int early_cpu_to_node(int cpu) #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA unsigned long __per_cpu_offset[NR_CPUS] __read_mostly; EXPORT_SYMBOL(__per_cpu_offset); +/* Used to calculate pcpu local address, the offset is same for all CPUs */ +unsigned long __per_cpu_local_off __ro_after_init; +EXPORT_SYMBOL(__per_cpu_local_off); =20 void __init setup_per_cpu_areas(void) { @@ -856,6 +859,7 @@ void __init setup_per_cpu_areas(void) delta =3D (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start; for_each_possible_cpu(cpu) __per_cpu_offset[cpu] =3D delta + pcpu_unit_offsets[cpu]; + __per_cpu_local_off =3D (unsigned long)pcpu_local_base - (unsigned long)_= _per_cpu_start; } #endif =20 diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 2f0e190aca29..5a27031dff1a 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -75,6 +75,8 @@ =20 extern void *pcpu_base_addr; extern const unsigned long *pcpu_unit_offsets; +/* percpu local mapping base */ +extern void *pcpu_local_base; =20 struct pcpu_group_info { int nr_units; /* aligned # of units */ diff --git a/mm/percpu.c b/mm/percpu.c index 15303611ecaf..8b273ed8e5a8 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -145,6 +145,8 @@ static unsigned int pcpu_high_unit_cpu __ro_after_init; =20 /* the address of the first chunk which starts with the kernel static area= */ void *pcpu_base_addr __ro_after_init; +/* The address of the first chunk local mapping */ +void *pcpu_local_base __ro_after_init; =20 static const int *pcpu_unit_map __ro_after_init; /* cpu -> unit */ const unsigned long *pcpu_unit_offsets __ro_after_init; /* cpu -> unit off= set */ @@ -3305,6 +3307,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size= , pcpu_fc_cpu_to_node_fn_t ai->reserved_size, ai->dyn_size); =20 pcpu_setup_first_chunk(ai, vm.addr, pcpu_vm.addr); + pcpu_local_base =3D pcpu_vm.addr; goto out_free_ar; =20 enomem: --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11020121.outbound.protection.outlook.com [52.101.61.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D0F242586A for ; Wed, 15 Jul 2026 18:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.121 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138805; cv=fail; b=uomFe94aciMzOJPaSQXy5Rbod2aS5mMN2lseGwEZLF3IosRbSb/3FifDe2EFZ1h6ngY5w40j48GNVc9OqZekeRS/FeoWeNWa9fbFflG0kqvg15+o6+YuLa7o4P/sV6OHyqSVbCZvTnj8EnkVzJS1+YlDdknTgsCrDn6ZuSKUXKE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138805; c=relaxed/simple; bh=k9K+ACUDza734RIDCxs8oR6XTwbTJUSbrC7uyChqAVk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=AEnFVc4au4YqhBnhXoZ+EhKYBBU+yszo+WvBWiTLmSa1HbraODpAil+N7Mentf/IMkx/9/3VqgRdYjB8F43y2cIPQxDNn162187WO1T5fsxwWvDMCng8ldMP9AI2UNeJSGucVqisgmQZ0JSM/9jue5jRmCICToNwCBlahAP8k/M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=MceQqPeZ; arc=fail smtp.client-ip=52.101.61.121 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="MceQqPeZ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=slt6WGMfpg5v+hO67u7WwX9IZ04+JvfDETkxeW7FA44E6sqHBuWAAYXUvirxjdXYHC8lIwYDFZy19IbuyHMaoUkr5DmVN89XNx2sujroJy6xm6wRiQTZEkp4GRh8zOJNn8wsWtKcPUgYPCWcB1pMggEZK3pAelnjFktiiC/pNB4EqAu0JXIwHb6p2TNaTLL3o0kE8t1FHqeH68EPdCiPcwSGwr4F68oIulLn/5PJ/fIEF/49E4JI+b3wKb7g8OyQNBKmLWkya1fILU0mxn2gAiyF+fxYWH4SLof/wuzwPfF3daU2DPx7MslMll775/b18FrJkF/sj7U7yIj2OANumA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=O0cNl+F+N/MuIxcquS5c8uEGtyAEfWEhi4ngJqfJiDk=; b=qLFk/kaKbT4zI9oBMRMBJ+hs6SZPLPJrmlhV9qvoSj0xd1/diJwKrd4AXBKIZItIpvlWAxAHJmlgLj05d0re1WyNaps/uRdoTECFCArbtjujvOxdzX2zgEYoY3csXD56OcuZb1qwXSm5Twgb8+M+xBuH0JG2tkuiyDKCvwPGblcdL8HZgJ034aDmOckuLYScW9tFLLpp2KJynRVIgGLLbKfPVEwufzm61cV+S+FDFB6bjrzi5avUgbCstVpUb14ku2JIcM5JmWvx1nZNYIR8Sx0xHZSxlRGrXXXa8cujcAca3CqQlTBvKqYWNVrsErOSS5GFA2ThJTzIXfanY0e6gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O0cNl+F+N/MuIxcquS5c8uEGtyAEfWEhi4ngJqfJiDk=; b=MceQqPeZ6xudZmh+n/2knrcG0PpsP3OYBNAjdP+YYO4GNUTDmvdkcbKN4Uks6HgOg29DtpsqzZJFg84e/eZCToUfOj8Tu1Hq8hXfOJhtAueoyjxg8h7IX/6ugvL9pFmgh6GMyWtnsE/N6g7/Sey19n8yfhf75NoLrjHZSRooczw= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:50 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:50 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 14/16] mm: percpu: allocate and free local percpu vm area Date: Wed, 15 Jul 2026 11:04:16 -0700 Message-ID: <20260715180455.515692-15-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: de4ff39c-60c5-4af2-94eb-08dee29bb3fc X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: aeYk+Ocf0i2ubpnXZYXyre0SXJ/oYioPwKXnGJuXVoutVTUWnq5XMJ4898xyNy2S9smymVWPco/dYmeWNiocrNlaKEx8WIY9QkNVU4TpvrW2+bbv9sXB1KzReHZc5mEZVD7HfEsY9xKioK0Q8gEgN76OPBMiaFQ1E+RytHmT4SKFsN3tfMnT7emPJ0XTx/VhEnU+mfO0FGuWlIOwjYfJl8CnJpfkqgXkTCNuSkK8yTDyx/+LRrzaCQUz41v52NK4NK8indZHuQkDWnOr/0prM1tASseJMMm+4j46EBjEn5Sk+5sB99l1ebU0mX5XCitf3IxiOQd9zBu5494iXmtbvakTz3w00os8zf81S/2vh5my4zPIY7wlpT4fCn9TyBZXAfL0O1EWiiOeTTDAZ8BpzQO+F4zFNpHyWWnAuXF07ANFhKcw4x6WQAg7C3RtNaRIdErLIb9Y66GZgtBNXcGdprGjAxCw2rTX0yRlTlrQocRAoyI5g2I/uN4I7Xp6ymqQSR4nt4FEb5pI/R+jWQeOJufd7bAeYbQ6fuDXUyodlCi8vRwTDYZ+nUMNZ3Losq4+qUgfJj9isT8h5c/DmRvDXY23phdgehP7Yr63G2EEjH82Aqjn/A0uGv9Q9ZRrkJ0/bo7BTPZ9HCzap1Nr+0n5AHFemjC7rPoYIx3hjDdmvu9lN8NPndY4seIaOMe6JIGIKTWQ9C97hHU4C4xBEI3jpw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?k3Z2msaU/uaNwTlauFJjp/J0xUWWs3b14Y3kcA5yCKa8+B3Qv/mvo2545I76?= =?us-ascii?Q?i/90C0YQ5SbzYnxA5YoewigG6EgjcHlKkLfmBpF95Ri9mcVUojLqMd7mRwcq?= =?us-ascii?Q?DwBoYSNaXgp6f4qv6t1JBPqlJSQl0s7gZ+LWbShL7iOcR4MXznbsJt10MQS5?= =?us-ascii?Q?2Ul7hN1/mnW5lZlALqJDEL3IFWBA2Zy6FjAPCCuyHLovbATyZxAJyLUas+JY?= =?us-ascii?Q?PJuI6djNkK4Xpl4lgnQ0UcAyphxhDPYAVrPpNl6WNtZ8/bIXbvpoQaqy217t?= =?us-ascii?Q?iXtHEyaAGQb9DVRP5ok3Qlgl3MxstsTISu/HHDhoKgq+FW6QI7T7wpUHCV8c?= =?us-ascii?Q?kQUgBNf8btr3sDHmdznQm1KCO6Qv/yk4swp8rYl+vIJ8oE54gXk3ySCLAnM4?= =?us-ascii?Q?m7a1ONg/NOwXXEb0gsRZfjtUCY2SvE43l6Lx89GtL96IMmIOjBvOpFNaR+Mw?= =?us-ascii?Q?T2nQ2eOv9wGK/ZOG0k0TPvYC9Ox8XD1CsNSQ8M959KDGKVVVEVWPRCOZpsMW?= =?us-ascii?Q?/2XSai/f1U4xopFgYlaibRe/oHFKe+nT3eX6ET/0bf9p+Xs+6WTKvEBp6Duo?= =?us-ascii?Q?ZXrD6M+8SOeBV1V5g+3MF+jkU9UXIVvKpOstx//zQwiQdA6RYSaBJ9BjmiUD?= =?us-ascii?Q?3R/NasebqmGfHkDb4cpD7Zgm2jbAJJUtGcdXJB60+4TjN9h8bbuALbcJmE+k?= =?us-ascii?Q?BvpD75svuycgNEiiD5loUDParOh34ePXKInK2KvYxJK/qh4KQw6kv6l3SBbM?= =?us-ascii?Q?XyqjdYwuriXxIf0oq/T+vBv2HXM5mSlL2f7wB1yODvXOUATq63MlXl0JE+4R?= =?us-ascii?Q?FzYySBQWeS2gyB5OIj3Aa6kkI1GYK0P/XgvS2TrvshO1IIs5aloLfABckjf0?= =?us-ascii?Q?3o9TUxOTo0HKCaUAWjEXAVWQYOVyivnazCNAtCzNfpkb9hsdUUjFE1hiSXKt?= =?us-ascii?Q?ao5srFY95xIuc11MOzOaUhG62uN7G/9/PRUQ2dTKrEO14LXbnZ5zkH0qi4tY?= =?us-ascii?Q?kXa2oCKtc3+zj6KcqLg6ZEJHprbsrys9bpC7JnQk1YxnKlsUrvATVyTew8Lq?= =?us-ascii?Q?p6RoRm9eyWZQGbi9I53bDfGizW6A5GywCl8Hw7OUvvGKveBE+rIAkLovwMVI?= =?us-ascii?Q?1VrbFhKmF4ElNPeOyIVQwz35LnsEShJT9KL4lBn2L0kTdEXY559Lxc5izJtm?= =?us-ascii?Q?2A2p+tN1NdzDFjUSSljGnuHCIeNG2zAIRY1lgTWUoOSPw2Cw39eTl2o/dJzQ?= =?us-ascii?Q?nEqT+/j8JPeRWBvNsrOqagveyG3KzoJ5fRgNpgvRQkdC9w7Zp69hk5f93d92?= =?us-ascii?Q?8u6fVf+xHDPiaC4X3Khdw8FwN6FFsvlkqU9IZkgihc+9myknS9dRRDlgneNY?= =?us-ascii?Q?xsviGonZ0yuCCUgvrkWAPcIH3S2vgdGWquS13mtge6ZHcRO4exB8L7UCKvnT?= =?us-ascii?Q?Wcbs40z0fz5TAVfA1NFNTfKwZVRqyp/GEvdxQeXajGbn4QVeyBrPx1vBy0T0?= =?us-ascii?Q?FcTtMqCCWjLZY7T69dcSDfEZzh6IGg23LMT7+xAPlZJtooRo7hmDaxPh7Iyk?= =?us-ascii?Q?NoU2e2UvACJE3bSC7KCFmNmSt4oO55HIGDWiItVEfDpOeuA1ca/UXsRfEHqe?= =?us-ascii?Q?Rj7BGIbrD7BvdwoqDE1MSePmOQW+wmbus7p+7U06jc3SIL3c4pxLSoxnY0t9?= =?us-ascii?Q?SkE7MoGfdEPGtw97AM+RLHFWLbYL+dxcb9LCqgr8SHEv+rMinUKJcBHT729q?= =?us-ascii?Q?oW2vj8KhcYGWucvD436mSD5YwTViYl8=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: de4ff39c-60c5-4af2-94eb-08dee29bb3fc X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:50.1822 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qcBg44gWhAxN3xigMQvYSm79TO+Ixq4KR9/dNG3leuBGu/wtAfqYx8I+saTHkzrdzOJrKqj1Nu7G0BiLE0AvnXHP32mMsPGfKyxsP774jXQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Allocate local percpu vm area. The delta between the allocated addr (chunk local base) and pcpu_local_base must be same with the delta between chunk base and pcpu_base_addr. Each CPU's local percpu area will be mapped to its own page table. This section of page table is not shared between CPUs. And free local percpu vm area. Also unmap from percpu page table. Signed-off-by: Yang Shi --- include/linux/vmalloc.h | 3 ++ mm/percpu-internal.h | 1 + mm/percpu-vm.c | 94 +++++++++++++++++++++++++++++++++++++++++ mm/vmalloc.c | 92 +++++++++++++++++++++++++++++++++++++--- 4 files changed, 185 insertions(+), 5 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index d87dc7f77f4e..daf9965a111d 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -313,6 +313,9 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned lon= g *offsets, size_t align); =20 void pcpu_free_vm_areas(struct vm_struct **vms, int nr_vms); +struct vm_struct *pcpu_get_local_vm_area(unsigned long hint, + int unit_size, size_t align); + # else static inline struct vm_struct ** pcpu_get_vm_areas(const unsigned long *offsets, diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index ec2e55a2f836..4d461305243b 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -70,6 +70,7 @@ struct pcpu_chunk { struct pcpu_block_md *md_blocks; /* metadata blocks */ =20 void *data; /* chunk data */ + void *local_data; /* chunk local vm */ bool immutable; /* no [de]population allowed */ bool isolated; /* isolated from active chunk slots */ diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index 4f5937090590..5decaad7640f 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -112,6 +112,10 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk, return -ENOMEM; } =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +extern pgd_t *percpu_pgd[NR_CPUS]; +#endif + /** * pcpu_pre_unmap_flush - flush cache prior to unmapping * @chunk: chunk the regions to be flushed belongs to @@ -130,6 +134,11 @@ static void pcpu_pre_unmap_flush(struct pcpu_chunk *ch= unk, flush_cache_vunmap( pcpu_chunk_addr(chunk, pcpu_low_unit_cpu, page_start), pcpu_chunk_addr(chunk, pcpu_high_unit_cpu, page_end)); + +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP + flush_cache_vunmap((unsigned long)chunk->local_base + (page_start << PAGE= _SHIFT), + (unsigned long)chunk->local_base + (page_end << PAGE_SHIFT)); +#endif } =20 static void __pcpu_unmap_pages(unsigned long addr, int nr_pages) @@ -137,6 +146,20 @@ static void __pcpu_unmap_pages(unsigned long addr, int= nr_pages) vunmap_range_noflush(addr, addr + (nr_pages << PAGE_SHIFT)); } =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +static void __pcpu_unmap_pages_local(unsigned int cpu, unsigned long virt, + int nr_pages) +{ + __vunmap_range_noflush(percpu_pgd[cpu], virt, virt + (nr_pages << PAGE_SH= IFT)); +} +#else +static void __pcpu_unmap_pages_local(unsigned int cpu, unsigned long virt, + int nr_pages) +{ + return; +} +#endif + /** * pcpu_unmap_pages - unmap pages out of a pcpu_chunk * @chunk: chunk of interest @@ -166,6 +189,10 @@ static void pcpu_unmap_pages(struct pcpu_chunk *chunk, } __pcpu_unmap_pages(pcpu_chunk_addr(chunk, cpu, page_start), page_end - page_start); + + __pcpu_unmap_pages_local(cpu, + (unsigned long)chunk->local_base + (page_start << PAGE_SHIFT), + page_end - page_start); } } =20 @@ -188,6 +215,12 @@ static void pcpu_post_unmap_tlb_flush(struct pcpu_chun= k *chunk, flush_tlb_kernel_range( pcpu_chunk_addr(chunk, pcpu_low_unit_cpu, page_start), pcpu_chunk_addr(chunk, pcpu_high_unit_cpu, page_end)); + +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP + flush_tlb_kernel_range( + (unsigned long)chunk->local_base + (page_start << PAGE_SHIFT), + (unsigned long)chunk->local_base + (page_end << PAGE_SHIFT)); +#endif } =20 static int __pcpu_map_pages(unsigned long addr, struct page **pages, @@ -197,6 +230,32 @@ static int __pcpu_map_pages(unsigned long addr, struct= page **pages, PAGE_KERNEL, pages, PAGE_SHIFT, GFP_KERNEL); } =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +static int __pcpu_map_pages_local(unsigned int cpu, unsigned long virt, st= ruct page **pages, + int nr_pages) +{ + unsigned int i; + int err =3D 0; + + for (i =3D 0; i < nr_pages; i++) { + err =3D vmap_range_noflush(percpu_pgd[cpu], virt, virt + PAGE_SIZE, + page_to_phys(pages[i]), PAGE_KERNEL, PAGE_SHIFT); + if (err) + return err; + + virt +=3D PAGE_SIZE; + } + + return err; +} +#else +static int __pcpu_map_pages_local(unsigned int cpu, unsigned long virt, st= ruct page **pages, + int nr_pages) +{ + return 0; +} +#endif + /** * pcpu_map_pages - map pages into a pcpu_chunk * @chunk: chunk of interest @@ -224,6 +283,13 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, if (err < 0) goto err; =20 + err =3D __pcpu_map_pages_local(cpu, + (unsigned long)chunk->local_base + (page_start << PAGE_SHIFT), + &pages[pcpu_page_idx(cpu, page_start)], + page_end - page_start); + if (err < 0) + goto err; + for (i =3D page_start; i < page_end; i++) pcpu_set_page_chunk(pages[pcpu_page_idx(cpu, i)], chunk); @@ -233,6 +299,9 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, for_each_possible_cpu(tcpu) { __pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start), page_end - page_start); + __pcpu_unmap_pages_local(tcpu, + (unsigned long)chunk->local_base + (page_start << PAGE_SHIFT), + page_end - page_start); if (tcpu =3D=3D cpu) break; } @@ -258,6 +327,11 @@ static void pcpu_post_map_flush(struct pcpu_chunk *chu= nk, flush_cache_vmap( pcpu_chunk_addr(chunk, pcpu_low_unit_cpu, page_start), pcpu_chunk_addr(chunk, pcpu_high_unit_cpu, page_end)); + +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP + flush_cache_vmap((unsigned long)chunk->local_base + (page_start << PAGE_S= HIFT), + (unsigned long)chunk->local_base + (page_end << PAGE_SHIFT)); +#endif } =20 /** @@ -349,6 +423,24 @@ static struct pcpu_chunk *pcpu_create_chunk(gfp_t gfp) chunk->data =3D vms; chunk->base_addr =3D vms[0]->addr - pcpu_group_offsets[0]; =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP + unsigned long delta =3D (unsigned long)chunk->base_addr - (unsigned long)= pcpu_base_addr; + unsigned long hint =3D delta + (unsigned long)pcpu_local_base; + struct vm_struct *local_vm =3D pcpu_get_local_vm_area(hint, + pcpu_unit_size, pcpu_atom_size); + if (!local_vm) { + pcpu_free_vm_areas(vms, pcpu_nr_groups); + pcpu_free_chunk(chunk); + return NULL; + } + + chunk->local_base =3D local_vm->addr; + chunk->local_data =3D (void *)local_vm; +#else + chunk->local_base =3D 0; + chunk->local_data =3D NULL; +#endif + pcpu_stats_chunk_alloc(); trace_percpu_create_chunk(chunk->base_addr); =20 @@ -365,6 +457,8 @@ static void pcpu_destroy_chunk(struct pcpu_chunk *chunk) =20 if (chunk->data) pcpu_free_vm_areas(chunk->data, pcpu_nr_groups); + if (chunk->local_data) + free_vm_area((struct vm_struct *)chunk->local_data); pcpu_free_chunk(chunk); } =20 diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9ec401918736..3d94ca4bbd17 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -4923,17 +4923,21 @@ pvm_find_va_enclose_addr(unsigned long addr) * in - the VA we start the search(reverse order); * out - the VA with the highest aligned end address. * @align: alignment for required highest address + * @pcpu: whether request allocation from local percpu area * * Returns: determined end address within vmap_area */ static unsigned long -pvm_determine_end_from_reverse(struct vmap_area **va, unsigned long align) +pvm_determine_end_from_reverse(struct vmap_area **va, unsigned long align,= bool pcpu) { unsigned long vmalloc_end; unsigned long addr; =20 #ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP - vmalloc_end =3D PERCPU_END & ~(align - 1); + if (pcpu) + vmalloc_end =3D LOCAL_PERCPU_END & ~(align - 1); + else + vmalloc_end =3D PERCPU_END & ~(align - 1); #else vmalloc_end =3D VMALLOC_END & ~(align - 1); #endif @@ -5042,7 +5046,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned l= ong *offsets, end =3D start + sizes[area]; =20 va =3D pvm_find_va_enclose_addr(vmalloc_end); - base =3D pvm_determine_end_from_reverse(&va, align) - end; + base =3D pvm_determine_end_from_reverse(&va, align, false) - end; =20 while (true) { /* @@ -5063,7 +5067,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned l= ong *offsets, * base downwards and then recheck. */ if (base + end > va->va_end) { - base =3D pvm_determine_end_from_reverse(&va, align) - end; + base =3D pvm_determine_end_from_reverse(&va, align, false) - end; term_area =3D area; continue; } @@ -5073,7 +5077,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned l= ong *offsets, */ if (base + start < va->va_start) { va =3D node_to_va(rb_prev(&va->rb_node)); - base =3D pvm_determine_end_from_reverse(&va, align) - end; + base =3D pvm_determine_end_from_reverse(&va, align, false) - end; term_area =3D area; continue; } @@ -5236,6 +5240,84 @@ void pcpu_free_vm_areas(struct vm_struct **vms, int = nr_vms) free_vm_area(vms[i]); kfree(vms); } + +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +/* Find free vm area starts from hint */ +struct vm_struct *pcpu_get_local_vm_area(unsigned long hint, + int unit_size, size_t align) +{ + struct vmap_area *tmp_va, *va; + struct vm_struct *vm; + struct vmap_node *vn; + unsigned long end; + int ret; + + va =3D kmem_cache_zalloc(vmap_area_cachep, GFP_KERNEL); + vm =3D kzalloc(sizeof(struct vm_struct), GFP_KERNEL); + if (!va || !vm) + goto err_free; + + spin_lock(&free_vmap_area_lock); + + tmp_va =3D pvm_find_va_enclose_addr(hint); + if (!tmp_va) { + spin_unlock(&free_vmap_area_lock); + goto err_free; + } + + end =3D pvm_determine_end_from_reverse(&tmp_va, align, true); + + if (hint + unit_size > end) { + spin_unlock(&free_vmap_area_lock); + goto err_free; + } + + ret =3D va_clip(&free_vmap_area_root, + &free_vmap_area_list, tmp_va, hint, unit_size); + if (ret) { + spin_unlock(&free_vmap_area_lock); + goto err_free; + } + + va->va_start =3D hint; + va->va_end =3D hint + unit_size; + + spin_unlock(&free_vmap_area_lock); + + if (kasan_populate_vmalloc(va->va_start, unit_size, GFP_KERNEL)) + goto err_free_shadow; + + vn =3D addr_to_node(va->va_start); + + spin_lock(&vn->busy.lock); + insert_vmap_area(va, &vn->busy.root, &vn->busy.head); + setup_vmalloc_vm(vm, va, VM_ALLOC, + pcpu_get_local_vm_area); + spin_unlock(&vn->busy.lock); + + kasan_unpoison_vmap_areas(&vm, 1, KASAN_VMALLOC_PROT_NORMAL); + + return vm; + +err_free: + kmem_cache_free(vmap_area_cachep, va); + kfree(vm); + + return NULL; + +err_free_shadow: + spin_lock(&free_vmap_area_lock); + va =3D merge_or_add_vmap_area_augment(va, &free_vmap_area_root, + &free_vmap_area_list); + if (va) + kasan_release_vmalloc(va->va_start, va->va_end, va->va_start, va->va_end, + KASAN_VMALLOC_PAGE_RANGE | KASAN_VMALLOC_TLB_FLUSH); + kfree(vm); + spin_unlock(&free_vmap_area_lock); + return NULL; + +} +#endif #endif /* CONFIG_SMP */ =20 #ifdef CONFIG_PRINTK --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022074.outbound.protection.outlook.com [52.101.53.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFD9D425873 for ; Wed, 15 Jul 2026 18:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138802; cv=fail; b=AFKe7OjU4tiaT7aDAf4r1LznIRWCih5VoMsDEn7Lzj1Hx9eP0nRKpNlgizQVwrN1fC83G7fieWSCK7UipNzNYd2IvaUHa5N0os4g0KSGelROzakIckDG2YfOpsPp9Ft7dUMvttqbXkbqir+1R10WzyjlomLS1YjPQH+TOZv5iek= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138802; c=relaxed/simple; bh=LuXgEoLZHcWebxhlGaO8nR5IsRn1xhGnkQdX3fIQ2jY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=urB5znR3W2KtEK/B97v1tReFPf/drGd5Yr/cpkDz16GT70v4HJr5anazAerSce/NUKgEA1/218I631BJ9fqtHMINpEaaxmpRmVxnLJilgBssv/3ggrsa/AiPlXT/n0a9oIcP/0g7bvw7z3A9+Z4EDXTGrpTIZwW9FJm2O7pxvVU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=dvGpYiPV; arc=fail smtp.client-ip=52.101.53.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="dvGpYiPV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=a1hJbx0H5xPh0gswX9vlaB2Uuo2ZVVsMJkmjW8vH6wNswhhXeCCEaV8OC6AeRClW4C1QNkf25xzvbiab8OLkcZK0/oGz/LbOqIr2QUElbdEB3ALLwvi4pP2ni6QAuhQCNHTDULRbjKAmH0bwLM3oqrIZIgPTDcn5P+WZ6tfSZXivz7LBxeOboSFudHL9VxDAo0drxlq+e79TqQfCkKENncwA/3ufJ38ofqc5j2moYPYq5yyFDm4hGOAokkr8vumew57uEjs8Ag6lc/BBlVl2YXFiG4VnUmzT853rFWsgfmEtEjddBzhIoSeSou4vqLENb/n3rqHDPPjXdk58wSW1bA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DpaTpdar5NmCn6KDifPZIoCGv+BUoPNwYtyDeM8tyOQ=; b=YxEjHtD/ORa0pk8+g8lRPqbuAo6e+oWCb6o86L8SzzH9UMeW3Y2Z5Cxh/Q4bQmSNREYuYxPCkuDBYvqWZJ7NMmDnze3ucgWZDfliWFkay3ndaaZOr7jEKQ79aICZxSmGBGwWTEkcrYN9Ph9D3L05lANzW83nxJiTdhoEoVMZH4hRNerGcWi6UzshWeKV4Vf+Ql2rh7CLbOxC6TeQ7O/9c8sHGZ8Un6PBbkGarVT079lLsO6iNAtMoGObfbOYOwWamU1gx+oSqQZduBbXv+3UREyFF+nOpuDvCzVWmZQ1xrlTSrZtR8nQh0u6fC44TT04FyCiIEIXRR17yRZIakC26Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DpaTpdar5NmCn6KDifPZIoCGv+BUoPNwYtyDeM8tyOQ=; b=dvGpYiPVywlL2zHHNhQB++U7y5TE4Efge/wqlQ0NQH2y88JhbTwCojiXrL2Iy+iqDAhXELMBStqc6azIO3QE2uYQaQNeub8yOQoEtBGBlPNKGjGKKzkMoH0mmI4PMVs1fqJsj8BimyO1bt1N8oQ5o4igwqBNFNiWONuowODVuig= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:51 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:51 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 15/16] arm64: kconfig: select HAVE_LOCAL_PER_CPU_MAP Date: Wed, 15 Jul 2026 11:04:17 -0700 Message-ID: <20260715180455.515692-16-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: ae6172e5-7a90-445c-9f29-08dee29bb4bb X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|921020; X-Microsoft-Antispam-Message-Info: Re9mHU3BS+kL/MeXeXlTBcQPME3ja3FPtdC8Uocrxx4D8wKbRL8wigsyY8OK2+dNjHbO9CsDeWRZSI/rfb0x0qOyyYf4u2aB3jPDXytY0aoSNaFGM9ixevnCuROGpgjsIlQtJSpVlHqPc8/6edMuodfjBwPzdeZQ32xnXJl31GagNI/mPz5dO8qobS6M3dAWODWmaQUBk+havdLF7gvca0VrEvCj/2tU+ul+eDm0XHLG2w2ui1eR42/pXEzmQpSH18voxUTNBWrT6LEpk8a/wf/MDg2hzxmsmX/w8TcrJiDI58fuyD8RwwglK3+Bl5EQkkf8BNXaRYtxqswDcbRvCeht9hgnhRPCtrlIV6d4fNGBDxfcRSavYoNu79a/XCzhH1t9wHHXkxKPP5gsEChc08fNNTOUPKhwXohwZdB6kyJjwmTR20toWKWLfGkG3LrLUvfTVIrVKe5ECDaEkSHacgtwq8ptL+nQaqTRbS/0GkZepjJsuXESmCl7ygA0sAoFPRFe0nfYmvmPJCL8omnyg3PNhryfcoYweAlyb9ji1z4znn5kgtBdcI+GCzbfl5iomgD3t+Rpx0D5B8q/mARkn4yo4QJ8uMne89a59J3HE+/pdFSZzTk1x9Ly7OFkQNM3F9I6757fcghBVu8z5UAcfbfBbjVzph0UXaUNYYF18E8ps0Bm4a5qFaQLuXaSYx2UMJoKMBHDo3iYFQMm7p+D9w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?8x+/HOtolt31i2OGfFuBtzNf+vV290Ccf22gpVKWTW6KEGyNZhN47rduwu1n?= =?us-ascii?Q?5ha5W0aNsmlcCj3HTTluMdjuoBaV+eQ7MQuk1bTw5wVxaeMQAzq3fSgJ6w7n?= =?us-ascii?Q?VYf0sPMkefePaDr5R+giEM3j+TYa2n3mIPoCD2VCVgDMWdcRzQE4U41i1i2R?= =?us-ascii?Q?bkZym31EC2Jd/Y4Nh9+OFbVZfbSS+SW1S/zDybelpIEtsHCxT9SD6Vc9Ifjt?= =?us-ascii?Q?E8xrQaI6tnMZr4kVeHSz1tE5Q+juS1WuR2J+ZIWFXa1Gz/DNUQhVtMdeZYp+?= =?us-ascii?Q?oXQyQ47+s2eUBsMiGkasS3RnBkvpICsjlaxK4f11MFQ/8yrI/ypJm/zjCrJi?= =?us-ascii?Q?r93LSuNrM6xA1HScWADV6naIDd9kLvaV7XHDwi6Qi9KqKwuoK3BtTKHgKf0u?= =?us-ascii?Q?UB1LMgh5QPEykIaVvPMvGHofkFRn1siwTYFgG2Pzs1WJjusAwrSsnbTi9dUl?= =?us-ascii?Q?QXVq+dzpA5Wupr+MFRk78x/qtr0i8OsYhmxSWJtRYiAK9XtMKRTXsY0fOYfY?= =?us-ascii?Q?TxKJvd+GLok6BGr5ll3uB25UhXvDL23jr95j6jtrjtoXk/qlHx0niqRdqgto?= =?us-ascii?Q?M4a8mPRJfuhpArJ4uQQ4SLnVEgmJyh+qnpuihweq6Mv89NlUObVXd+MXM9R3?= =?us-ascii?Q?8LvRxllIWaSgjWmoZYppA7NwlV9pC5qXBIqRpT8B6WL2W3qAjMAdRV/S+koU?= =?us-ascii?Q?1DD426sm23ibxVVTDuV0yta9QhHRLWOghUNejfZCticukaSHGbZsQCd33h5y?= =?us-ascii?Q?9GA4dhX5t6hAU4HfdG1iNoSGtOFsRRknz//Fjf6+lQtIX4IBfZFbBTac9/64?= =?us-ascii?Q?tvNAFE4fnQeMG/KMNTvSt9BJekqbouGyj62vYiydE/w9qKnwjOkZ8EdTYIuj?= =?us-ascii?Q?uQPY/vOabOX2/OYJftDV9KtdSKAk48Jts6RhrACA79sEJyKVUlkh9BlpVvar?= =?us-ascii?Q?X5FxDWQYd033h965X7r3YbEdC8NBe3RAvjObtsEWjZQ+v9BtduD8BNeN8l3Y?= =?us-ascii?Q?wC4fz6AIlK7A//kSS/ejNf2Lq5G4OKbnqB1VUHquLIXDtDEKOY6itYsOcAqZ?= =?us-ascii?Q?QvwychLGkBhY512fEhXTmhQ8vh9KsIdvq0LEqRUPgClb3m4hsTd1z5HQ+uty?= =?us-ascii?Q?VYmcvg2veGk+0FwsyfXqT9Tvv1fi8nS/CgGz3UQ3m0gZFNUWCP+GLcPBYr8Y?= =?us-ascii?Q?/dCDzs8psCixqgtTN0kcBWgLReFNMdnltgkmDf9mbEpfAqIcw0k2ADJ0guOj?= =?us-ascii?Q?E7n+qVTm15UOn5U6e6djzddroUKh2+cjgXAbMJc9RTjwQ9NCc+MytGfTxTjE?= =?us-ascii?Q?SQSBop+srj3LJ5ikR/4r7/Hd7jmmMpHbr9oaL+AZPRql7wWpWS6lJperrlCV?= =?us-ascii?Q?CIbzkPf29Yd5LVKTwtVFX0qYF1axPKzvIRLLQP0Bw/BQmhMoONxZd9SuwC3j?= =?us-ascii?Q?mkbD9J177+fd6ArC/RAE6hyIujYoicO9KiZKnMANvCHENgkNPnJwqikTXsHa?= =?us-ascii?Q?6WUHu2FK4GAG/Fyq7tXL3O/ms2bYo+j627wlxGGEBAyTHy9F8iFmRY4gnBm3?= =?us-ascii?Q?mV1dLZv6GMVyUNSOsTSGxKK25QP7oubuhsZF6b5prRdxcqj+Yey8EgAb3yBG?= =?us-ascii?Q?ZEI4lPu+JNXnNHMmzfGNmPZtC4NQDK2REHbIXvRd+kMMEEM+4/a1WSEgtcrM?= =?us-ascii?Q?f0G4zM/GWfqHZpPMIOE0yYc9we5RCpIM2Jq25U+KtVdz1E3eLbKFgqIfe/BX?= =?us-ascii?Q?BKgVr8OxX9Ko3/Ofz0XPhlMgDPjTj3Y=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: ae6172e5-7a90-445c-9f29-08dee29bb4bb X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:51.5077 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: InASsdjKg1A3e4RsMCaX8w1kZT7het6FeFfB/SHJceVsoNpREyLMiGnZD5MpLskX/fiCLY8N+iA4fwKnaBu4ghSVfWWG3BtNvYTjvbO861k= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" ARM64 supports local percpu map, so select HAVE_LOCAL_PER_CPU_MAP. But it depends on VA bits and page size for now. Signed-off-by: Yang Shi --- arch/arm64/Kconfig | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index df55e6bdf81f..95ffd151187f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -415,6 +415,11 @@ config KASAN_SHADOW_OFFSET config UNWIND_TABLES bool =20 +config HAVE_LOCAL_PER_CPU_MAP + def_bool y + depends on (ARM64_VA_BITS_48 && (ARM64_4K_PAGES || ARM64_64K_PAGES)) || \ + (ARM64_VA_BITS_47 && ARM64_16K_PAGES) + source "arch/arm64/Kconfig.platforms" =20 menu "Kernel Features" --=20 2.47.0 From nobody Sat Jul 25 16:48:31 2026 Received: from BL0PR03CU003.outbound.protection.outlook.com (mail-eastusazon11022103.outbound.protection.outlook.com [52.101.53.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51CDD40097F for ; Wed, 15 Jul 2026 18:06:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.53.103 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138806; cv=fail; b=BnwIc58AnkL3AVQFR7oAgDe3uXA84ffyvw6CkQSlSsg4HXVbwZuXZEB1/FZb2O+aB5sdFsoXebJCY4UUUsUYIQRjfAuLdaykVDoLQXxWGuMtqNicaffZYTlA3OHqGmMZPjGuhGytPzfh9lAYaKBVT30qobvdiM/zTiJAEcSPSvE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784138806; c=relaxed/simple; bh=R9og4W61loa89Z+K8Jdkr82Wwq09DD0c3ehS/Bgf6jg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=EoIOmKLVrn2mpWU7MQ/cB6RppmgU3YZxtYlCfI5AXqs20mdv23b4V3sFBDQLDRmZMka0I9ia2dJeW4rV/AeaEgg/umjEEh8i8ObX0vIijQQ6+zyub+41t+lmKcGfKQVWYEDuFPcbFcRsUphsoVOJlWTNPEV5LqMp5qX6TO8KV7Y= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=hi47ZyJZ; arc=fail smtp.client-ip=52.101.53.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="hi47ZyJZ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AxaGw3mOBZa3MsKXfTHKuXoBKuOFCD1qbdb0sG6I4NuSV/FhmCpdLKZSIZdW9HcpKqW3vjrHGXZ0xVu0X54AugAoYVPY2jKxv9sQz2dRXS/ir0SX6CF2db7PASMNqs+Pf21fDCH/RuBDxdZjk0ge2fm85/paA91UWmlXttJGxRtv+195wx+O86SYhfYiyzGZEnd/dqwfDZs+uHX7RN/HPO4dRL/IzpJrogGEx2Oo57cSHyyDdMpdM1qW6/qCqy1d/4Q/QtbbWJiVjkiiWRBYmx6UTLfnuwXliIHUzpvfTBrK4M5qFesZqoLYqJsMxP9UI//k58+LSQMZsLLH1z2kdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pNNyNTfsOG8Ek82+6Ao7fqrhzwIJ/uNOESZvwBJE5oA=; b=gDqy/DAePtXq4IK2T0GikM35GPEgG0cTeo1FS6taXEKKuWhQpdSkymqcJObV44OxylLajPJwefJoWpPu//BHk2LvajH08/rHZ+kF9HdQ4wSFGGWYFqz1b2OkVkcIIrDrOyKFDHl4uLoiwyYwIlazNqHUV1G4bKbneviCUuwSWMAlrgjIY3oufbmTuw9OF8XntY7pI7CNsWhfmeDYIQb3gfHoADoizwiFe/JOKd6ZWEPXbIgLsdt0hmkoT0CdWNXnUDH90Pw4nyZewmMBtT1EIfyg7PVB61n+/2UjNAXzmio2PQFQn4LYvLefvi2aRej6q1ddeLZEx+xaEwNgfajgzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pNNyNTfsOG8Ek82+6Ao7fqrhzwIJ/uNOESZvwBJE5oA=; b=hi47ZyJZivknmFizomT0fHWJB8r10WKH7qsNxJ5oOBriZ5YNO+wNYVOouxRsv5OhEjs6vxJm38WKiTyKzG4QSwJ7fpuOcsBT+An58E8fU3ri9xo1J+VhwElaWrEahaasNaliO6Jgbv3YHFMYGXdhs7UD9YZ4jRlta0rXh6bkOhc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7767.prod.exchangelabs.com (2603:10b6:408:170::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Wed, 15 Jul 2026 18:05:53 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::46eb:64a3:667c:c1a0%3]) with mapi id 15.21.0223.008; Wed, 15 Jul 2026 18:05:52 +0000 From: Yang Shi To: cl@gentwo.org, dennis@kernel.org, tj@kernel.org, urezki@gmail.com, catalin.marinas@arm.com, will@kernel.org, ryan.roberts@arm.com, david@kernel.org, akpm@linux-foundation.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com Cc: yang@os.amperecomputing.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 16/16] arm64: percpu: use local percpu for this_cpu_*() APIs Date: Wed, 15 Jul 2026 11:04:18 -0700 Message-ID: <20260715180455.515692-17-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260715180455.515692-1-yang@os.amperecomputing.com> References: <20260715180455.515692-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CYZPR05CA0010.namprd05.prod.outlook.com (2603:10b6:930:89::18) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7767:EE_ X-MS-Office365-Filtering-Correlation-Id: f2e6bd10-9355-443a-96db-08dee29bb58f X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|1800799024|376014|7416014|11063799006|56012099006|10067099003|22082099003|55112099003|18002099003|6133799003|921020; X-Microsoft-Antispam-Message-Info: L9R+xQx5kWDFPcFDD6qFkPMWZ400S+0lBFNfpLAFsF7JHHEY4S8McnpQB6OQIfyahqySFtgO+dEZ5gl5JE6WKLVg94a2Cjl017cvNk6AeQK+LNAw8suPT3qwgp8xxKv8gpJBhqVW/9rq46cRuUyAxB7I68/7aNBEcOYaG149ZMedA4y44CW2fBt5Zq6EwzDFH+/grMc7fLF9u6fgqmEhaL1cG2Sm53BaZ3DKd9jYzX4gCI38mlGsV809EDza5cIEgSrhjkHPeQH6OHoOp8h1PP8bcVAcinm4G4EK8ndoK4VDB9EIXECV5NS4OjFYO5q4awb8j5uVACAhMi/4egJkIly6RPeNdtFYskK+EqkCohpRNPqtJ304AFTYqmWxRFsxQDoayrKxfK8wGeJzjicsqA8Fdau3kG+vGSZA6dU3Y5XxuDH1tb7sDWutug8xOq2ZnhngDowbRc5qcQYa/qzJcbpiMFKQaVXQXsCh9uJ1taa5WA7E4ClNV7hCI3X7Fzsso7PoaXHBS3PgDxiT0Cq2KLHGGarOBC22EtDHpPUCeVBDemXM9W8GGn58RULPPcHPnytxDRC19ZrOAAVBdUba1WgJnei09G+dOmUlwB5XzH5YGNIidoADkYzWk2J4uEKwY+TWxtxaFsZeg8a8AyqBsTI2t/EXzKUQ7XZxvr9hzxp/7PKqx+zuBY/F0OQsfaHAmiq6GtZJYNaU5V+N/5xnow== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(1800799024)(376014)(7416014)(11063799006)(56012099006)(10067099003)(22082099003)(55112099003)(18002099003)(6133799003)(921020);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?kRBCXe1zfDpwzag8m9k3KlHDh9kQX0L8el2SVhvcuufrw6SYpNVtub2ZxdI1?= =?us-ascii?Q?Y+K7NxUfDi9+68DtCP96bFldxQjr/d7KSIC8mcQtemDm/ybdRiIVH0WBfT4y?= =?us-ascii?Q?rAtZSD+9wh46yyuU8tqga6OlWLCBm7AjLxmZiEA+ieiPsK8ziMse8ThIqL36?= =?us-ascii?Q?5ACJ5661bhQrME7jcrrm/mDIMXTa0CtOO84RZkHe715SqaDdzz9c4aAaOXwb?= =?us-ascii?Q?tQNVnAdB8sMopPybbiVNf42tC8r3A+MsDd+hyXbmO+Phn1tRAZKTkKgTeLbi?= =?us-ascii?Q?sR4ArrcpZUhjwZfRrFfTG4rSRIfdO9sIeyxjOZdXv48sXBuavMZp7z5uS5E1?= =?us-ascii?Q?INPLfOHIIWLjjK+UQJ9g6xfs7bVQ/P+xU2SgK21F1VAP0auOVLJZp1w9Pa2d?= =?us-ascii?Q?iqAafmKziWW+7j6yD9nq8C3Fd5FNYyyXKczztvi5iNFg3gRLRM6msnznEYCI?= =?us-ascii?Q?rQpHuFJN7f3dUpvZ3JM5pojP/Nd2IkbaRCDorZoNhKTMNJcJJCxxcw7f2LFm?= =?us-ascii?Q?yccFjb137mt05Pd4CLW5gV5chRFrB7vcCK/0Q+KGkm6ZDs/hncb7XfuO7+lv?= =?us-ascii?Q?Wq338BgncAgE/28HpqvIqL0WvJ9ceTULu7CGSbznkS6grZH14pDhNcYqHW4k?= =?us-ascii?Q?WMUOFLa5qmbuHAt0DTDKRukoPxzIldMZZ10Mhq3zKVaAJJZk2964w7iNrt3A?= =?us-ascii?Q?SlUgVZ2EmsHk9JbhOtCFcPdFIWzp+9A5u3O+LW+Q/eAR6vzKxEgScvKZUgCk?= =?us-ascii?Q?fifoENid+7FWvoNDhtrfcfXeOnrpgv7+AyMDPayUhlfhNyLoCJAZX6hvREdP?= =?us-ascii?Q?njjgYkX77x48HvSu17HjVIB/ERGmyqXPdD4Wzd1btDCUlwSmtupOxSAmTlXe?= =?us-ascii?Q?7oniM72Ga1T16qrEPeTAEt+lqEZ4h2YBHZkRO/UtM6cQNPHeeo0JI/iI0JkJ?= =?us-ascii?Q?uuTxR7MbLeyCivxol3Nh0c0ywFjyC6BkZ1ApdVhOLFyliariaXe+V8V1UpjY?= =?us-ascii?Q?bqAR72LyLClgF+Ab6DMKKPhUi+YkMWEnkLIQv05ll59ebR5hYUMtM4Xsc+fq?= =?us-ascii?Q?ZBNCCkZpxUW+4VIDLO/rbshbpa8v4RBJlowdGWksXFCZwTyx6fIM2dE/Btia?= =?us-ascii?Q?FOzCh7e3jlgj74B1q5R3zR4lwr96JIplDzRwfjLtjH8as14S+N5aZyq4964h?= =?us-ascii?Q?t4IWhN0lTO0UtSC1edyJ2XAFJnGAcgSp//HoDu4pYOHkUAJPLLHS69mhEASi?= =?us-ascii?Q?ahFO27A2wP3RZ0u2DfmufgSaE5OzmqxGvPAYYymv1huercG62nj1QUQdD653?= =?us-ascii?Q?Y/WRHr/AKwRE8bV8u1fbnLqQRBBjBYx8SvdWgo2LcPau+sAdH8xhQEo9BEqz?= =?us-ascii?Q?vI6MWwhtK2cGdHpn6bO45RBlw+CMmslJGhAAyz3FGK1OtxWpdtHuNVorhroq?= =?us-ascii?Q?DJFMzQcxMRwNPKohQFyEzeTwCV4OlGeWCsLCbU61URK7RAKY8c9WndEtoYL7?= =?us-ascii?Q?7GR7xpAoGkKsv+zXNhHI4w2TRZ9OWrkA5eWBtyMVjRs4YJ4LE0Uud6LVbpgt?= =?us-ascii?Q?tQLfo469WoDsAyzzm1biuV6ZW5No6p+eSnKQHAiCs6k0ZfRUZYmY01GX/oae?= =?us-ascii?Q?fOu2VAGGAiVc+71X8KOPoxq/MumXmBeVfCWYTIckcRdLZm1F8jqkWtXtT3mI?= =?us-ascii?Q?ykm2dKsan3aQxDSVTPNOKdWZBIDgKcZmv9G5AjgtpOeJPIklRhIlYeUDH1vf?= =?us-ascii?Q?bqTxDKnnRsqTiMZwocuzgTwnlOPZVoI=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: f2e6bd10-9355-443a-96db-08dee29bb58f X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2026 18:05:52.8239 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ti8PNjmsQafulMW1Ycr4B5cCUE51/odRdr7LQ2HQdOwkoeCjdE5I9WXRKuGfkYsVAD98YgOX4eBDXWIsQOv46LbRQFzDSENNPIy2uF5bMIM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7767 Content-Type: text/plain; charset="utf-8" Use local percpu address for this_cpu_*() APIs. Because the percpu variable is mapped to the same virtual address, their address can be calculated by using __per_cpu_local_off which has same value for all CPUs. So preempt_disable/preempt_enable is not needed anymore. This optimization can improve the performance for this_cpu_*() operations. Kernel build test on AmpereOne (160 cores) with default Fedora kernel config in a memcg roughly showed 13% - 15% sys time improvement. Signed-off-by: Yang Shi --- arch/arm64/include/asm/percpu.h | 37 ++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percp= u.h index b57b2bb00967..03d149f9177c 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -142,6 +142,27 @@ PERCPU_RET_OP(add, add, ldadd) #undef PERCPU_OP #undef PERCPU_RET_OP =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +extern unsigned long __per_cpu_local_off; + +#define local_cpu_ptr(ptr) \ +({ \ + __verify_pcpu_ptr(ptr); \ + SHIFT_PERCPU_PTR(ptr, __per_cpu_local_off); \ +}) + +#define _pcp_protect(op, pcp, ...) \ +({ \ + op(local_cpu_ptr(&(pcp)), __VA_ARGS__); \ +}) + +#define _pcp_protect_return(op, pcp, args...) \ +({ \ + typeof(pcp) __retval; \ + __retval =3D (typeof(pcp))op(local_cpu_ptr(&(pcp)), ##args); \ + __retval; \ +}) +#else /* * It would be nice to avoid the conditional call into the scheduler when * re-enabling preemption for preemptible kernels, but doing that in a way @@ -152,7 +173,6 @@ PERCPU_RET_OP(add, add, ldadd) * preemption point when TIF_NEED_RESCHED gets set while preemption is * disabled. */ - #define _pcp_protect(op, pcp, ...) \ ({ \ preempt_disable_notrace(); \ @@ -168,6 +188,7 @@ PERCPU_RET_OP(add, add, ldadd) preempt_enable_notrace(); \ __retval; \ }) +#endif =20 #define this_cpu_read_1(pcp) \ _pcp_protect_return(__percpu_read_8, pcp) @@ -243,6 +264,19 @@ PERCPU_RET_OP(add, add, ldadd) =20 #define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) =20 +#ifdef CONFIG_HAVE_LOCAL_PER_CPU_MAP +#define this_cpu_cmpxchg128(pcp, o, n) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + u128 old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__ =3D o; \ + new__ =3D n; \ + ptr__ =3D local_cpu_ptr(&(pcp)); \ + ret__ =3D cmpxchg128_local((void *)ptr__, old__, new__); \ + ret__; \ +}) +#else #define this_cpu_cmpxchg128(pcp, o, n) \ ({ \ typedef typeof(pcp) pcp_op_T__; \ @@ -256,6 +290,7 @@ PERCPU_RET_OP(add, add, ldadd) preempt_enable_notrace(); \ ret__; \ }) +#endif =20 #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); --=20 2.47.0