From nobody Mon Oct 6 04:59:35 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2108.outbound.protection.outlook.com [40.107.236.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52EE32367CD for ; Thu, 24 Jul 2025 22:12:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.108 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395177; cv=fail; b=R+b3IcI67yRC+4FNGs/YogmuyfF+ubDiW0lo3Z6yjzf+026iABiylXZdkHvz0kX1ab5+LMp/afcW3DaWr4sxqyGk09KDKdIqXNL6DK7Xu5WOUbS/U/7YZHJuJVRQcItWC+l68O7kh73Q45dAATob1jGgFl5bpsZiP4/ZCjbBd2Q= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395177; c=relaxed/simple; bh=XBhjm7MP9kvombmbBPcWdmFbFp/AAYNiK64YnYjMcVc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=PAGXAF1dmlP1yFfQT7OkyI2sLzfVKgUXaZt/hnkxYTYQRgztoBhmQ8a8rXRwZz0Td11U67FtnQJU6sDix8qgLjy9CW6JDMzuKkmZWjpPu7wLKMsxM2gTa28Cqugaf3ktf1EV2r+lly+g8VnF/XYZy+OU1ff4Wq2QaYpEO5nMszg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=Q3P5+xW4; arc=fail smtp.client-ip=40.107.236.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="Q3P5+xW4" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wGHbiWAnItsLqax/ky8KH0p7ZMC3G0U5Za54WS4fWLD68AYqyDUj+8O46h/qQx+2MFLt6bzC9TmMlUClOptO9nEMKk5K1Cr6BoWRayMiiL0bU5UScUIN2iL4mkh1vVw1Hh5uVEA5YADfITY+Yoz0TwAxW01OYdPn0lK+6f8Le4K6U80yb45VixrtKCyouoPsFXcLn3dl4/h6eGMJvGksKKfYqcdc5QflfxfEZe5Ph3dyBXcAZX205kbjspOT8sQptvZTYUDzDMu9Qj4erS+7+hvy9e+fkmEwz3sGsxrAce8Um2+WhUPf77L9lsapI2lX9BYC52K1glR1aZOvHmK0fA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YA6ktP8Ndf8PLhiEiY8OWVhBinV4MfC1mPsiughW0yU=; b=Pvi0Fz4rBrvobPSex1rLry3u27f8TzaosKFJtBl4t0ZbsfmC+wSBm29w3mimCfKqLiS0N+kJZXLqnvqyv8IRg7GlOIUXdjDkoe2TsS7Tpf0VANYNtob1Zxwi8wIjJvVM0cl/+nQ/V8302f35tSgEsFjZQ5T4vQAWDDklxHpv3pC7T5l9kz55d1rtZTK8MPCKeZDxemsjbkdRc3xEvavy9WBPh+BQhyDpCyjthl/q3T6zaQFQMuooXGObThZpYfly4ae1/VzqUXvh+feJmipX831kPDGrPMlqVRBwZkwG0tyXV/veMuJsJM8dNF1awrgnVl1V3m3G/X8iUt5pNyyYbA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YA6ktP8Ndf8PLhiEiY8OWVhBinV4MfC1mPsiughW0yU=; b=Q3P5+xW4DuvWzQxAg750+PaMDdHhuZgwG70vK3eTGas0edU5UtFOCij17ceWwX+huBlTVjKDSESw/sD2ZBObsN4siptQs8AdNA4dPPwck8bh5EQPEzd3m/NXKHS91ijFaCaKgepVKRasU5fk8+3oSQ1n2xLKtW0O0sBAgSubQPE= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SN7PR01MB8089.prod.exchangelabs.com (2603:10b6:806:32f::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8964.21; Thu, 24 Jul 2025 22:12:50 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8964.019; Thu, 24 Jul 2025 22:12:50 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, akpm@linux-foundation.org, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] arm64: Enable permission change on arm64 kernel block mappings Date: Thu, 24 Jul 2025 15:11:03 -0700 Message-ID: <20250724221216.1998696-2-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.50.0 In-Reply-To: <20250724221216.1998696-1-yang@os.amperecomputing.com> References: <20250724221216.1998696-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR15CA0236.namprd15.prod.outlook.com (2603:10b6:930:66::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SN7PR01MB8089:EE_ X-MS-Office365-Filtering-Correlation-Id: 6e7b9dcf-e068-48d7-7bf7-08ddcaff3a90 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|52116014|1800799024|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?AfeNXkfmAPAdfqWySOyIKlaCBsAZKftjx+f5yHhoyA7zT227BihbyaBQaSGl?= =?us-ascii?Q?w2uxkmaGBhSXG3GHspmloMiLVaU3laIvhGQj4w6Rp7Vsju8qs39Riqpt/CZd?= =?us-ascii?Q?7LPWLmhqv8cHup6FSbWdTW9iXTGOfIPH83M4s0P6/doPulok34Ww+SXnJnE3?= =?us-ascii?Q?rqgJrBGvG5N0j05eLZ/10/zqmxKrNZVDwyhDN4RKdBZWAlx6rE5wVolKmW++?= =?us-ascii?Q?GaLV+MRYWCLAWwDr0H/U8VdoSVEDMT+yKHk7TDJVTLyVMHYE8WH0gnP72jgq?= =?us-ascii?Q?xl6fr9PCQXo0oywK6PmKH306rwQiDoXkIynBFiKx/x1JB0JksnZvvKCLMqIL?= =?us-ascii?Q?19SVBehSxc5osL9mRy4QQlbCVCqemxHdTfuQuCLszNbIPXbYppmcgAkqgbCl?= =?us-ascii?Q?NPrJRO9s8umDIzRzM0pDSq8M3lLukqhnK7VjXr8WkBsUxa97wQ/Z0bTTU06c?= =?us-ascii?Q?cXuChVWqrsLpOiE37k2QQuJ692rvhaDBt5py7FtnUsVXmlDwoIf40CD3jI1h?= =?us-ascii?Q?GAdbUm6Tvb2oSdOoG5qub1KIx3Q2C74KxgUfEm4n5+1UojfOL8diS5+Hp6es?= =?us-ascii?Q?6xD1LgyjuY3Ur7hXZUOqY3UWZEz7D4/VZCQfjHh9b1dg80d6bOvJc8k3AX7r?= =?us-ascii?Q?4pW7YMsglRPPFHpMAakuxNgSx8HJ85XBU0jTqD4N5on6nk62e6jMPpWmYZGI?= =?us-ascii?Q?8jOAE/GPueX5fFfNRZGVw1mh5Ob/qt/urIHXluiErBNM/pi9+rsDqmquoCD3?= =?us-ascii?Q?Ysbhe4/jgUZrLhw2TiOR0+iBaXaI4f4jH1MQcNbSyknyFZP7odsx1s31KXLa?= =?us-ascii?Q?LCgvdci+doVa/cqnmGPsVuCTrsYOuskc0ZLaeWcUNozGfy/r8NZPILuMbyCF?= =?us-ascii?Q?OiOAvIudBzVGKnpARDE5LS1o8ZoDW4IbDLhbvI1/W9jO9dWmll3R+aRW9b73?= =?us-ascii?Q?cjW1NtV5r0gQjAcbPYDXeOAJGL5RJ+eZkhhnVQB/YO3rIqsmd0XIVjpmrubS?= =?us-ascii?Q?g1VPdaMT7U/2zimcyLd3ctQUbffsbZmWdQ2Aprkiwhygn5QEP2m1EEPXx0/z?= =?us-ascii?Q?n/xwOhY4Pt1OAb03XhGzugPWebd0Yhi/bsSSq89wg4pURLJh4g0kTb1TMjxy?= =?us-ascii?Q?DrlJZtJ712ajmCeDse3z8PFJZjoUaxmsU6iBjtX+EDTVvA1ks+hFe8tkQpWY?= =?us-ascii?Q?IBVNshfpFNmv6cpUT5Um5CcFJHQXpVTt21m4Z6yKpO9mrv1FtsO/8WudXa+k?= =?us-ascii?Q?k5UhlI/LhEu/jQJ+ARuMIWsvnJHR3O3p8HoOu1ZZLMfAeR3lCRSxImwIzv0A?= =?us-ascii?Q?3nBSzC2hgCHAu2WSGWrVkLDScavdY6kyfgdefDfUQ/wcuGlv5EboY6RqEnxQ?= =?us-ascii?Q?27lOkRDgsrQsCmc2XzVYJ3HoPBkhiqiySt8er5s8LpZ2jQlQ3BxVXUC5UFQH?= =?us-ascii?Q?oX7GtWCkg47QyVnUqKXM90jozZSXeTP2uqo5xVViUQl+qws4cI+Cq94jKhmi?= =?us-ascii?Q?4YdgWc6iMsoRt28=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(52116014)(1800799024)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NSQZv6LzN+mZB2i7ed8BbF4W8jFfOJe6/aT6VoYQwM7LGrAfByuR37ly3keN?= =?us-ascii?Q?sZBo0iheEFJghR/rClGwr/EoGnCKBTFEbN8Q1Q0zY2/l57Ad+VuP93e+ZM7O?= =?us-ascii?Q?E8n/e/XCk1NVocUSLTKRH5CagE2J7Ww23f+84G5kVaLVVHVaJlLtvIrqi42Y?= =?us-ascii?Q?uDD0NwGccdsSnhCfZtXXjKLBVaHR51ift2C3mqD7aptJSvEON9/FAnuNlKoS?= =?us-ascii?Q?dgb0zwq5Iaca1pSjOG55cNsSiqWHvZnZ63mDHtJNZzrcrgyhLn3UAjfNhPFv?= =?us-ascii?Q?k89r/QnJ2qdjAr9C47PJGeHN1o7/0sSC32efh5vEsfSdU+wS5bmzyfbvKAkl?= =?us-ascii?Q?1fsgSiB1wAQM9/95rDNt4YXQPqyq86pMCLvSx/kiIRkkt/lAaMQjjL2bnc7L?= =?us-ascii?Q?pdFHePFxNDqD3Rbifk6OT00oXPZvAzCXGWbFPWxFQETbODeJbCuhP2ezg5Sm?= =?us-ascii?Q?WXAsoExyV6pN7WYIG3vE51NYcHgXs76xyOwf0jwULbGPThlTgojgUewvZ8WI?= =?us-ascii?Q?1ZmIjZ9ozMW+Joy8wEJG0OiqHCoFAO4YHRoLrVtProcFJ4ZTb6d3efscdNak?= =?us-ascii?Q?lf0eMb6ItKhl/cquo3xKARbqtZBHYSIepLJwi2nHnWyqq1ROq17GCSTLB5P4?= =?us-ascii?Q?HcjCcolc/2kHsbYONlGKpyh7vV8K+gjUIxjDgR1QxQ8/cKwrIxKqhRNQLMx8?= =?us-ascii?Q?MOaCvX1wQX7bHtnvMNGGLzO4r5jQZ5OMBBKafO56cg96AuDT1iLYTGLmZ7FU?= =?us-ascii?Q?7Yum/zxIdzGv9WZRHCuMzH293OoeXW30hwXAKhpMCLMtZE+EMKjBtlDwW6aw?= =?us-ascii?Q?DUw5vuzLDXTTLxSXZPJKfmFsDt3C4Cf0i6hAeqpXf9Oaq2NNy9TU/QHSG20T?= =?us-ascii?Q?67kvOTGGJHszdrBDmrxhfzlp0tJs4HlRnxb/Hphb8rckg4EUtoGHPJu6Jb57?= =?us-ascii?Q?HUqUg+nnbSNtdHuQ9epqRUrXyZXIhSW4QHB7v4aX14Y1AY+KYPbbOFz5uFj3?= =?us-ascii?Q?wvH0l08PMJC9EAkEu22PZQbC/kLwrW+saBY471HWU+BQUmVa+ZaJaVOG5T7d?= =?us-ascii?Q?4XBpOemF8lMrrWYtLXeryTzoEV2md4WXgiveB8svLyifKnu3YJcnetSlsDeZ?= =?us-ascii?Q?0EgOGYgGLFncwwWMz7dRy0QsA4NNBVBNBJ6YCvmO0I+hbGipkzxmvKKOqVKm?= =?us-ascii?Q?rnNsZrW6Iewqya0W549k/n7hgdSnztKCyi/a+vnmLd+rlb6I+TgRwGHqgPPd?= =?us-ascii?Q?kT/sRJMC70NjMfM1LTFzJWuzI8AtrDU/LB3ln2VFgmCRNX6PdqJg61sPQJ9B?= =?us-ascii?Q?mqPz4DolrpmpanCgPeanUxGn7jiWjf3VCBVjfXORGhghdDR9zYOxoZqxUjiW?= =?us-ascii?Q?UVvNhs/d1u0UJN1qVsNWylSXMRedT4hE+Z5j5NcDSvZ0Ci6pqItcQe3GYRks?= =?us-ascii?Q?tEivo96vAzah2pnRPK9vygVbrL7nT18m/wytWnZEN7bqzpet5bgDw3Wt2oKB?= =?us-ascii?Q?hZViCZcFmGt48xYvi904eShddNx4cJ4gCWhA1fKi1kesTSqPLwSvubB+D4mY?= =?us-ascii?Q?DsxnRnHq+Csz7KN3apPRr3UqkZFqO3Sx3QOB4fJ8Zt2o8YjoRYqmgYj8HPg1?= =?us-ascii?Q?60OXU+wrkwN3tXuYRDKgvzE=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6e7b9dcf-e068-48d7-7bf7-08ddcaff3a90 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2025 22:12:50.6841 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PjRgVLT3mZsc/yi3bKgLJcPBUL3vHBU94lByOPJSBSombcwgD35dWWnUm8nmLlNyCLj5ecYYcCjS5hlqMYyOYLCU8tltUD32Picwp0+csp8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR01MB8089 Content-Type: text/plain; charset="utf-8" From: Dev Jain This patch paves the path to enable huge mappings in vmalloc space and linear map space by default on arm64. For this we must ensure that we can handle any permission games on the kernel (init_mm) pagetable. Currently, __change_memory_common() uses apply_to_page_range() which does not support changing permissions for block mappings. We attempt to move away from this by using the pagewalk API, similar to what riscv does right now; however, it is the responsibility of the caller to ensure that we do not pass a range overlapping a partial block mapping or cont mapping; in such a case, the system must be able to support range splitting. This patch is tied with Yang Shi's attempt [1] at using huge mappings in the linear mapping in case the system supports BBML2, in which case we will be able to split the linear mapping if needed without break-before-make. Thus, Yang's series, IIUC, will be one such user of my patch; suppose we are changing permissions on a range of the linear map backed by PMD-hugepages, then the sequence of operations should look like the following: split_range(start) split_range(end); __change_memory_common(start, end); However, this patch can be used independently of Yang's; since currently permission games are being played only on pte mappings (due to apply_to_page_range not supporting otherwise), this patch provides the mechanism for enabling huge mappings for various kernel mappings like linear map and vmalloc. --------------------- Implementation --------------------- arm64 currently changes permissions on vmalloc objects locklessly, via apply_to_page_range, whose limitation is to deny changing permissions for block mappings. Therefore, we move away to use the generic pagewalk API, thus paving the path for enabling huge mappings by default on kernel space mappings, thus leading to more efficient TLB usage. However, the API currently enforces the init_mm.mmap_lock to be held. To avoid the unnecessary bottleneck of the mmap_lock for our usecase, this patch extends this generic API to be used locklessly, so as to retain the existing behaviour for changing permissions. Apart from this reason, it is noted at [2] that KFENCE can manipulate kernel pgtable entries during softirqs. It does this by calling set_memory_valid() -> __change_memory_com= mon(). This being a non-sleepable context, we cannot take the init_mm mmap lock. Add comments to highlight the conditions under which we can use the lockless variant - no underlying VMA, and the user having exclusive control over the range, thus guaranteeing no concurrent access. We require that the start and end of a given range do not partially overlap block mappings, or cont mappings. Return -EINVAL in case a partial block mapping is detected in any of the PGD/P4D/PUD/PMD levels; add a corresponding comment in update_range_prot() to warn that eliminating such a condition is the responsibility of the caller. Note that, the pte level callback may change permissions for a whole contpte block, and that will be done one pte at a time, as opposed to an atomic operation for the block mappings. This is fine as any access will decode either the old or the new permission until the TLBI. apply_to_page_range() currently performs all pte level callbacks while in lazy mmu mode. Since arm64 can optimize performance by batching barriers when modifying kernel pgtables in lazy mmu mode, we would like to continue to benefit from this optimisation. Unfortunately walk_kernel_page_table_ran= ge() does not use lazy mmu mode. However, since the pagewalk framework is not allocating any memory, we can safely bracket the whole operation inside lazy mmu mode ourselves. Therefore, wrap the call to walk_kernel_page_table_range() with the lazy MMU helpers. [1] https://lore.kernel.org/all/20250304222018.615808-1-yang@os.amperecompu= ting.com/ [2] https://lore.kernel.org/linux-arm-kernel/89d0ad18-4772-4d8f-ae8a-7c48d2= 6a927e@arm.com/ Signed-off-by: Dev Jain --- arch/arm64/mm/pageattr.c | 155 +++++++++++++++++++++++++++++++-------- include/linux/pagewalk.h | 3 + mm/pagewalk.c | 24 ++++++ 3 files changed, 150 insertions(+), 32 deletions(-) diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 04d4a8f676db..c6a85000fa0e 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -8,6 +8,7 @@ #include #include #include +#include =20 #include #include @@ -20,6 +21,99 @@ struct page_change_data { pgprot_t clear_mask; }; =20 +static ptdesc_t set_pageattr_masks(ptdesc_t val, struct mm_walk *walk) +{ + struct page_change_data *masks =3D walk->private; + + val &=3D ~(pgprot_val(masks->clear_mask)); + val |=3D (pgprot_val(masks->set_mask)); + + return val; +} + +static int pageattr_pgd_entry(pgd_t *pgd, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + pgd_t val =3D pgdp_get(pgd); + + if (pgd_leaf(val)) { + if (WARN_ON_ONCE((next - addr) !=3D PGDIR_SIZE)) + return -EINVAL; + val =3D __pgd(set_pageattr_masks(pgd_val(val), walk)); + set_pgd(pgd, val); + walk->action =3D ACTION_CONTINUE; + } + + return 0; +} + +static int pageattr_p4d_entry(p4d_t *p4d, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + p4d_t val =3D p4dp_get(p4d); + + if (p4d_leaf(val)) { + if (WARN_ON_ONCE((next - addr) !=3D P4D_SIZE)) + return -EINVAL; + val =3D __p4d(set_pageattr_masks(p4d_val(val), walk)); + set_p4d(p4d, val); + walk->action =3D ACTION_CONTINUE; + } + + return 0; +} + +static int pageattr_pud_entry(pud_t *pud, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + pud_t val =3D pudp_get(pud); + + if (pud_leaf(val)) { + if (WARN_ON_ONCE((next - addr) !=3D PUD_SIZE)) + return -EINVAL; + val =3D __pud(set_pageattr_masks(pud_val(val), walk)); + set_pud(pud, val); + walk->action =3D ACTION_CONTINUE; + } + + return 0; +} + +static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + pmd_t val =3D pmdp_get(pmd); + + if (pmd_leaf(val)) { + if (WARN_ON_ONCE((next - addr) !=3D PMD_SIZE)) + return -EINVAL; + val =3D __pmd(set_pageattr_masks(pmd_val(val), walk)); + set_pmd(pmd, val); + walk->action =3D ACTION_CONTINUE; + } + + return 0; +} + +static int pageattr_pte_entry(pte_t *pte, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + pte_t val =3D __ptep_get(pte); + + val =3D __pte(set_pageattr_masks(pte_val(val), walk)); + __set_pte(pte, val); + + return 0; +} + +static const struct mm_walk_ops pageattr_ops =3D { + .pgd_entry =3D pageattr_pgd_entry, + .p4d_entry =3D pageattr_p4d_entry, + .pud_entry =3D pageattr_pud_entry, + .pmd_entry =3D pageattr_pmd_entry, + .pte_entry =3D pageattr_pte_entry, +}; + bool rodata_full __ro_after_init =3D IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT= _ENABLED); =20 bool can_set_direct_map(void) @@ -37,33 +131,35 @@ bool can_set_direct_map(void) arm64_kfence_can_set_direct_map() || is_realm_world(); } =20 -static int change_page_range(pte_t *ptep, unsigned long addr, void *data) +static int update_range_prot(unsigned long start, unsigned long size, + pgprot_t set_mask, pgprot_t clear_mask) { - struct page_change_data *cdata =3D data; - pte_t pte =3D __ptep_get(ptep); + struct page_change_data data; + int ret; =20 - pte =3D clear_pte_bit(pte, cdata->clear_mask); - pte =3D set_pte_bit(pte, cdata->set_mask); + data.set_mask =3D set_mask; + data.clear_mask =3D clear_mask; =20 - __set_pte(ptep, pte); - return 0; + arch_enter_lazy_mmu_mode(); + + /* + * The caller must ensure that the range we are operating on does not + * partially overlap a block mapping, or a cont mapping. Any such case + * must be eliminated by splitting the mapping. + */ + ret =3D walk_kernel_page_table_range_lockless(start, start + size, + &pageattr_ops, &data); + arch_leave_lazy_mmu_mode(); + + return ret; } =20 -/* - * This function assumes that the range is mapped with PAGE_SIZE pages. - */ static int __change_memory_common(unsigned long start, unsigned long size, - pgprot_t set_mask, pgprot_t clear_mask) + pgprot_t set_mask, pgprot_t clear_mask) { - struct page_change_data data; int ret; =20 - data.set_mask =3D set_mask; - data.clear_mask =3D clear_mask; - - ret =3D apply_to_page_range(&init_mm, start, size, change_page_range, - &data); - + ret =3D update_range_prot(start, size, set_mask, clear_mask); /* * If the memory is being made valid without changing any other bits * then a TLBI isn't required as a non-valid entry cannot be cached in @@ -71,6 +167,7 @@ static int __change_memory_common(unsigned long start, u= nsigned long size, */ if (pgprot_val(set_mask) !=3D PTE_VALID || pgprot_val(clear_mask)) flush_tlb_kernel_range(start, start + size); + return ret; } =20 @@ -174,32 +271,26 @@ int set_memory_valid(unsigned long addr, int numpages= , int enable) =20 int set_direct_map_invalid_noflush(struct page *page) { - struct page_change_data data =3D { - .set_mask =3D __pgprot(0), - .clear_mask =3D __pgprot(PTE_VALID), - }; + pgprot_t clear_mask =3D __pgprot(PTE_VALID); + pgprot_t set_mask =3D __pgprot(0); =20 if (!can_set_direct_map()) return 0; =20 - return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + return update_range_prot((unsigned long)page_address(page), + PAGE_SIZE, set_mask, clear_mask); } =20 int set_direct_map_default_noflush(struct page *page) { - struct page_change_data data =3D { - .set_mask =3D __pgprot(PTE_VALID | PTE_WRITE), - .clear_mask =3D __pgprot(PTE_RDONLY), - }; + pgprot_t set_mask =3D __pgprot(PTE_VALID | PTE_WRITE); + pgprot_t clear_mask =3D __pgprot(PTE_RDONLY); =20 if (!can_set_direct_map()) return 0; =20 - return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + return update_range_prot((unsigned long)page_address(page), + PAGE_SIZE, set_mask, clear_mask); } =20 static int __set_memory_enc_dec(unsigned long addr, diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index 682472c15495..8212e8f2d2d5 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -134,6 +134,9 @@ int walk_page_range(struct mm_struct *mm, unsigned long= start, int walk_kernel_page_table_range(unsigned long start, unsigned long end, const struct mm_walk_ops *ops, pgd_t *pgd, void *private); +int walk_kernel_page_table_range_lockless(unsigned long start, + unsigned long end, const struct mm_walk_ops *ops, + void *private); int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, void *private); diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 648038247a8d..18a675ab87cf 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -633,6 +633,30 @@ int walk_kernel_page_table_range(unsigned long start, = unsigned long end, return walk_pgd_range(start, end, &walk); } =20 +/* + * Use this function to walk the kernel page tables locklessly. It should = be + * guaranteed that the caller has exclusive access over the range they are + * operating on - that there should be no concurrent access, for example, + * changing permissions for vmalloc objects. + */ +int walk_kernel_page_table_range_lockless(unsigned long start, unsigned lo= ng end, + const struct mm_walk_ops *ops, void *private) +{ + struct mm_walk walk =3D { + .ops =3D ops, + .mm =3D &init_mm, + .private =3D private, + .no_vma =3D true + }; + + if (start >=3D end) + return -EINVAL; + if (!check_ops_valid(ops)) + return -EINVAL; + + return walk_pgd_range(start, end, &walk); +} + /** * walk_page_range_debug - walk a range of pagetables not backed by a vma * @mm: mm_struct representing the target process of page table walk --=20 2.50.0 From nobody Mon Oct 6 04:59:35 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2108.outbound.protection.outlook.com [40.107.236.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6842C239E6B for ; Thu, 24 Jul 2025 22:12:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.108 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395178; cv=fail; b=VPQo2xynb9hJ1037C3dduKwcopDz8Wvl3Nk7qaG4au/iN1NR2/5100zaq96lFNBsrd5aYZJML7qMbBqBugg7esaq5NXStJIa2YgnAJdtiHgvtUmZChWsOdHB8MTCyJsCSxmRXi8ZpI2qPQ9wN1dRnXe5xnuEtdWTHgYy/gLqSzs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395178; c=relaxed/simple; bh=IJK+Ir0tA81uQBjcbA8kZcA3NomCl5bI1FTIjPIlJVU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=UaaDZPIMz/7VCRGP0NHCjB7pATIj7pBkX9vWDChRONglqrimh4CSiBCtHMpIwX0JtALPK/4HCgKpTROqJHL2Lucgcc6pjX7dogFSUfWH5XTTJ5IsFDttpLbjExa6L5h5Zme4c1+mTpCiBirLm8N3YfurN9d7Tt3CT45yKkuMj44= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=YFEekSaF; arc=fail smtp.client-ip=40.107.236.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="YFEekSaF" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=eiUGM/dM4ygxgy9hxveRZa1DPIEuxsj8E2K4jIMiqmdOz4DpKyxbrCsWZzja+bb2y82Z8HV41PYyFzbI8U9p+m6/OwUXVfAChWD/Y8QBMHbADyxGu3N8msSetofFv83NFnAmdENBRsa1+lIrf8IgC/MW3FNH97KixawQPqI41smXuv0Y7hmSUjLx5nuHzETNRjhlFPpyOAbKhwNpCk1AkWVDyTb/+iWzVXy4JwWifO6cu6/LqfJ5YZqD0jl6KdOR1VzAroVNlwgBp98u3Xgoegt8ZbQBom09Vx+CNRcg19f2cROZvr8ioKtjpR8fe45a9qR3pkJ16D5qxlVPoYsg/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JBEyo8bg4ozRRld/qGFPTsbvmKm10Quqq/Wi0K/VYaY=; b=df5cRSuonU7nyToH1/tLoustgNnA39r3N3+GlO5ptcX22zHhmm38liwlZq3uWv1RZGTFqU8XaK3cC3EAtRcde83q6xMiFnzfnzfWZ6ZK6/lliYESzihJcvFSdG16B2hR0f2MmKoKNUoJpnADlbqzJx1I9KWPt4jTDLP5vhRwzsvWKzMMxrePyafpOA2qUUGOkR9H35VR7XdX98qYe50f7tfGJgxyVh/TEEME1VnFm8rx2cB1h7PyFKHwO7S/7fucXlR3FIAwmu4Vij07mon9U/Gu+hE2ZnQojV7YOU5UoJd4f25Yx9Q97YnLAUp84HX0cIjYKbJXRj4/QVNKGoU01g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JBEyo8bg4ozRRld/qGFPTsbvmKm10Quqq/Wi0K/VYaY=; b=YFEekSaFW4iW+yAvZQvOwpwdX7+izlBx320dFNYQx75P4fzna/AcBBnv3NS26jVi3zUGASjMdCi6AmJokJFYFoGXfXtLqGJkPNgcDA/QIIoEgWPLScvPkfuQ61c4+2iFqccqMjF/h2LPmNF3KKK5uUms4YU634UZvNYwVRUzdeE= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SN7PR01MB8089.prod.exchangelabs.com (2603:10b6:806:32f::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8964.21; Thu, 24 Jul 2025 22:12:51 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8964.019; Thu, 24 Jul 2025 22:12:51 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, akpm@linux-foundation.org, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] arm64: cpufeature: add AmpereOne to BBML2 allow list Date: Thu, 24 Jul 2025 15:11:04 -0700 Message-ID: <20250724221216.1998696-3-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.50.0 In-Reply-To: <20250724221216.1998696-1-yang@os.amperecomputing.com> References: <20250724221216.1998696-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR15CA0236.namprd15.prod.outlook.com (2603:10b6:930:66::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SN7PR01MB8089:EE_ X-MS-Office365-Filtering-Correlation-Id: 95e598df-dae9-411b-69ef-08ddcaff3b3e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|52116014|1800799024|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?FGOcDkrRHKn7ny9zRn+37g/RjXojS67zdsmYVM0Hfc1A0GfXZD+ljs37kSu5?= =?us-ascii?Q?cP9JT0EmE+zz12veS9Zz+A9WrG/aOYnrY/5cOsP8PM4x5814eiSJBVcBUXp9?= =?us-ascii?Q?VN/pmhAcOjBAV/UWX/1AASXv+cc5jguaveySUZ0fJIqhOqmY1c3H2LXZRoQS?= =?us-ascii?Q?iH6J0Y91iIb1JfrfhycgvNhkkIojRyQw4/hqPzggal0z2oZMNlrXueLt0fCT?= =?us-ascii?Q?zkjUfr46fjAaF0mrOM2SAdlywjbH/bZ3mbDGu54c8JSRTkpFUrDARrgy1oaf?= =?us-ascii?Q?P7tidKdnp3xM1zjFKolYDMshkPMXsV35DQDHJUtJPCvSd03MxdWOM6CjziRo?= =?us-ascii?Q?iXizgIBj8TGJObC65IjwNcdKjrRx+3U5QQd9QilRgzRE/okbc6Pn/8vGMzRz?= =?us-ascii?Q?Pz+GayO5VIYaxvrm+NYsjLGwp/E6mQ69uphFXVyFvcMG37JHJeP9dzQnRcIu?= =?us-ascii?Q?/nrWOU/UyCaJbZFd30Bv72iPdKvtIMJPYnzStC7yOjBV9X0kjqsodEVanW8d?= =?us-ascii?Q?X8Otcvqvi1qTqJ4OmOZW4pIbDvE/GhGuojaj58GQzwN3lvcYOz25HtYPFc2c?= =?us-ascii?Q?7gYmVmXp5Vz/jYhRzBzG9yI6z5lJp3MRbiKRkQqNpocQKnyU/TrtsVlYMz1v?= =?us-ascii?Q?2ZfDKFBJIC9rEYvzdUMYzXqk6vw/bPDGl7zXWylnLXYp5eqHJD/M/A+e3cjF?= =?us-ascii?Q?5JrfrvDSZrc3T6cU5aHN25zQPns4nygo/wh87V0J7miUgH8XxvFST2ZZcJIg?= =?us-ascii?Q?HfRMxKnZVc6OFUQKe1XWQmGU5uK6KPKFFrFN72MDVy4uCZM34SFPi3qB0iqL?= =?us-ascii?Q?Uw78uKTkVXIRoNApw2Xn5qacbdoXp5VSzy5wfaNb1d1ZysQ8E56U4JBVTnsv?= =?us-ascii?Q?EmAxL1recGRB4JLcDFGzG+yOj8kmXnr+I/O6k7CETayfbCNZ1aEXCKWk+zBJ?= =?us-ascii?Q?Dh8ugPZITC/SHSKcD9BbEYGSkNu5za8u5ApSMI7b9GPNaLql6Fy2aKcGwWfN?= =?us-ascii?Q?CNtR5AihZEOLtUDoKFDxzEvPPJzt1ETRhYfkIw+6LUSTeF6D4nxpmuTqMz2u?= =?us-ascii?Q?SUaIeq4mnlvgJAhsEAU1rTfFeeMV+hr7XI5aCpaFnYIn3gWFky4pZ9d1q72f?= =?us-ascii?Q?CCe5pw5kEsbBmQhcS0Omf/GNIL8QC4pdOCaAXArgSoerNvUW3JKR9MkzQUlD?= =?us-ascii?Q?TOHnSUTSd/HGL+0jBE/kJv6U1RnRIBOW4UUlFH5H3ZYvezKQM8kA9Fh1uMBx?= =?us-ascii?Q?PVaBQwZmjButppMhi+0TJGz+PR5Q/GgxcFEeBXe4An2XgSAs+WAHirA5PMZ3?= =?us-ascii?Q?LgJ35CZBNVzUAVdVLEk2YGn42Hvhm5jIl5nmMTOYk6CQbDomTa9YIwiwNclc?= =?us-ascii?Q?UxRIXIoavjjPPXg5PFJrROCZhczep4QY0WTmcLfGrapZf1vVSRvXSlm99mmc?= =?us-ascii?Q?XdDaNwESnUsXvua31Mg+NFm0elMLOrerYIm8f8vpAChkoSZI/EjtjA=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(52116014)(1800799024)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?47eO5PcadLQvRC/RIabKSkl7I0lAcCGOAJaUI56eFU+GCIoeHn5hOTU8IPu+?= =?us-ascii?Q?S7pVS2RS8dU3qYgiVmRZLnzcwd86KKtIzJb98FgKLMgzpzorc0bl/IAzyrzS?= =?us-ascii?Q?Dfcv/+/fw74W1Bgk4UUBbWBcQyLyqonqFrFALGPfJDqvHivqUlYhXzDZQluS?= =?us-ascii?Q?30UIJfjml3UcGf6Njv6fLFk6bI3WkQ1qpEiM18zkGojByOgpq0wTYO4tglae?= =?us-ascii?Q?vJ8xHqtdGVeAp2rP3SEdz3VoOqZ9WadkT5d0rURbdS7y5dwjREKa9cyST+ga?= =?us-ascii?Q?MdfGjwEOVcOuU4dFrgD9atRkX7Wu4n9fTbAX33cXjgErU1RvUbGdUQKxMAg3?= =?us-ascii?Q?4vl7VeDcgE23N0mNut/VX9U5B3nJzqFoeRJpfRUFjQuoQ5aVntln0Tt4C1Em?= =?us-ascii?Q?GGukuhaChrA1JNggv5/gbMhwbxsOG6NfE8nJUroQ7HZxXNfTKvws2aa4Bp18?= =?us-ascii?Q?VPuX3eHOPXBbpmX2xTbxM/ntIRYpetp/EHlwvplBzusO9fm8xRkkxb2tYQB2?= =?us-ascii?Q?ajNWmeiw/cTDbpgHIPj6jYnf3MENpIRMAHVtlamPSM3Gxux2HSXe8uiQauqr?= =?us-ascii?Q?pEZfub/0TRGRxysenL+8ThvhXA5FlK8arPghuts0yItZ+MrbNDFyIqGBIW6Y?= =?us-ascii?Q?6tch7Gk79PMQn2Nx7drWOeA4CIa0QTOIG1Y7E1XZhDibRCKRspgbFFNiLeKs?= =?us-ascii?Q?SGTXgGoX8EMT/c2GoOWWFfB7VMX/kDunsvJN9rUd9+3TBddN1XqftwUamX6K?= =?us-ascii?Q?8dEwbigxAvnkkqWIdEhwnZUhPRWxOKrspRlM5kf/qh+nmt4gBKKjWjDTXAlJ?= =?us-ascii?Q?LiEYf2GV+dwTEgk7WE1fyJLXOUdixnHU6x0jRRN1l78RPwnXS0Y3Em/Z/F+f?= =?us-ascii?Q?pKc+qCitRy4EljPjqAeeV9a/gZmwMjxQg9WR1IhsPmV6ySVaprlasxkfzoDD?= =?us-ascii?Q?zfEJulA+uz5PwiLwuvZ7DQiE6BEivYWbHwiXX64Jh0AEBupE2wW+nDtTPweO?= =?us-ascii?Q?E9RlHIDVQpaysuwk73FmAdBeon/25lkriDswUj9HDACeonlbGjg45CHZc7pf?= =?us-ascii?Q?dDvR+1rGcGTU1YlKWsV25cz9vgMp6IHm5DcJMBmtHSlp2ke+npj/65jwSGES?= =?us-ascii?Q?A79vRUGQkgog2UIcj+DM0rjVtBCxbivYiK4qfZ3dtilUQZ8WZbsxllrFzGmC?= =?us-ascii?Q?2WzlryIsAsAhaCTA0393bdlRa1rCL7yodjwULf01nTnNe7zKyhHgt2ROYjQ2?= =?us-ascii?Q?o2rnH6DRzEYj9KDp6Fbc0VinPj/zP0z41Q33t2bhcg88HGicrSaRHYqenucr?= =?us-ascii?Q?/sQFmFeS3b8kdXxpnFa02osGuuZrfiD/eKw1N3QMacK78O6qsrI3NtOSEGRv?= =?us-ascii?Q?O86ly/eXS3jjZ2PxdIfFby/oC5voIGIxAknk9uoxnY8m0Jvrn09bTns2xLi8?= =?us-ascii?Q?QUEpkU0Hd89VPgKZgxyvUwde3Q6rOC3LIiQoflomN/MvLDOd7TUDhnMpAu6+?= =?us-ascii?Q?MdUooJKqUzyZ4pJ0X7hU/dbkNLM40JxvCbHoqQ2EQMgYbZvtb7UVQbaBDler?= =?us-ascii?Q?UnWQ7ZE1vfWlDxM76xiQg8xuwhc7yPhAHpMKkvW9z6OoZK/egBkND0pgquM9?= =?us-ascii?Q?wMFo5Ulmic1v2sR04q9mPg8=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 95e598df-dae9-411b-69ef-08ddcaff3b3e X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2025 22:12:51.7679 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mSE+HW2L6pZCQqrlj9b6mRRyR/NtFljmOjinofLhtYKafcrhNHrzZXct5GbonjEuc4BL3Vh1Qh00qCQaU4tJGIa0RiO7+cXO93l8TUle+c4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR01MB8089 Content-Type: text/plain; charset="utf-8" AmpereOne supports BBML2 without conflict abort, add to the allow list. Signed-off-by: Yang Shi Reviewed-by: Christoph Lameter (Ampere) Reviewed-by: Ryan Roberts --- arch/arm64/kernel/cpufeature.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7a3c404288d5..d2a8a509a58e 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2230,6 +2230,8 @@ static bool has_bbml2_noabort(const struct arm64_cpu_= capabilities *caps, int sco static const struct midr_range supports_bbml2_noabort_list[] =3D { MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf), MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), + MIDR_ALL_VERSIONS(MIDR_AMPERE1), + MIDR_ALL_VERSIONS(MIDR_AMPERE1A), {} }; =20 --=20 2.50.0 From nobody Mon Oct 6 04:59:35 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2108.outbound.protection.outlook.com [40.107.236.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1491823BD04 for ; Thu, 24 Jul 2025 22:12:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.108 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395181; cv=fail; b=RhnmYiRwVsXYXX3LLmEzCcIUA5uYl3U+Bh1TXeA7o57kVgqL1WViwK4/DTuJXQ4w/9VK0tqXQNnY6FwgBH91DdLMw6con5FKWlqhgue1LT39C6W1Br56+FigPu9zIzKFb5mWno29h28rM3P6TI6qKxGywlV/ON6BH4LTxRIFEcQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395181; c=relaxed/simple; bh=V515ThAale+V+52t/xGRn3kr7J65eDC4xSUY5KoVNAw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=aZZ90P5le1SI8P7Nt4CNHNgEz5RzgzkpofIsCoJVJPuW81JHsyfMChSCWWhG/5sAEFAyxsqbk4UVtyO+DmWvbrCh4Fm2qlpdu7W+kHmrEE2+H4ksdSwHX7IpEzUjp/P4v1lD4lTB9pODvZ+KuiVLvt/J1lPM1yVUfB8gQBTT28I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=L3irG6b+; arc=fail smtp.client-ip=40.107.236.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="L3irG6b+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=H4Fex9+vO2p/wgnjIbWbDKrLn5MBwOIlDVtP7KIGMNhcppL6uPV5P3TKzJEEAuEy9/XcepLDkoQZ+xjoO/yWRTUnm+5U5lGaYMyPTEnXztEIFYnLGYhxp14gT8NGE7t8EZE91wUgPNPMLYfDNdTHHjKW3DobEgGGtmEfCSsDo4/FjURWUG9jpT2mIlJrOlIQPvF3Mn7+Tehi2y5vh3kCEtS5G6QRtEGEjd38cK0jlTtSozvJsqI1XeYi/bunmglNX0C57pYSmrGrf9tJzeq8iF7vtzlUpGepxV2Ou8CYZESNVjVO2drx7TO3EW9Vei/7jZbLBS4W5JlMLWJ1PixAmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8oTVAal//IBUPIN6FW4+hjl5AAxBKJJvciNxVJuQ0+4=; b=TKpujrn9MH16eyvK16yOaSQ4B0QSGx/C/EVayGlESMrb0sPhqQMgBDtQ+mPMvYgOfDn+juFj70qmvmP1BUMuHJ1Jp4vh7Vk1hCjA6FiyaimAKQ0Aqoug4mPxmCBhADzoTpySC0/cNM/1mzA9q34pl7pMNgvpb5F+c47RDwwJvttiw+gwH+bmhUOeJkrP1am+f3gcgRw0Nn5oEZONQgoSSa+f94Mj+dDFEU9sSXmdFfCO465hHZACzqJaIt4x079u8CU1pVbNUcJkAftqXpaDndtYAnAa0+QVwnGZD0eD+ag9WpQPov7Q2zJyxZ8sl6SexllICOIJe0TfQmYm5+rvkw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8oTVAal//IBUPIN6FW4+hjl5AAxBKJJvciNxVJuQ0+4=; b=L3irG6b+OjlMMKKD7YQydRH7pGWoPhT393XLvpkVo1Z+1ZnUTc8Y5qHAB59oXQyYv9pO6sYgo0MLBK0spCbY9YQRVO/SnkSWSeZVPRyuOZZNAEk9tnN54Jrk0zScfW5uEaGNmtvNiOCD135mTzlBrVmqDXgpiFZCPGEvTrg2Qjw= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SN7PR01MB8089.prod.exchangelabs.com (2603:10b6:806:32f::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8964.21; Thu, 24 Jul 2025 22:12:53 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8964.019; Thu, 24 Jul 2025 22:12:53 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, akpm@linux-foundation.org, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] arm64: mm: support large block mapping when rodata=full Date: Thu, 24 Jul 2025 15:11:05 -0700 Message-ID: <20250724221216.1998696-4-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.50.0 In-Reply-To: <20250724221216.1998696-1-yang@os.amperecomputing.com> References: <20250724221216.1998696-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR15CA0236.namprd15.prod.outlook.com (2603:10b6:930:66::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SN7PR01MB8089:EE_ X-MS-Office365-Filtering-Correlation-Id: b743ba8a-893b-4f0f-7ee8-08ddcaff3bf0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|52116014|1800799024|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Wym0LDrNwNPPX8KhKj7eNNa91arJpaNMQgTFdl0vQQPVvUhrYEC5Vz9c5Tge?= =?us-ascii?Q?Knxl+ku7TNla22e3C/o4x6yNHL4TetyXzgrWpmjjxRBVTOBeTCvXXcXRw2/U?= =?us-ascii?Q?RIrU84BmSpxn+LF0n7nGwptkuJBddLXcMDMB9ywb9Ckba2mJWYgYtj48Igrp?= =?us-ascii?Q?QdR20PQpjykb5MVyIS37neC8kzhxhz1S6JZqJYOQ87Onf5sXIlooZnIt9Zoj?= =?us-ascii?Q?wFrsAufCawE4yUz0ZMYORebYlP3diPPsMYbi+ktnIrCLRwBS4HGE4JKeuMoF?= =?us-ascii?Q?gbvwZBCsR39dQbcVNTqfX9i70qcM4KDxyuOiWeMyBcbUuHNDAHUWtISbjnbL?= =?us-ascii?Q?uaH2T3zQuGC3CHnZEoBmdJ9tOuo8N6QXKNnw/oYxSh8jL6R9UXNbjl5XWvKK?= =?us-ascii?Q?/XuRS+z1tGpcnb32WBrhfF0lMJlbRDSJixzp1b9CpoRG4IhlbsBcB29S7ivQ?= =?us-ascii?Q?bsRKQsbofzmBmoDKnrzgnMghLWD5MlMTpho9kJ6RdHH2c7F1LRSWiLSpt74X?= =?us-ascii?Q?HupD1jE0fFygjTEmjz8Qh5IvERUWCDrKYU0G8or8vMBM3cb4xma37hlXWYxo?= =?us-ascii?Q?BIFSoFky7tdPIpiEc4tGZ8RGu5DuxGDpWkLIWiBXZqUkB6mCqES4fVGVeUpo?= =?us-ascii?Q?kLCKFmZ06s61Dqpxvkm/1lmZ18DazMyZDQsas+6NXcKrrkbAzpj58HCoV4LM?= =?us-ascii?Q?HtFj5nK8i0HpeR4+4SYPdqbgvtOd0iKho8is3EDGto3yi5V14vJ4UStcId9J?= =?us-ascii?Q?cZfG5F1AJeZH13EjQuTgYizQXTwKTskabhi1H017gMhXQJZDj1JMLE6SicvE?= =?us-ascii?Q?9hAxYAJZsaiZMvPfCp+s2Cm4UfDcrRIooXiPhJzorzBmLPiITRM65JyqHmWR?= =?us-ascii?Q?mbdCvfRz2+rDzXT64N+o8lSXrSXCqNyrlNHLagGRaEYHH5rDMKL60YFqaO0k?= =?us-ascii?Q?UQ+/i/ZuG7SkwaUZqFjgurH62YLliel4c7kJdLbIUSulU22fGM+igcCbBHOX?= =?us-ascii?Q?H4PvozxFleM+u9a5zRbkFhh+jL7HT4S6k1PofAVeubJey4LqG3qtt2jDn9IU?= =?us-ascii?Q?LfCclsd3mFYzgVqIOQD8Skp7tNUs30xLm1cwYZNtLG3PtozAeuzjAM1GKCSk?= =?us-ascii?Q?/s6VTAg9h0HiO/fh7EDk7YPtL4EeAiMD+Z0hSXbVdFUO+6XkHQ3cTXOCkqIV?= =?us-ascii?Q?fgrr9Ml0+FUeBaesc0AriLVysay8YgiW85Ge2q3EYk7il0fciSReEb2UjtmT?= =?us-ascii?Q?ygZPUgSIYQK49dc4JZHdgz18b09G7n2LALP3KqNjetUs7d/D9tFbYbu7Merc?= =?us-ascii?Q?Ivluu5XKdHhAEDglRHU6EKMYabkTLjiM6KBdhZ1V0eX5jVyOGWvT15hH8sff?= =?us-ascii?Q?eGSesUTEm96V0btWjnTn8kPRN7TUWWA/vEsHdU0aWnyAE9Ii2gZKZmgtg48R?= =?us-ascii?Q?AvCI1nhzQ9H446SdnTLe+/svLhg1Cpd/Tm2Mcz0cDgOU3S9eEhulsA=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(52116014)(1800799024)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7TR3zxDmr6I4O7HTRdkFT2Ndaqaxk6ZDEuo22NCPsaH+FA2wvxn5BYEm6/+2?= =?us-ascii?Q?4xgZ1o5Bpi9CVGXWwrcy9iGMEjeHv6QXVwUrra0ImjXPCGG4IdHvzNPjARtj?= =?us-ascii?Q?wsK9WDmCdmJ+peB41nRVfZ1FtzITfVBDvliB0XdlW7fZsJDwPKgwWVwCQ3Fr?= =?us-ascii?Q?rGO9nvOj7J/jjVDafiCBQdxDaqJXqRLIuu01i58UqZ2PJTHnqkXIjXIMYJsO?= =?us-ascii?Q?koJdy5R2fSAbXw+QTaCMBLWykGcN5x2Mx3By/PYgnXTtdl8g/5J6XAVKiRVs?= =?us-ascii?Q?BopGKJFXfCQumkyLx/ZPEZSEIdn16yQ7T5Wheub9tjIsEUjtSJwozKpTpgg0?= =?us-ascii?Q?T4iPZAJ7Fy380paWT7dqgUNX5Iold7knnHCb1eEJ62jVRQuoR6kVaDTbiXMO?= =?us-ascii?Q?bprKiMfO7guctwZrForXFJ1MigIHfZFjYTVVaBneMWYiI6JW/IgvjB5xSSTh?= =?us-ascii?Q?9ee5x6PLToc+47RabyrYVdk9yliaviA904Fm6SDI2YrZvpyJhh0sD7rd1RGf?= =?us-ascii?Q?qKKjOV8VNOmpBBRdpC9ZIH2mHsU+6YPYocxigus5C0zUzQNsuoWlRP25TRsL?= =?us-ascii?Q?nobvSGiePewUm9ATFQM01254gM95DRACp10GkQoHhYniHON9q86x7gdsXEIu?= =?us-ascii?Q?r8PJ5ipe4RZWBvKtPE8s6V6Copzz0GKoQy81WXRNfmV7ANRmr0qELuugF6G+?= =?us-ascii?Q?xcV/OLK8AB3pXUCyIAuP79zz3QS9ksl9NHq5E1UhbXWNhHu6L13VSu6k4Epm?= =?us-ascii?Q?Y04Y1mVpj1qxQPS/sUk0BkCud0XC0xaL6oBb5caEl/9vethiKGKarRU8QJjr?= =?us-ascii?Q?tg7ppBiXYKxAWRt8G5kbcDAcw0BtGrp5FoRFsNhMWJTZXE6OQi/P8Vvc7mhf?= =?us-ascii?Q?ka0owInxAtdCeyxMiLLPY/4pWlr1MGOtbq4f2B1ZdnwXxrFs81FTNf/bUGzf?= =?us-ascii?Q?6u29nVPXECYlbiI5PZrKupVNw2AANxKw+WwkYj847vZbKFkpK16jX6VlRJbX?= =?us-ascii?Q?1yuIarvsYps2bJk68ovzO2g1Bwq9DHkLTMcZW+N3kE6X8ID1gA1kaUpd9udd?= =?us-ascii?Q?YBRiZwdAu7ba1hAM7mq+YC7SyBsscyNAFqbk4TZi/vsJvuAkY3DMPg0g6K4M?= =?us-ascii?Q?EP4HnXq4KPlz4yywpDlG3b7SbXIgsKBCrtv1dVKCdlA5qYcs20RaqXwXVp2Q?= =?us-ascii?Q?d5RKRnGUQqsELVnTKS3Ib4LzrMb4S74kZtYuDgrMA428ItkqNitiKq5kHD09?= =?us-ascii?Q?EyJicFU6QqfB4d13KsEg1ZvcghySdet6dvgb/5yijy9bpq68VNUhcbqDsgjE?= =?us-ascii?Q?1b8GCCUj5G8qrCSWUhhcN5W1en8NFdetg6YSpQUpKCMqVQpn7KPTtBW4/1M4?= =?us-ascii?Q?WRCyi3BOlCj/nN47ind5qXrxhQCSM6M092Wo/ijGmYofboq8iA8LthsNCxsM?= =?us-ascii?Q?PmAS1KrJJ7y4BBO/iJKl6U60YqHc0nv7s1mDpXVnimb3xN1A5aqy71zXQ/eT?= =?us-ascii?Q?NbIU14DW64F9yHjpFG/LTrDz4i7uhfoNlMjGH0whrQ0hSA8BFMeYmVI4g8We?= =?us-ascii?Q?JdWMtGk69TFIKVPDMb6/l6O5BGVZGI51k1GEVQX2yBxI2pFF3v4Y21cSfNJe?= =?us-ascii?Q?oA5Nn6pmSEQV3j5cqOGRPhI=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: b743ba8a-893b-4f0f-7ee8-08ddcaff3bf0 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2025 22:12:52.9308 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hc1pD/fThjQhkU2LesG/a+Jxs7VqfOyCIEGCpPMYco7SmTEsKrsSKuTMH7W/hio7AkGhwFSOA6G0QaWdQJUFQIckn1p/29xldvFT1fdhTe0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR01MB8089 Content-Type: text/plain; charset="utf-8" When rodata=3Dfull is specified, kernel linear mapping has to be mapped at PTE level since large page table can't be split due to break-before-make rule on ARM64. This resulted in a couple of problems: - performance degradation - more TLB pressure - memory waste for kernel page table With FEAT_BBM level 2 support, splitting large block page table to smaller ones doesn't need to make the page table entry invalid anymore. This allows kernel split large block mapping on the fly. Add kernel page table split support and use large block mapping by default when FEAT_BBM level 2 is supported for rodata=3Dfull. When changing permissions for kernel linear mapping, the page table will be split to smaller size. The machine without FEAT_BBM level 2 will fallback to have kernel linear mapping PTE-mapped when rodata=3Dfull. With this we saw significant performance boost with some benchmarks and much less memory consumption on my AmpereOne machine (192 cores, 1P) with 256GB memory. * Memory use after boot Before: MemTotal: 258988984 kB MemFree: 254821700 kB After: MemTotal: 259505132 kB MemFree: 255410264 kB Around 500MB more memory are free to use. The larger the machine, the more memory saved. * Memcached We saw performance degradation when running Memcached benchmark with rodata=3Dfull vs rodata=3Don. Our profiling pointed to kernel TLB pressure. With this patchset we saw ops/sec is increased by around 3.5%, P99 latency is reduced by around 9.6%. The gain mainly came from reduced kernel TLB misses. The kernel TLB MPKI is reduced by 28.5%. The benchmark data is now on par with rodata=3Don too. * Disk encryption (dm-crypt) benchmark Ran fio benchmark with the below command on a 128G ramdisk (ext4) with disk encryption (by dm-crypt). fio --directory=3D/data --random_generator=3Dlfsr --norandommap --randrepea= t 1 \ --status-interval=3D999 --rw=3Dwrite --bs=3D4k --loops=3D1 --ioengine= =3Dsync \ --iodepth=3D1 --numjobs=3D1 --fsync_on_close=3D1 --group_reporting --th= read \ --name=3Diops-test-job --eta-newline=3D1 --size 100G The IOPS is increased by 90% - 150% (the variance is high, but the worst number of good case is around 90% more than the best number of bad case). The bandwidth is increased and the avg clat is reduced proportionally. * Sequential file read Read 100G file sequentially on XFS (xfs_io read with page cache populated). The bandwidth is increased by 150%. Signed-off-by: Yang Shi --- arch/arm64/include/asm/cpufeature.h | 34 ++++ arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/pgtable.h | 5 + arch/arm64/kernel/cpufeature.c | 31 +-- arch/arm64/mm/mmu.c | 293 +++++++++++++++++++++++++++- arch/arm64/mm/pageattr.c | 4 + 6 files changed, 333 insertions(+), 35 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/c= pufeature.h index bf13d676aae2..d0d394cc837d 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -871,6 +871,40 @@ static inline bool system_supports_pmuv3(void) return cpus_have_final_cap(ARM64_HAS_PMUV3); } =20 +static inline bool bbml2_noabort_available(void) +{ + /* + * We want to allow usage of BBML2 in as wide a range of kernel contexts + * as possible. This list is therefore an allow-list of known-good + * implementations that both support BBML2 and additionally, fulfill the + * extra constraint of never generating TLB conflict aborts when using + * the relaxed BBML2 semantics (such aborts make use of BBML2 in certain + * kernel contexts difficult to prove safe against recursive aborts). + * + * Note that implementations can only be considered "known-good" if their + * implementors attest to the fact that the implementation never raises + * TLB conflict aborts for BBML2 mapping granularity changes. + */ + static const struct midr_range supports_bbml2_noabort_list[] =3D { + MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf), + MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), + MIDR_ALL_VERSIONS(MIDR_AMPERE1), + MIDR_ALL_VERSIONS(MIDR_AMPERE1A), + {} + }; + + /* Does our cpu guarantee to never raise TLB conflict aborts? */ + if (!is_midr_in_range_list(supports_bbml2_noabort_list)) + return false; + + /* + * We currently ignore the ID_AA64MMFR2_EL1 register, and only care + * about whether the MIDR check passes. + */ + + return true; +} + static inline bool system_supports_bbml2_noabort(void) { return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT); diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 6e8aa8e72601..57f4b25e6f33 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -71,6 +71,7 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys= _addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t pro= t); extern void mark_linear_text_alias_ro(void); +extern int split_kernel_pgtable_mapping(unsigned long start, unsigned long= end); =20 /* * This check is triggered during the early boot before the cpufeature diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index ba63c8736666..ad2a6a7e86b0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -371,6 +371,11 @@ static inline pmd_t pmd_mkcont(pmd_t pmd) return __pmd(pmd_val(pmd) | PMD_SECT_CONT); } =20 +static inline pmd_t pmd_mknoncont(pmd_t pmd) +{ + return __pmd(pmd_val(pmd) & ~PMD_SECT_CONT); +} + #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp(pte_t pte) { diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d2a8a509a58e..1c96016a7a41 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2215,36 +2215,7 @@ static bool hvhe_possible(const struct arm64_cpu_cap= abilities *entry, =20 static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, i= nt scope) { - /* - * We want to allow usage of BBML2 in as wide a range of kernel contexts - * as possible. This list is therefore an allow-list of known-good - * implementations that both support BBML2 and additionally, fulfill the - * extra constraint of never generating TLB conflict aborts when using - * the relaxed BBML2 semantics (such aborts make use of BBML2 in certain - * kernel contexts difficult to prove safe against recursive aborts). - * - * Note that implementations can only be considered "known-good" if their - * implementors attest to the fact that the implementation never raises - * TLB conflict aborts for BBML2 mapping granularity changes. - */ - static const struct midr_range supports_bbml2_noabort_list[] =3D { - MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf), - MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), - MIDR_ALL_VERSIONS(MIDR_AMPERE1), - MIDR_ALL_VERSIONS(MIDR_AMPERE1A), - {} - }; - - /* Does our cpu guarantee to never raise TLB conflict aborts? */ - if (!is_midr_in_range_list(supports_bbml2_noabort_list)) - return false; - - /* - * We currently ignore the ID_AA64MMFR2_EL1 register, and only care - * about whether the MIDR check passes. - */ - - return true; + return bbml2_noabort_available(); } =20 #ifdef CONFIG_ARM64_PAN diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 3d5fb37424ab..f63b39613571 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -480,6 +480,8 @@ void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t = phys, unsigned long virt, int flags); #endif =20 +#define INVALID_PHYS_ADDR -1 + static phys_addr_t __pgd_pgtable_alloc(struct mm_struct *mm, enum pgtable_type pgtable_type) { @@ -487,7 +489,9 @@ static phys_addr_t __pgd_pgtable_alloc(struct mm_struct= *mm, struct ptdesc *ptdesc =3D pagetable_alloc(GFP_PGTABLE_KERNEL & ~__GFP_ZER= O, 0); phys_addr_t pa; =20 - BUG_ON(!ptdesc); + if (!ptdesc) + return INVALID_PHYS_ADDR; + pa =3D page_to_phys(ptdesc_page(ptdesc)); =20 switch (pgtable_type) { @@ -509,15 +513,29 @@ static phys_addr_t __pgd_pgtable_alloc(struct mm_stru= ct *mm, } =20 static phys_addr_t __maybe_unused -pgd_pgtable_alloc_init_mm(enum pgtable_type pgtable_type) +split_pgtable_alloc(enum pgtable_type pgtable_type) { return __pgd_pgtable_alloc(&init_mm, pgtable_type); } =20 +static phys_addr_t __maybe_unused +pgd_pgtable_alloc_init_mm(enum pgtable_type pgtable_type) +{ + phys_addr_t pa; + + pa =3D __pgd_pgtable_alloc(&init_mm, pgtable_type); + BUG_ON(pa =3D=3D INVALID_PHYS_ADDR); + return pa; +} + static phys_addr_t pgd_pgtable_alloc_special_mm(enum pgtable_type pgtable_type) { - return __pgd_pgtable_alloc(NULL, pgtable_type); + phys_addr_t pa; + + pa =3D __pgd_pgtable_alloc(NULL, pgtable_type); + BUG_ON(pa =3D=3D INVALID_PHYS_ADDR); + return pa; } =20 /* @@ -552,6 +570,254 @@ void __init create_pgd_mapping(struct mm_struct *mm, = phys_addr_t phys, pgd_pgtable_alloc_special_mm, flags); } =20 +static DEFINE_MUTEX(pgtable_split_lock); + +static int split_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long e= nd) +{ + pte_t *ptep; + unsigned long next; + unsigned int nr; + unsigned long span; + + ptep =3D pte_offset_kernel(pmdp, addr); + + do { + pte_t *_ptep; + + nr =3D 0; + next =3D pte_cont_addr_end(addr, end); + if (next < end) + nr =3D max(nr, ((end - next) / CONT_PTE_SIZE)); + span =3D nr * CONT_PTE_SIZE; + + _ptep =3D PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); + ptep +=3D pte_index(next) - pte_index(addr) + nr * CONT_PTES; + + if (((addr | next) & ~CONT_PTE_MASK) =3D=3D 0) + continue; + + if (!pte_cont(__ptep_get(_ptep))) + continue; + + for (int i =3D 0; i < CONT_PTES; i++, _ptep++) + __set_pte(_ptep, pte_mknoncont(__ptep_get(_ptep))); + } while (addr =3D next + span, addr !=3D end); + + return 0; +} + +static int split_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end) +{ + unsigned long next; + unsigned int nr; + unsigned long span; + int ret =3D 0; + + do { + pmd_t pmd; + + nr =3D 1; + next =3D pmd_addr_end(addr, end); + if (next < end) + nr =3D max(nr, ((end - next) / PMD_SIZE)); + span =3D (nr - 1) * PMD_SIZE; + + if (((addr | next) & ~PMD_MASK) =3D=3D 0) + continue; + + pmd =3D pmdp_get(pmdp); + if (pmd_leaf(pmd)) { + phys_addr_t pte_phys; + pte_t *ptep; + pmdval_t pmdval =3D PMD_TYPE_TABLE | PMD_TABLE_UXN | + PMD_TABLE_AF; + unsigned long pfn =3D pmd_pfn(pmd); + pgprot_t prot =3D pmd_pgprot(pmd); + + pte_phys =3D split_pgtable_alloc(TABLE_PTE); + if (pte_phys =3D=3D INVALID_PHYS_ADDR) + return -ENOMEM; + + if (pgprot_val(prot) & PMD_SECT_PXN) + pmdval |=3D PMD_TABLE_PXN; + + prot =3D __pgprot((pgprot_val(prot) & ~PTE_TYPE_MASK) | + PTE_TYPE_PAGE); + prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + ptep =3D (pte_t *)phys_to_virt(pte_phys); + for (int i =3D 0; i < PTRS_PER_PTE; i++, ptep++, pfn++) + __set_pte(ptep, pfn_pte(pfn, prot)); + + dsb(ishst); + + __pmd_populate(pmdp, pte_phys, pmdval); + } + + ret =3D split_cont_pte(pmdp, addr, next); + if (ret) + break; + } while (pmdp +=3D nr, addr =3D next + span, addr !=3D end); + + return ret; +} + +static int split_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long e= nd) +{ + pmd_t *pmdp; + unsigned long next; + unsigned int nr; + unsigned long span; + int ret =3D 0; + + pmdp =3D pmd_offset(pudp, addr); + + do { + pmd_t *_pmdp; + + nr =3D 0; + next =3D pmd_cont_addr_end(addr, end); + if (next < end) + nr =3D max(nr, ((end - next) / CONT_PMD_SIZE)); + span =3D nr * CONT_PMD_SIZE; + + if (((addr | next) & ~CONT_PMD_MASK) =3D=3D 0) { + pmdp +=3D pmd_index(next) - pmd_index(addr) + + nr * CONT_PMDS; + continue; + } + + _pmdp =3D PTR_ALIGN_DOWN(pmdp, sizeof(*pmdp) * CONT_PMDS); + if (!pmd_cont(pmdp_get(_pmdp))) + goto split; + + for (int i =3D 0; i < CONT_PMDS; i++, _pmdp++) + set_pmd(_pmdp, pmd_mknoncont(pmdp_get(_pmdp))); + +split: + ret =3D split_pmd(pmdp, addr, next); + if (ret) + break; + + pmdp +=3D pmd_index(next) - pmd_index(addr) + nr * CONT_PMDS; + } while (addr =3D next + span, addr !=3D end); + + return ret; +} + +static int split_pud(p4d_t *p4dp, unsigned long addr, unsigned long end) +{ + pud_t *pudp; + unsigned long next; + unsigned int nr; + unsigned long span; + int ret =3D 0; + + pudp =3D pud_offset(p4dp, addr); + + do { + pud_t pud; + + nr =3D 1; + next =3D pud_addr_end(addr, end); + if (next < end) + nr =3D max(nr, ((end - next) / PUD_SIZE)); + span =3D (nr - 1) * PUD_SIZE; + + if (((addr | next) & ~PUD_MASK) =3D=3D 0) + continue; + + pud =3D pudp_get(pudp); + if (pud_leaf(pud)) { + phys_addr_t pmd_phys; + pmd_t *pmdp; + pudval_t pudval =3D PUD_TYPE_TABLE | PUD_TABLE_UXN | + PUD_TABLE_AF; + unsigned long pfn =3D pud_pfn(pud); + pgprot_t prot =3D pud_pgprot(pud); + unsigned int step =3D PMD_SIZE >> PAGE_SHIFT; + + pmd_phys =3D split_pgtable_alloc(TABLE_PMD); + if (pmd_phys =3D=3D INVALID_PHYS_ADDR) + return -ENOMEM; + + if (pgprot_val(prot) & PMD_SECT_PXN) + pudval |=3D PUD_TABLE_PXN; + + prot =3D __pgprot((pgprot_val(prot) & ~PMD_TYPE_MASK) | + PMD_TYPE_SECT); + prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + pmdp =3D (pmd_t *)phys_to_virt(pmd_phys); + for (int i =3D 0; i < PTRS_PER_PMD; i++, pmdp++) { + set_pmd(pmdp, pfn_pmd(pfn, prot)); + pfn +=3D step; + } + + dsb(ishst); + + __pud_populate(pudp, pmd_phys, pudval); + } + + ret =3D split_cont_pmd(pudp, addr, next); + if (ret) + break; + } while (pudp +=3D nr, addr =3D next + span, addr !=3D end); + + return ret; +} + +static int split_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end) +{ + p4d_t *p4dp; + unsigned long next; + int ret =3D 0; + + p4dp =3D p4d_offset(pgdp, addr); + + do { + next =3D p4d_addr_end(addr, end); + + ret =3D split_pud(p4dp, addr, next); + if (ret) + break; + } while (p4dp++, addr =3D next, addr !=3D end); + + return ret; +} + +static int split_pgd(pgd_t *pgdp, unsigned long addr, unsigned long end) +{ + unsigned long next; + int ret =3D 0; + + do { + next =3D pgd_addr_end(addr, end); + ret =3D split_p4d(pgdp, addr, next); + if (ret) + break; + } while (pgdp++, addr =3D next, addr !=3D end); + + return ret; +} + +int split_kernel_pgtable_mapping(unsigned long start, unsigned long end) +{ + int ret; + + if (!system_supports_bbml2_noabort()) + return 0; + + if (start !=3D PAGE_ALIGN(start) || end !=3D PAGE_ALIGN(end)) + return -EINVAL; + + mutex_lock(&pgtable_split_lock); + arch_enter_lazy_mmu_mode(); + ret =3D split_pgd(pgd_offset_k(start), start, end); + arch_leave_lazy_mmu_mode(); + mutex_unlock(&pgtable_split_lock); + + return ret; +} + static void update_mapping_prot(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) { @@ -639,6 +905,20 @@ static inline void arm64_kfence_map_pool(phys_addr_t k= fence_pool, pgd_t *pgdp) { =20 #endif /* CONFIG_KFENCE */ =20 +bool linear_map_requires_bbml2; + +static inline bool force_pte_mapping(void) +{ + /* + * Can't use cpufeature API to determine whether BBM level 2 + * is supported or not since cpufeature have not been + * finalized yet. + */ + return (!bbml2_noabort_available() && (rodata_full || + arm64_kfence_can_set_direct_map() || is_realm_world())) || + debug_pagealloc_enabled(); +} + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end =3D _PAGE_END(VA_BITS_MIN); @@ -664,7 +944,9 @@ static void __init map_mem(pgd_t *pgdp) =20 early_kfence_pool =3D arm64_kfence_alloc_pool(); =20 - if (can_set_direct_map()) + linear_map_requires_bbml2 =3D !force_pte_mapping() && rodata_full; + + if (force_pte_mapping()) flags |=3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; =20 /* @@ -1366,7 +1648,8 @@ int arch_add_memory(int nid, u64 start, u64 size, =20 VM_BUG_ON(!mhp_range_allowed(start, size, true)); =20 - if (can_set_direct_map()) + if (force_pte_mapping() || + (linear_map_requires_bbml2 && !system_supports_bbml2_noabort())) flags |=3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; =20 __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index c6a85000fa0e..6566aa9d8abb 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -140,6 +140,10 @@ static int update_range_prot(unsigned long start, unsi= gned long size, data.set_mask =3D set_mask; data.clear_mask =3D clear_mask; =20 + ret =3D split_kernel_pgtable_mapping(start, start + size); + if (WARN_ON_ONCE(ret)) + return ret; + arch_enter_lazy_mmu_mode(); =20 /* --=20 2.50.0 From nobody Mon Oct 6 04:59:35 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2108.outbound.protection.outlook.com [40.107.236.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B58723C51C for ; Thu, 24 Jul 2025 22:13:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.108 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395183; cv=fail; b=G+pTGr2nX+kfzbGqGRohkOuk5uwBZkl7XBZiWnOT9rY+jN+7fmjcw6BzWjtZYrbCNZGHL4ca5b8ZqXc29rCgOzNW+HZbceu8g+tsj4L9BcC2gmoS8cRz2wtC2SZ7YTi3nHssgOQBz1phyoKFeUP56av0SzGN0AzBxvsfOHQO9hc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753395183; c=relaxed/simple; bh=RVwGPo+RnLGDJjnClM0QkjmgoiFd4O3QOkniDWOu+og=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=kiioD9AGS8LgaRMP9pFXA0FNmO1F+LQ+OdlfxE1+PF+QCt5FIPJco1mZkB7/ldTEkYjSlz3s00zN82xC1K57xtKjgwqG7JNHFFI2zAxy200B1RvckqED5qDA3mEYziLHUauGahZMSTByOFPrW/pxs/++mDcYWnTPPKOhF4jONF0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=VBFBsq7C; arc=fail smtp.client-ip=40.107.236.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="VBFBsq7C" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UpeHlUzTURePt64LcQ/rNRwYhbxcRr4eg9OJxAV/1nXOkvkSk/qw//QJk7vpNAy9TqPKOQ20uyX5pMmGkzR0yxfVWJ5zvwjWAqUY46NPp1JiWWd8XtABxOgoOgtwqip0GSQplR4vE72FzjmWfeSOUDx4ntMRLkJYjkdQh+S+lMXUgpvn0Qh9S1c9UIE8U+SQUVcsrPWjHvGQw6mbIVAKmdPrrLVgBE3vrYBQ79w3iPLgnrVrPb7LmaOOiA0y/g5sJo3sc35lSRK1dbIO8TTCtihDftE80SiA8vGe89jnHdM1S7cFFhrZktv7unZ5DUrDgvkfkFBTsCIQmnViX0o/Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tcbv42TPa7Jcf4GEOFALnqkkmk7FqZ59lhPBWrpWfk4=; b=MxbnIH775EI7OjesfDzF+HZmI0bUjd7UQspsjBMllnA25yj8Gcz8eB4dkLO0Au/aVdGC+PmFrwLLaoFHOL6m38jHs5fpe8AtV6C3ZD8Pwu4RfPjtp9cc/48ozd4fHR6UInj8v+0gBR8WqNSkhOvptSPWw+xkiHgmY9fQfbBYppru0dJzdVqeIiCvKwK+cnat+4CRYoh+D+lRybb2J+nh/YBTnndg/N6tuweA32caba1W6GSaZb1BrAjDPY1A2wyO1Ah4dLMszr8Sz+psO8cgw5mFUEGnPegFC/TVC9U1TPAm9o93VEi0txNswj0CxonR7+0eKuSITrdYe0HJYZwaPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tcbv42TPa7Jcf4GEOFALnqkkmk7FqZ59lhPBWrpWfk4=; b=VBFBsq7CEhDHlIQBYWSqN09qb4dDh2D4zt79bQmRS1c6rbLBoOjyGNj8xL4b1BlGHgPmPQPhhaHedAQetR/WnFva5qTPorfb3cZ42cyzmh32Q8YbDqbmQwKkxjx6khXYUdXEh+b/i5+0zapjqpG8E/Z10GhMYBnj7MsBmBmG7iM= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by SN7PR01MB8089.prod.exchangelabs.com (2603:10b6:806:32f::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8964.21; Thu, 24 Jul 2025 22:12:54 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8964.019; Thu, 24 Jul 2025 22:12:54 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, akpm@linux-foundation.org, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] arm64: mm: split linear mapping if BBML2 is not supported on secondary CPUs Date: Thu, 24 Jul 2025 15:11:06 -0700 Message-ID: <20250724221216.1998696-5-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.50.0 In-Reply-To: <20250724221216.1998696-1-yang@os.amperecomputing.com> References: <20250724221216.1998696-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR15CA0236.namprd15.prod.outlook.com (2603:10b6:930:66::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|SN7PR01MB8089:EE_ X-MS-Office365-Filtering-Correlation-Id: 8e629f54-783d-4313-3fcb-08ddcaff3c97 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|52116014|1800799024|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?cg4xlaJ660iOWKgTKOMtYzYmEsygYXMgU/ejrpr5ZN0Wgmh1Tck5HFD6PZyJ?= =?us-ascii?Q?jrh/V+aTOnoJE760fO2csvR2cb46jYeR6EXZdG+0GhxaU9KVj92FpUkJ/N9j?= =?us-ascii?Q?USpUTMjR+CvsoxUH4hEDG/jd3H18wv1HQ/g/xQcWLn3H2aHzEj5OJnYKKJbv?= =?us-ascii?Q?B1VrEy7Keh+5n1OehWiPSsflxJ5n07SRrfY9PrTqV+OtDOIuLcNcoBG73DvS?= =?us-ascii?Q?RYHLrAJt0Gysbe+d7IvrR0XzdL+VWbi2oBSQBd8RX3ZkAoeqj9/4f+HqP52I?= =?us-ascii?Q?CcC0J1HLEqWve6mE/sjLo3hJVDIVNg5xuCN3IbXjm0VbwIqmqRh1KHOAPkWa?= =?us-ascii?Q?S/BhC3SXFFfot+OYRO+loP/Dkg5VOPOZbK3fm1fgeAb7Pn4jw25TU4lxoNLe?= =?us-ascii?Q?4ZdWk9PCsDVIvrag8ZLQzQxthRKLPzmv8EtRARSwCtvsCKqgZblEfXINiIeE?= =?us-ascii?Q?/dEFjdUgYbjMd9PhFhRd83JBaPx4dUUpp+VWwjl93cTEz3RSjQWoNQHSIlPr?= =?us-ascii?Q?S4o1pOf3HmI0uPu9Gzy/K+HJXXtX9s9HJqjDWb02XbsR5mZyMwLzM4anMacC?= =?us-ascii?Q?zraUbn+FbpB84h2El0pNu0dq9rg6V1ylCD1lF4fkivyg94mKCp97sxdglWnq?= =?us-ascii?Q?RvrkEX/yqknJflRYTWZP3ZVKV9iQUG1r3Ix1AYo69G5Q5IHoPO1EZD8+oZyT?= =?us-ascii?Q?Ma9RLlNHcbLAIgeWyudNTPVPQ2mKtYafLoCrrqk6Dbsypuz7Q7JrVEcjxrQP?= =?us-ascii?Q?EdR511m/EHz1XndQttgkFPr3R6/gb91uTBvUQE3WA9maVNv8TPYxFgEzKZx4?= =?us-ascii?Q?umESXwEw/GXaNvK+bgsQsx0IomzZFzpnC9+eHPYHVPg/OAlGUkl3St7QbJ3C?= =?us-ascii?Q?iDbl8w6cmQoKnjOTrFDMUtdTOxorToXP/kFuBY3Lt4ToH7uAxlxizpvMVwHR?= =?us-ascii?Q?+2b3Ip+r8uee81NyH1StSLhYkUz6J0JaV7frKAKhlAUaW7quCNUODPIaW25R?= =?us-ascii?Q?ta+XCi4c4jwvHdiaE7huVIIR4xBPW9EgrZBWiX5kavk8eOcdnJLLDQPkWPa7?= =?us-ascii?Q?r7usweWVXafR4DiGAltOjxImhWfSUG/z45PI6kSo+IkmmrZyI1QYVZvM38om?= =?us-ascii?Q?lOQxJoymQlIzC7sMClbjw8HhAYyxuzoE4rwR36KKWz7mdjktbR6VAqHwGnsT?= =?us-ascii?Q?7IaMnO+yEuGOiqZ5AOuoJpEM7s4GZyrGdr+4ETdylh5LZdCNm2ptsiaLHLYc?= =?us-ascii?Q?0mJWRaGrcU9Xdd+VAsL7g7DKyPg/3c/ASvXUlah0HQQsHPnBVEL+y9tMQgR7?= =?us-ascii?Q?Mz5EtT0dDTqkLoEnH0lDDu3ZR/fI2Z7Mq9LxpoYN3H4BXm0CijXhS8cqch9X?= =?us-ascii?Q?ZgTFdV3Ex8HcOUUD+rzk/llz0WV29rsiycQY+0FHVkep7RWjJstCX22IHVbO?= =?us-ascii?Q?pe5BfxVqq3zcwFPjmBJnfvjg1Bp+FjiGlWehWvRu96Z3nzU7+NOV/A=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(52116014)(1800799024)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ofyE3uJubak1LrTXXZ1ce3h1DixHLEkMTleChVGTWhWRN+W/z07Ioqjswifb?= =?us-ascii?Q?QaMkeZOQe2C7pglU6wBOtrhq9ISuavTDVeTxaj6QQDPzl7ruXdGXSsWvtNPg?= =?us-ascii?Q?7pg7WBt+PdzPTpt7kEpBbMQ2EQ7lxl332jF3bpKLkAJ+ZFZO+Ds9paby8VTX?= =?us-ascii?Q?qZwviK7V3iVr5V+2niE4Wq38yaNMDAQWFQ8bvCnRllbX1si+vGbFYs6NhjDr?= =?us-ascii?Q?HqNlDNK/Vv9nYUOu63h1J7HRqiX8JU2pAjE+2jy39/VGakbK46hK7RzC0pqp?= =?us-ascii?Q?QpBUt+l6JAKRiqsn11P5J3xKN8yyfvpXKgSNcHZMPWIbIlzlSjtAm+vr0fil?= =?us-ascii?Q?vIbblRCcb0pIsGGRXeFk4Ifn7LxttrUAGdSTBnTVuuHYpyJn80kKDzWaKuWE?= =?us-ascii?Q?fHUYKPcUbynPZRLUu5npHxMg+V4VnMXqbl5vAlayrZQLdAruXIsbm8m4IV0T?= =?us-ascii?Q?NhEiS2VRtn7dz4AfGcF08Ulg/RLp/ghdtR96rCW7JuC5crn6057kc+o7eGLg?= =?us-ascii?Q?UIUCwThmiKgyOQ8eLYx5l3Z/6FjJuFnSpC09gUrLtPgjZwvdNtaHFHaFt1dN?= =?us-ascii?Q?KkCs1aFtnfe72TlQDQWGiWdsQcER2Iz0zZT/noHUAlq0H4SCKnKwt++sKMIj?= =?us-ascii?Q?Hr8nPGK8q1j8latqFGbrtHSGbeQ+y5fM+5aeLyt2QZ3KxCbuO3PRDAUA44Ae?= =?us-ascii?Q?18YcsfNH4JqHXT/EKBYYVth/5EzIpAnbWS6Ni89G12hhNvxMivriz17mW0bH?= =?us-ascii?Q?lKy5d5wOOXGuJUe+MpZDLb9fPJs7FrfOdfhnvBWgvU5rhmAQPszkbb1+6tGl?= =?us-ascii?Q?Flo+wrvtQ+dD0iM5WWKC1KwlLU5YoD1p9F0CB0hWWtU4FnHu/0C4+jYFUJqZ?= =?us-ascii?Q?hoGvVFmCOIeUXCrnv10DPTOPvkNc3nDehyP7YL0MNrIonUo8va+0YyVvnm6r?= =?us-ascii?Q?0nQ+9jRSbCFPf6GQh0ulCG4DCCLBhklRnEqyABiFDcElFx2+6eYE5qtMaoxy?= =?us-ascii?Q?r9Z0WG5TvrgXhEnezALlkE3x9TpeoxrfAbhTkFZBrHvnRfaB/1svBBN5+f7M?= =?us-ascii?Q?n01jXPVFtD/VoXuOzVJFmvhvlBTmRFDC7uLdbfKL1jFLutcEFouHiy1ZJMbT?= =?us-ascii?Q?doh0hEQ+am5XuITyQ+8LcA/Yl4CcJBzrM1IYc9YbDzkiGKOAMU4ar6GrGe+u?= =?us-ascii?Q?2pSmnVClHDsP3iuBWRwszU1vNWezHoemYk7oLB/p8ZHgesLMIEEmtZXwgvCC?= =?us-ascii?Q?efrM9qEP4r0nLT3+erUJPDxY3+qqZ/6aRT0iAESmTdru8EYGwV8ivbUWj6ED?= =?us-ascii?Q?T2OvaL0QbqPmgJb6QDmvJ1RybYN1X2R5bb/e4oN/tbG0VAdNGpTPvN9acw4G?= =?us-ascii?Q?q/L1VWDt0N+yE+P7D1cYFZUrb9lkC2wCz9y+ytBX6zjyl4jv2NZ40hS+dBP+?= =?us-ascii?Q?y1oYTm+To2lQICtL/+nXw0dfxHR3qjycus8sIm/zLZ08pD71O3XH+yupJ4Fq?= =?us-ascii?Q?UIqWWv5hlvPsmH6XzYw/nt6xR8VUVSxk7/zFvVFJT2qQzPBJXlouotn/fwG8?= =?us-ascii?Q?9PQfX+7lQ9i+9LrU4kCp8tVhR36whhZZsVvDwQjUL5F6Sy00MhhNx0GgIGXs?= =?us-ascii?Q?wbC1OkleVHQD/ReMZ5RezGQ=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8e629f54-783d-4313-3fcb-08ddcaff3c97 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2025 22:12:54.0502 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YUQQJpDrSMmf9UebFVpjQTRXvC3JxGXihCIa0pX42ZcnLTkMxV1Tiak9/oRtHjAZMrzJpkCWNAh3J8uuM2Sd2bRKEl3QZZ/n2wvy9V8pqwg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR01MB8089 Content-Type: text/plain; charset="utf-8" The kernel linear mapping is painted in very early stage of system boot. The cpufeature has not been finalized yet at this point. So the linear mapping is determined by the capability of boot CPU. If the boot CPU supports BBML2, large block mapping will be used for linear mapping. But the secondary CPUs may not support BBML2, so repaint the linear mapping if large block mapping is used and the secondary CPUs don't support BBML2 once cpufeature is finalized on all CPUs. If the boot CPU doesn't support BBML2 or the secondary CPUs have the same BBML2 capability with the boot CPU, repainting the linear mapping is not needed. Signed-off-by: Yang Shi --- arch/arm64/include/asm/mmu.h | 6 +- arch/arm64/kernel/cpufeature.c | 8 ++ arch/arm64/mm/mmu.c | 173 +++++++++++++++++++++++++++------ arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/proc.S | 57 ++++++++--- 5 files changed, 196 insertions(+), 50 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 57f4b25e6f33..9bf50e8897e2 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -56,6 +56,8 @@ typedef struct { */ #define ASID(mm) (atomic64_read(&(mm)->context.id) & 0xffff) =20 +extern bool linear_map_requires_bbml2; + static inline bool arm64_kernel_unmapped_at_el0(void) { return alternative_has_cap_unlikely(ARM64_UNMAP_KERNEL_AT_EL0); @@ -71,7 +73,9 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys= _addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t pro= t); extern void mark_linear_text_alias_ro(void); -extern int split_kernel_pgtable_mapping(unsigned long start, unsigned long= end); +extern int split_kernel_pgtable_mapping(unsigned long start, unsigned long= end, + unsigned int flags); +extern int linear_map_split_to_ptes(void *__unused); =20 /* * This check is triggered during the early boot before the cpufeature diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 1c96016a7a41..23c01d679c40 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -2009,6 +2010,12 @@ static int __init __kpti_install_ng_mappings(void *_= _unused) return 0; } =20 +static void __init linear_map_maybe_split_to_ptes(void) +{ + if (linear_map_requires_bbml2 && !system_supports_bbml2_noabort()) + stop_machine(linear_map_split_to_ptes, NULL, cpu_online_mask); +} + static void __init kpti_install_ng_mappings(void) { /* Check whether KPTI is going to be used */ @@ -3855,6 +3862,7 @@ void __init setup_system_features(void) { setup_system_capabilities(); =20 + linear_map_maybe_split_to_ptes(); kpti_install_ng_mappings(); =20 sve_setup(); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f63b39613571..22f2d0869fdd 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -482,11 +482,11 @@ void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_= t phys, unsigned long virt, =20 #define INVALID_PHYS_ADDR -1 =20 -static phys_addr_t __pgd_pgtable_alloc(struct mm_struct *mm, +static phys_addr_t __pgd_pgtable_alloc(struct mm_struct *mm, gfp_t gfp, enum pgtable_type pgtable_type) { /* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ - struct ptdesc *ptdesc =3D pagetable_alloc(GFP_PGTABLE_KERNEL & ~__GFP_ZER= O, 0); + struct ptdesc *ptdesc =3D pagetable_alloc(gfp, 0); phys_addr_t pa; =20 if (!ptdesc) @@ -513,9 +513,16 @@ static phys_addr_t __pgd_pgtable_alloc(struct mm_struc= t *mm, } =20 static phys_addr_t __maybe_unused -split_pgtable_alloc(enum pgtable_type pgtable_type) +split_pgtable_alloc(enum pgtable_type pgtable_type, int flags) { - return __pgd_pgtable_alloc(&init_mm, pgtable_type); + gfp_t gfp; + + if ((flags & (NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS)) =3D=3D 0) + gfp =3D GFP_PGTABLE_KERNEL & ~__GFP_ZERO; + else + gfp =3D GFP_ATOMIC; + + return __pgd_pgtable_alloc(&init_mm, gfp, pgtable_type); } =20 static phys_addr_t __maybe_unused @@ -523,7 +530,8 @@ pgd_pgtable_alloc_init_mm(enum pgtable_type pgtable_typ= e) { phys_addr_t pa; =20 - pa =3D __pgd_pgtable_alloc(&init_mm, pgtable_type); + pa =3D __pgd_pgtable_alloc(&init_mm, GFP_PGTABLE_KERNEL & ~__GFP_ZERO, + pgtable_type); BUG_ON(pa =3D=3D INVALID_PHYS_ADDR); return pa; } @@ -533,7 +541,8 @@ pgd_pgtable_alloc_special_mm(enum pgtable_type pgtable_= type) { phys_addr_t pa; =20 - pa =3D __pgd_pgtable_alloc(NULL, pgtable_type); + pa =3D __pgd_pgtable_alloc(NULL, GFP_PGTABLE_KERNEL & ~__GFP_ZERO, + pgtable_type); BUG_ON(pa =3D=3D INVALID_PHYS_ADDR); return pa; } @@ -572,7 +581,8 @@ void __init create_pgd_mapping(struct mm_struct *mm, ph= ys_addr_t phys, =20 static DEFINE_MUTEX(pgtable_split_lock); =20 -static int split_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long e= nd) +static int split_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long e= nd, + unsigned int flags) { pte_t *ptep; unsigned long next; @@ -586,14 +596,16 @@ static int split_cont_pte(pmd_t *pmdp, unsigned long = addr, unsigned long end) =20 nr =3D 0; next =3D pte_cont_addr_end(addr, end); - if (next < end) + if (next < end && + (flags & NO_CONT_MAPPINGS) =3D=3D 0) nr =3D max(nr, ((end - next) / CONT_PTE_SIZE)); span =3D nr * CONT_PTE_SIZE; =20 _ptep =3D PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); ptep +=3D pte_index(next) - pte_index(addr) + nr * CONT_PTES; =20 - if (((addr | next) & ~CONT_PTE_MASK) =3D=3D 0) + if (((addr | next) & ~CONT_PTE_MASK) =3D=3D 0 && + (flags & NO_CONT_MAPPINGS) =3D=3D 0) continue; =20 if (!pte_cont(__ptep_get(_ptep))) @@ -606,7 +618,8 @@ static int split_cont_pte(pmd_t *pmdp, unsigned long ad= dr, unsigned long end) return 0; } =20 -static int split_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end) +static int split_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, + unsigned int flags) { unsigned long next; unsigned int nr; @@ -618,11 +631,13 @@ static int split_pmd(pmd_t *pmdp, unsigned long addr,= unsigned long end) =20 nr =3D 1; next =3D pmd_addr_end(addr, end); - if (next < end) + if (next < end && + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) nr =3D max(nr, ((end - next) / PMD_SIZE)); span =3D (nr - 1) * PMD_SIZE; =20 - if (((addr | next) & ~PMD_MASK) =3D=3D 0) + if (((addr | next) & ~PMD_MASK) =3D=3D 0 && + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) continue; =20 pmd =3D pmdp_get(pmdp); @@ -634,7 +649,7 @@ static int split_pmd(pmd_t *pmdp, unsigned long addr, u= nsigned long end) unsigned long pfn =3D pmd_pfn(pmd); pgprot_t prot =3D pmd_pgprot(pmd); =20 - pte_phys =3D split_pgtable_alloc(TABLE_PTE); + pte_phys =3D split_pgtable_alloc(TABLE_PTE, flags); if (pte_phys =3D=3D INVALID_PHYS_ADDR) return -ENOMEM; =20 @@ -643,7 +658,8 @@ static int split_pmd(pmd_t *pmdp, unsigned long addr, u= nsigned long end) =20 prot =3D __pgprot((pgprot_val(prot) & ~PTE_TYPE_MASK) | PTE_TYPE_PAGE); - prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + if ((flags & NO_CONT_MAPPINGS) =3D=3D 0) + prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); ptep =3D (pte_t *)phys_to_virt(pte_phys); for (int i =3D 0; i < PTRS_PER_PTE; i++, ptep++, pfn++) __set_pte(ptep, pfn_pte(pfn, prot)); @@ -653,7 +669,7 @@ static int split_pmd(pmd_t *pmdp, unsigned long addr, u= nsigned long end) __pmd_populate(pmdp, pte_phys, pmdval); } =20 - ret =3D split_cont_pte(pmdp, addr, next); + ret =3D split_cont_pte(pmdp, addr, next, flags); if (ret) break; } while (pmdp +=3D nr, addr =3D next + span, addr !=3D end); @@ -661,7 +677,8 @@ static int split_pmd(pmd_t *pmdp, unsigned long addr, u= nsigned long end) return ret; } =20 -static int split_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long e= nd) +static int split_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long e= nd, + unsigned int flags) { pmd_t *pmdp; unsigned long next; @@ -676,11 +693,13 @@ static int split_cont_pmd(pud_t *pudp, unsigned long = addr, unsigned long end) =20 nr =3D 0; next =3D pmd_cont_addr_end(addr, end); - if (next < end) + if (next < end && + (flags & NO_CONT_MAPPINGS) =3D=3D 0) nr =3D max(nr, ((end - next) / CONT_PMD_SIZE)); span =3D nr * CONT_PMD_SIZE; =20 - if (((addr | next) & ~CONT_PMD_MASK) =3D=3D 0) { + if (((addr | next) & ~CONT_PMD_MASK) =3D=3D 0 && + (flags & NO_CONT_MAPPINGS) =3D=3D 0) { pmdp +=3D pmd_index(next) - pmd_index(addr) + nr * CONT_PMDS; continue; @@ -694,7 +713,7 @@ static int split_cont_pmd(pud_t *pudp, unsigned long ad= dr, unsigned long end) set_pmd(_pmdp, pmd_mknoncont(pmdp_get(_pmdp))); =20 split: - ret =3D split_pmd(pmdp, addr, next); + ret =3D split_pmd(pmdp, addr, next, flags); if (ret) break; =20 @@ -704,7 +723,8 @@ static int split_cont_pmd(pud_t *pudp, unsigned long ad= dr, unsigned long end) return ret; } =20 -static int split_pud(p4d_t *p4dp, unsigned long addr, unsigned long end) +static int split_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, + unsigned int flags) { pud_t *pudp; unsigned long next; @@ -719,11 +739,13 @@ static int split_pud(p4d_t *p4dp, unsigned long addr,= unsigned long end) =20 nr =3D 1; next =3D pud_addr_end(addr, end); - if (next < end) + if (next < end && + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) nr =3D max(nr, ((end - next) / PUD_SIZE)); span =3D (nr - 1) * PUD_SIZE; =20 - if (((addr | next) & ~PUD_MASK) =3D=3D 0) + if (((addr | next) & ~PUD_MASK) =3D=3D 0 && + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) continue; =20 pud =3D pudp_get(pudp); @@ -736,7 +758,7 @@ static int split_pud(p4d_t *p4dp, unsigned long addr, u= nsigned long end) pgprot_t prot =3D pud_pgprot(pud); unsigned int step =3D PMD_SIZE >> PAGE_SHIFT; =20 - pmd_phys =3D split_pgtable_alloc(TABLE_PMD); + pmd_phys =3D split_pgtable_alloc(TABLE_PMD, flags); if (pmd_phys =3D=3D INVALID_PHYS_ADDR) return -ENOMEM; =20 @@ -745,7 +767,8 @@ static int split_pud(p4d_t *p4dp, unsigned long addr, u= nsigned long end) =20 prot =3D __pgprot((pgprot_val(prot) & ~PMD_TYPE_MASK) | PMD_TYPE_SECT); - prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + if ((flags & NO_CONT_MAPPINGS) =3D=3D 0) + prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); pmdp =3D (pmd_t *)phys_to_virt(pmd_phys); for (int i =3D 0; i < PTRS_PER_PMD; i++, pmdp++) { set_pmd(pmdp, pfn_pmd(pfn, prot)); @@ -757,7 +780,7 @@ static int split_pud(p4d_t *p4dp, unsigned long addr, u= nsigned long end) __pud_populate(pudp, pmd_phys, pudval); } =20 - ret =3D split_cont_pmd(pudp, addr, next); + ret =3D split_cont_pmd(pudp, addr, next, flags); if (ret) break; } while (pudp +=3D nr, addr =3D next + span, addr !=3D end); @@ -765,7 +788,8 @@ static int split_pud(p4d_t *p4dp, unsigned long addr, u= nsigned long end) return ret; } =20 -static int split_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end) +static int split_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, + unsigned int flags) { p4d_t *p4dp; unsigned long next; @@ -776,7 +800,7 @@ static int split_p4d(pgd_t *pgdp, unsigned long addr, u= nsigned long end) do { next =3D p4d_addr_end(addr, end); =20 - ret =3D split_pud(p4dp, addr, next); + ret =3D split_pud(p4dp, addr, next, flags); if (ret) break; } while (p4dp++, addr =3D next, addr !=3D end); @@ -784,14 +808,15 @@ static int split_p4d(pgd_t *pgdp, unsigned long addr,= unsigned long end) return ret; } =20 -static int split_pgd(pgd_t *pgdp, unsigned long addr, unsigned long end) +static int split_pgd(pgd_t *pgdp, unsigned long addr, unsigned long end, + unsigned int flags) { unsigned long next; int ret =3D 0; =20 do { next =3D pgd_addr_end(addr, end); - ret =3D split_p4d(pgdp, addr, next); + ret =3D split_p4d(pgdp, addr, next, flags); if (ret) break; } while (pgdp++, addr =3D next, addr !=3D end); @@ -799,7 +824,8 @@ static int split_pgd(pgd_t *pgdp, unsigned long addr, u= nsigned long end) return ret; } =20 -int split_kernel_pgtable_mapping(unsigned long start, unsigned long end) +int split_kernel_pgtable_mapping(unsigned long start, unsigned long end, + unsigned int flags) { int ret; =20 @@ -811,7 +837,7 @@ int split_kernel_pgtable_mapping(unsigned long start, u= nsigned long end) =20 mutex_lock(&pgtable_split_lock); arch_enter_lazy_mmu_mode(); - ret =3D split_pgd(pgd_offset_k(start), start, end); + ret =3D split_pgd(pgd_offset_k(start), start, end, flags); arch_leave_lazy_mmu_mode(); mutex_unlock(&pgtable_split_lock); =20 @@ -851,6 +877,75 @@ void __init mark_linear_text_alias_ro(void) PAGE_KERNEL_RO); } =20 +extern u32 repaint_done; + +int __init linear_map_split_to_ptes(void *__unused) +{ + typedef void (repaint_wait_fn)(void); + extern repaint_wait_fn bbml2_wait_for_repainting; + repaint_wait_fn *wait_fn; + + int cpu =3D smp_processor_id(); + + wait_fn =3D (void *)__pa_symbol(bbml2_wait_for_repainting); + + /* + * Repainting just can be run on CPU 0 because we just can be sure + * CPU 0 supports BBML2. + */ + if (!cpu) { + phys_addr_t kernel_start =3D __pa_symbol(_stext); + phys_addr_t kernel_end =3D __pa_symbol(__init_begin); + phys_addr_t start, end; + unsigned long vstart, vend; + int flags =3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; + u64 i; + int ret; + + /* + * Wait for all secondary CPUs get prepared for repainting + * the linear mapping. + */ + smp_cond_load_acquire(&repaint_done, VAL =3D=3D num_online_cpus()); + + memblock_mark_nomap(kernel_start, kernel_end - kernel_start); + /* Split the whole linear mapping */ + for_each_mem_range(i, &start, &end) { + if (start >=3D end) + return -EINVAL; + + vstart =3D __phys_to_virt(start); + vend =3D __phys_to_virt(end); + ret =3D split_kernel_pgtable_mapping(vstart, vend, flags); + if (ret) + panic("Failed to split linear mappings\n"); + + flush_tlb_kernel_range(vstart, vend); + } + memblock_clear_nomap(kernel_start, kernel_end - kernel_start); + + /* + * Relies on dsb in flush_tlb_kernel_range() to avoid + * reordering before any page table split operations. + */ + WRITE_ONCE(repaint_done, 0); + } else { + /* + * The secondary CPUs can't run in the same address space + * with CPU 0 because accessing the linear mapping address + * when CPU 0 is repainting it is not safe. + * + * Let the secondary CPUs run busy loop in idmap address + * space when repainting is ongoing. + */ + cpu_install_idmap(); + wait_fn(); + cpu_uninstall_idmap(); + } + + return 0; +} + #ifdef CONFIG_KFENCE =20 bool __ro_after_init kfence_early_init =3D !!CONFIG_KFENCE_SAMPLE_INTERVAL; @@ -1079,7 +1174,8 @@ void __pi_map_range(u64 *pgd, u64 start, u64 end, u64= pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); =20 static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __r= o_after_init, - kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_= init; + kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_= init, + bbml2_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after= _init; =20 static void __init create_idmap(void) { @@ -1104,6 +1200,19 @@ static void __init create_idmap(void) IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, __phys_to_virt(ptep) - ptep); } + + /* + * Setup idmap mapping for repaint_done flag. It will be used if + * repainting the linear mapping is needed later. + */ + if (linear_map_requires_bbml2) { + u64 pa =3D __pa_symbol(&repaint_done); + ptep =3D __pa_symbol(bbml2_ptes); + + __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); + } } =20 void __init paging_init(void) diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 6566aa9d8abb..4471d7e510a1 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -140,7 +140,7 @@ static int update_range_prot(unsigned long start, unsig= ned long size, data.set_mask =3D set_mask; data.clear_mask =3D clear_mask; =20 - ret =3D split_kernel_pgtable_mapping(start, start + size); + ret =3D split_kernel_pgtable_mapping(start, start + size, 0); if (WARN_ON_ONCE(ret)) return ret; =20 diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 80d470aa469d..f0f9c49a4466 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -239,6 +239,25 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu= _replace_ttbr1) dsb nshst .endm =20 + .macro wait_for_boot_cpu, tmp1, tmp2, tmp3, tmp4 + /* Increment the flag to let the boot CPU know we're ready */ +1: ldxr \tmp3, [\tmp2] + add \tmp3, \tmp3, #1 + stxr \tmp4, \tmp3, [\tmp2] + cbnz \tmp4, 1b + + /* Wait for the boot CPU to finish its job */ + sevl +1: wfe + ldxr \tmp3, [\tmp2] + cbnz \tmp3, 1b + + /* All done, act like nothing happened */ + msr ttbr1_el1, \tmp1 + isb + ret + .endm + /* * void __kpti_install_ng_mappings(int cpu, int num_secondaries, phys_addr= _t temp_pgd, * unsigned long temp_pte_va) @@ -416,29 +435,35 @@ alternative_else_nop_endif __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ __idmap_cpu_set_reserved_ttbr1 x16, x17 + wait_for_boot_cpu swapper_ttb, flag_ptr, w16, w17 =20 - /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w16, [flag_ptr] - add w16, w16, #1 - stxr w17, w16, [flag_ptr] - cbnz w17, 1b + .unreq swapper_ttb + .unreq flag_ptr +SYM_FUNC_END(idmap_kpti_install_ng_mappings) + .popsection +#endif =20 - /* Wait for the boot CPU to finish messing around with swapper */ - sevl -1: wfe - ldxr w16, [flag_ptr] - cbnz w16, 1b +/* + * Wait for repainting is done. Run on secondary CPUs + * only. + */ + .pushsection ".data", "aw", %progbits +SYM_DATA(repaint_done, .long 1) + .popsection =20 - /* All done, act like nothing happened */ - msr ttbr1_el1, swapper_ttb - isb - ret + .pushsection ".idmap.text", "a" +SYM_TYPED_FUNC_START(bbml2_wait_for_repainting) + swapper_ttb .req x0 + flag_ptr .req x1 + mrs swapper_ttb, ttbr1_el1 + adr_l flag_ptr, repaint_done + __idmap_cpu_set_reserved_ttbr1 x16, x17 + wait_for_boot_cpu swapper_ttb, flag_ptr, w16, w17 =20 .unreq swapper_ttb .unreq flag_ptr -SYM_FUNC_END(idmap_kpti_install_ng_mappings) +SYM_FUNC_END(bbml2_wait_for_repainting) .popsection -#endif =20 /* * __cpu_setup --=20 2.50.0