From nobody Tue Dec 16 22:14:32 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2125.outbound.protection.outlook.com [40.107.223.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02C5C20EB for ; Sat, 31 May 2025 02:46:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.125 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659585; cv=fail; b=ClSGId7gThGVHFC/zdGFvMqOBEgFcbYegNgh6x4tEwpmQTRtGJqQONpcs4a1wbfPNF5MX1IBiCj8wXd0taLTi+3BWHKHmup7hkDr89xy+NXwC3/is4bD2MHLpZQYYOOlcJYt3pyrQQ9MIhu3/5NX4G1vfR8xsngy2lASaXZoC3s= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659585; c=relaxed/simple; bh=62APer/11vCcYnwmOd2GVWQYl12Whta79lUzjVFsarE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=JLDd4s43pgU/ws8jAY/Syqj2F9zSv7E8vYKiuxKTOLUxxVRUZJ7Z/3J5KGe01MUelkaLAiCjSCgCFr35u6czUhMwbRs1oU2lhQsZDHqMASka4OQwbOuXJT1zlCkDwFvzX1xcOXuykWTIxzChBEhol88LjqOl7AF+UuKm6vNcExY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=c9y9RJ9h; arc=fail smtp.client-ip=40.107.223.125 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="c9y9RJ9h" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cMU1lo8LTTxD1Y127ea4u2VYG2Mh6i1fjfmJ9mG9MlmNkX+7y+cVmGHv9H/i6LIO6TXL28PPp2FWc/rU16xKk5BYusX8G4T5ZnE7jk41kMDPhUmY1HEc9JwbQF/KU2VaylisQOmtpe8adCue3UnOe9SWkJTKZcAOUum+uhQSVK+8Qqtwm5OZmmPCo8NJE9PSkACmDjN+b732quV28RGs505IKCbm3Xw8rI7+zZUWaXKURwDq05k9aFPHWEs+XT3mp1KKIzwCo/NKUNnVG8iDeDhKXG8pHvacXPfi+vWxt/f/JewyVW17wOT1DGm5VwIm+2zFvVuJeK9i55Jhaa8UPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SOM3Ls3n7MIJAl/Jog+tWTcFCZakeDrJw68tj5/D+Vc=; b=rqjhe4B2DGloGlmcyTeO6O3Xnxbc/U+CF3uYJ9QQyKOxlR8XEYor2v6uozvywIGU4OGZtdPeO1bBPpwBkR+iHAYZ96dNfyUieeENSNTA6m7n0SLT5Z1g+dPF1zz+gM3gDL9ZEObPCxAxvCdlB9fldx+h7S3U2XD/mVVsT3oYQnJSmiExyKiK4Us/qHVGsQ4T2LgsaNyRiWqjt+d3SS+kviNWPKOowjqn9f24hA/t6uNrXZMEdR2H3+4YC5ocB8fzNUuhWWwZI2gBHK0W/NaB50i1rVssTFL4JyxrF6HS0cVyRsIguuwlKY0xa75M51aWqZ3YN7/3ANlnLv03cqZCqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SOM3Ls3n7MIJAl/Jog+tWTcFCZakeDrJw68tj5/D+Vc=; b=c9y9RJ9hUknRBLQ3cZ02PvQ9Bezoltb9XrgfozQP2qa6peUMsaVB0D/Gq/yepZWk3ZfFRvTONI6LszJcnNcef7fIVDRB0iOqOZsiatDT1NqRxUuGZ283kyasl8NwE0lBidhpHq+brbIVaZ/QIhRcxpbyeHl/9qjoC7iu1ZyY4b4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7551.prod.exchangelabs.com (2603:10b6:408:17d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8769.26; Sat, 31 May 2025 02:46:21 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%6]) with mapi id 15.20.8769.029; Sat, 31 May 2025 02:46:21 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] arm64: cpufeature: add AmpereOne to BBML2 allow list Date: Fri, 30 May 2025 19:41:51 -0700 Message-ID: <20250531024545.1101304-2-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250531024545.1101304-1-yang@os.amperecomputing.com> References: <20250531024545.1101304-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR22CA0079.namprd22.prod.outlook.com (2603:10b6:930:80::26) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7551:EE_ X-MS-Office365-Filtering-Correlation-Id: eed7c664-9a5f-46cd-ad6a-08dd9fed5356 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|376014|366016|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?LTua0J4o3oT6EedLalv9WuEgb8JgnrhOZ/Cn9VGsL6jGccspn/S/u/n2zu/2?= =?us-ascii?Q?c7Yy3oaKQFkfQ2K9v3dVNcOYKEDTeOVv1ybu8fJY9tL57ksCNJx/K64wkRUY?= =?us-ascii?Q?kK4r4D5+HPl4a0Xz88IMGWpzW6frbUJhXBjsYHo5x9+XjIBx2HBQN6qYQ7dU?= =?us-ascii?Q?sRB4k5z7C2yTzt02NzUo0ZsB/sJXxuArVbI9+kQYhEfJJpMoB/XUqzjxZhwO?= =?us-ascii?Q?tVcKwdotFjNm8H1lL0W2L26c1yBUWzcQ6hbI47RltHzTtpepLIQnxXyjNa6h?= =?us-ascii?Q?i7IYf+4H7kScp3+xhloCvfclS7Xj6HBlIxjWfE4XxDBc+z8pTpoikNSK7h4v?= =?us-ascii?Q?xWSjN67dKEaHv7anAG8Pd4N6EAmIqfGjvEc+a2Zk3Yg/GuFtUoKgW0M6M49J?= =?us-ascii?Q?y8VsG7b7SCj5n5Shsp7DjItPRRbMhIVmvh6eSof6uVypak/REbZBwKliyzgJ?= =?us-ascii?Q?vkmuuVaeTOVa5J8HDct1JV5pLKO1PyXzCUuQ0XD2uAW8aq6N9zK1Dq1L+rhs?= =?us-ascii?Q?UuIBJHrd/WaZsBDUXbOyWxLmRy0/p6IPkig8JIa5cyvsbb3olrfR0gzrUOy5?= =?us-ascii?Q?NWZkuhORIMC79rWxVw0qA8iJdJ0Oc8dOuSrY2xsOyIF3tqCRHbcbBvuoGbem?= =?us-ascii?Q?uQn4596FxGOuYap/xlLQLsorNTppVoX+gYhjZCQXcyYBJF/Epj4U1wUyDVs+?= =?us-ascii?Q?J0iBQ7w+X8kYZUDvHG70Z1X1BJKzb53SbMR2PIcGDmteCaxl+VkpdQpnjMbj?= =?us-ascii?Q?5/PnWeesztPjpI0LamKSguFOv6wzv5k6OKIpfllbz/HcR+GCh4Os9MuUMbbx?= =?us-ascii?Q?KLiO4lrsNf5/ER+zlT7pHU7L2s4UivcfICuQtk4oZQ/jUMa9XsNzFB/FKR0P?= =?us-ascii?Q?ZOVbotXKiyOOVq62BjalGkR8WqtVDXFcNYgq6zdfhLoYcfadpl78238UV/Ue?= =?us-ascii?Q?tCyY3VR1FHyx7g2psJ32k5lzo3oUrQb4iMdkXYjGOYSl2TnKV6lGWQUIhUpJ?= =?us-ascii?Q?82oqyZOCfSGcSV4tO8x7kWrtrP6ufy7mp8VUUQvYjfxENzFyB2KXPdlBwt6d?= =?us-ascii?Q?O6ii1wIMTWeHKx/WYB+dwhSZuK4AuzUH4Z/UG2RjsPCGdqFqjK65hGJPhKgT?= =?us-ascii?Q?mrJZXDNRiIdDufU4897QlOaDZz2n7pDxuzcHvuxYBfkk6Yd30JxXsWzApHkT?= =?us-ascii?Q?lAEkqRWBUEGN+c2QQ7QoCocogNBvCHVh584M+4w9ziXgHk9P3I1HltRJC8B8?= =?us-ascii?Q?ry+d15pdyLZjv30GkAOzuyrM//z+8l4gEfHK1ynsgJQoYNPtYjShvtUFeTwc?= =?us-ascii?Q?yVAzDdBuGUNJAKV7RlYKLw6buBDUESQoCXVIFM8AsMC/4gdzhPs1R+k0uCFy?= =?us-ascii?Q?BzwvoVaTG/FMx75TfC5zHLcFRAbUKB3VNrVHuUnnht0KfeTH8LPgxIakMiR7?= =?us-ascii?Q?EgE/2k4bKHAIS6P8McOzLLagRQzE+6s+pPeiJg5gPOXkUKmuYmyF3Q=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(376014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?wGxSPUSaNxqpTFBr67xBNIG4AeQYK+jdxOgF91tG0BinxkXzufJQmuWXPC4/?= =?us-ascii?Q?JMXaNEkPc5RNKqGqhkuf+QcZ5H2Uur/e9H7Zw7XdLiYgve4C/ahfd2BgIGtY?= =?us-ascii?Q?TS3PuVWLnvpJMkU2VX2rQ9TMCXOEqUtz8N3Ei8XsFzDGtk9mMCmReVoM+Y/S?= =?us-ascii?Q?5XvWzJym5MqLUfYeX/Ngyph5wA4XTtDbSxncS1km1Q1ICe7pXZuWSOH7O8Ww?= =?us-ascii?Q?0A1cDIjBDQawCvuAcUX2kNwwORZlKWFX+YyJIDRVd010rmQZwHNb/stH78TX?= =?us-ascii?Q?ZRndijt4n6YPnpHTfDKBiYw5hB52a5eFnOMcVs1g35Fe1ZblxJHL9MCe4bfB?= =?us-ascii?Q?w8gH2IsgZ0N4b6yA/z64Xzhf+Fwd3Ddr0xfSyFwdwNMpoRcWSx6J8Mj6YCmf?= =?us-ascii?Q?ufjhYuYZJrTWWlVrI/FHoUQR2xOWWTz1SH5mWhtX8LEfjVSa2LhUZfZL91nJ?= =?us-ascii?Q?2zSCs9g4pNdSzKTOU+LB9GJIiCX/v5FnylAja39gh4z5MvXkPELJOBMb7eF7?= =?us-ascii?Q?CaJhRxgT5bEGdswTkw4KL95YCaNLBbLaY4MilDwg5Swzz4yjrIKsPJs8Bk4F?= =?us-ascii?Q?o9G56btMpzWYkGKR4FngO3sGW+WM6+gup9s5FH/B72xXHOaMYWuLkIwXow5Z?= =?us-ascii?Q?TisyKxm82v7Oni73pG0rm1QOAwr5G2CG+II2idkJzYlmWNfVZACK7lYD0Itc?= =?us-ascii?Q?o6E14tAksnepO4mzqpkYnJ2IypOQGi+H9U0CPXkImONS0UTxpUDd+bQ1MNvv?= =?us-ascii?Q?/11vxHt4RhhVOOgBX4t5/z4kxCKzBx+dh1x20hXt9N0RUpCT172M7MOedvND?= =?us-ascii?Q?moM3TGEpECk+pdopViBX8zHVCRWYI4lfKA4ucilwy/qMoXP6ChKVox8QKDy8?= =?us-ascii?Q?XHmqfsxCb9wLFuk94PHbIT/qZq8fEzK0g0M9TLu82Tq79+Fx7fN5yfYH6W1D?= =?us-ascii?Q?ZyYTxNZpiXGrm/dTMqpsNwVHeloU1Lf49ovvwasUBbFFmnoLJgl7G9sP2zSl?= =?us-ascii?Q?k1EkL8Eq/TGebSOnsJnKdMLTuj34uBDG2v3TS7dQhyaPr2nEJ7msOOuGOfkN?= =?us-ascii?Q?BmkUjxeWaHc8twY1gkh1H6wv7JXk5LiId5yxO+Yhy1d6SI/fzX1Yf5SHVJz6?= =?us-ascii?Q?U5aB2woVqopXft9zAqzj4pQ1qMMWmMU9IilRtt7Rt/Fm0wjA0sI73Rj4vKtm?= =?us-ascii?Q?t9qtfsr2KjenFkt6aYdoHlxLEasWlJ+MIiRp+9fyLm3kly4Icg9vFXAz1grk?= =?us-ascii?Q?wgqZ0pmj6Zh3njFfaDgMcbUm0fIUzKP4ZB/13Fk8H3zA7ZVUSZBXJuVf4Kvi?= =?us-ascii?Q?ul9wLBmRUDu128dNt4vFAokcOPuJ8NeCWOCBmmuyjcVrXl0eS6kTW3YKGeSt?= =?us-ascii?Q?DnvdLY9eHBz3p//8qBuVKZTFr0mA8JKUTxHoqa9YXVH21cOrIG8u6202oEEh?= =?us-ascii?Q?wmEy3HevoaiKZ99zhGZkrlBNA7541cnOToFxXkEryG1CTvLoV68FDHOeX3Me?= =?us-ascii?Q?5GuyYUqkTwYWBCOj9ygcjB+CoNB348jYHxF+b2LsCo+gxmYUWJ2qpKSJCH+g?= =?us-ascii?Q?OQ6FNmgBf9sn7Gd8Wr2CqqOOdhqHrwwM4Yr0aWZTK9mOt+AT/Vr/F6NmYCin?= =?us-ascii?Q?EfBgmyJJAHR9zZIRUSGsKmo=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: eed7c664-9a5f-46cd-ad6a-08dd9fed5356 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2025 02:46:21.3435 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0QkEkHXxlaOx2aHIsu3pBe7CssTPwscKkBmwJHzv1Us82BSb9UU7EGR9lcvZHJrpVYxcg9kQFbUZFd6u3te5vGkvRRw2bQK5R5isAjjz4Uw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7551 Content-Type: text/plain; charset="utf-8" AmpereOne supports BBML2 without conflict abort, add to the allow list. Signed-off-by: Yang Shi --- arch/arm64/kernel/cpufeature.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 327eeabbb449..25e1fbfab6a3 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2224,6 +2224,8 @@ static bool cpu_has_bbml2_noabort(unsigned int cpu_mi= dr) static const struct midr_range supports_bbml2_noabort_list[] =3D { MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf), MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), + MIDR_ALL_VERSIONS(MIDR_AMPERE1), + MIDR_ALL_VERSIONS(MIDR_AMPERE1A), {} }; =20 --=20 2.48.1 From nobody Tue Dec 16 22:14:32 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2125.outbound.protection.outlook.com [40.107.223.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2F2613212A for ; Sat, 31 May 2025 02:46:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.125 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659589; cv=fail; b=IbZtxBdsrfRd+yWHE5VYXzd+rHiwfGQQIOGKONZzZ2Oy+fwPN8myn6vSPmttnea8Cfu+0YVPA0BPU75rR02Bdmjmr+dS9CDGwq7rmtClAfz5vR5Z2bo6fSizQO4N/u4iv5nPBHx5x+O+7/IyJgTAOXL48qKXOqqtfCy0JaWVjzA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659589; c=relaxed/simple; bh=33sYFCG9noB56VoJygupeJeY+LFPR3aLk5M6ulwmHWE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=TRpmRsHbv+NYcs8GJjOEjaS6y50JXptV+Qm3Lcvs67qT7AV8VkjHkXkG57ZTbLlQ+nDoAxsnHD+nqVfd9znNQuHf/wELjVtgwsug8AOTRTk9H0pyEOpqDs5B1TRXpVeUzpOw++4BFqlcF7zqz0k8HsB0dpjZ3WxOSFDPbaR16Vc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=DklCaIBl; arc=fail smtp.client-ip=40.107.223.125 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="DklCaIBl" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tRXlxD1CfpBxxgPaTNahB21maxikZZj0PBW9vgTaztC1pHRgQIpNqtAOOtkPgrI8Cb5ZeZgW91vnxlC+rdM8KAefodpY2baEMak/7sQHM4BOLhES6QUijlTAffmcrcPDkTwSTwlo+fC9o/423clOI72uDUbTEdcgLAm/7qSrkuJhuNiI9krBicwRFcanunYqr9CoyAPp8L8D2SzEfj/c7BSxcJOAYTAm9bn6VBaY3LaXsajMwK8lw4S2b8Keh4BzUN/yIeubyIka7LChP6JWFYkI8KbrlyC/Ik/VvXUbFSsuftNUgRjQpbEDhDqtiKA1rg4DvuTgf4o6VJ+xKcvAyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qq1yUacXyMU3P0fW6hQ2kmRrM2THBFn+cxuUfsZ5kBA=; b=WcqV8mkArHqZA6PdIcmkBHC9zhH6aqUctGkF3YikEzvZgv/BWCI/IpZInrggm+Ayj0kwfr79ivYK8ohypREAErsPMztyy3VoA/M9E6mFvfqceqV5SgU/TtLEtuelqDiYMrMWWI/WYIDYIPAOdxKkqJJbp1F5Em08oPaEs3N4lqqRKFWVU1iBr+aw7R1mdm3GZKo7xQU5ooMh7/5iNZDM8FapBBear7CHPdpNRnGmtsIjwjoXaQqDN7CzqRMTzfViKfANq1RzRGYPEIPmSqhkaI+gLqCrdpNKJd8x/Mx5tHw1M1On4k1FTQWj28nVg0w7ejFV5R2LpaK9ZY7BCyQOZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qq1yUacXyMU3P0fW6hQ2kmRrM2THBFn+cxuUfsZ5kBA=; b=DklCaIBlYWAmvlOhYXQkzgnqdFquCBUxhzdrW17wSKCrLleFJFBolt4iQmCZM5U/egkFirsCECfqIvDF8fFrMDEntEXCE6Flm+CghAWxgpiFm+DIrGmMgtw8opgZ04LAggvfvLAU+AtBGp10GodRUab1iqpcFJ9MkPbYRJcbFpA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7551.prod.exchangelabs.com (2603:10b6:408:17d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8769.26; Sat, 31 May 2025 02:46:22 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%6]) with mapi id 15.20.8769.029; Sat, 31 May 2025 02:46:22 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] arm64: mm: make __create_pgd_mapping() and helpers non-void Date: Fri, 30 May 2025 19:41:52 -0700 Message-ID: <20250531024545.1101304-3-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250531024545.1101304-1-yang@os.amperecomputing.com> References: <20250531024545.1101304-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR22CA0079.namprd22.prod.outlook.com (2603:10b6:930:80::26) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7551:EE_ X-MS-Office365-Filtering-Correlation-Id: 81789021-3abc-4c1a-194c-08dd9fed5402 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|376014|366016|7053199007|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?4P2TwkSh8j8UV+ZA6sdDvxyfqHjXMJJPioMdN83+mfgWZf/P/sEI6P5WAmLn?= =?us-ascii?Q?JM7Tivy40vij6KOwvYZNudzpSbTdJZ2M0uxWVvZwSN44bQ08GYjUy5JSldmc?= =?us-ascii?Q?Zrje3p3526Ied8KTNcKf2ZSAAQs13Nf1kznf0Mr8fau1/KdSLK55IWt7AOP/?= =?us-ascii?Q?MmCezO338fZFiak203lmSomc+82Ty6+mGgfMbQBEY3eamn0IxCKFFTVjDNnA?= =?us-ascii?Q?DaxYmKP5yo1R2xrg/NXdXk0ad8lf7ac7EvFBYkigobCaA2D2m5jjbiNvkBMD?= =?us-ascii?Q?HWOqmh1rscL2jihu0qa6kuwYnVvBpXYv8zo5bYW3fZ9Z9zKiCFgDYO7TGu3A?= =?us-ascii?Q?0TLmNlZRbspxhIA6+jI7owCdHZIgWAIX4/Hfeiykedf126paRqadpAGf+8NT?= =?us-ascii?Q?sTqwedq+n3yZJeSj797oAIPX5dKnC/TLWndMhuouxG0MdmM52sJ9Q67813zP?= =?us-ascii?Q?6HTKASWi5ZyH06X4NalS7AVNtPftp3G4TZFfhWZ56rLxUPJS/+EePCOau8yx?= =?us-ascii?Q?0L84+ef/jq78cvWO4fldWlEGzXqLCzCE0uh8YTN9Cz+V/gUxBe0eI4TlHCSi?= =?us-ascii?Q?aXI5uzGKbcaBthctQmPW+E6SrOaYzTr06VI4EmKrcfKRdi8h8SzdZH+xg5j8?= =?us-ascii?Q?to8+1KrLWhUNeGamrB0v9VPromBvTCRDkbiP6XPtPXhaoQU8AD8IJdD+vAZN?= =?us-ascii?Q?zc8i6TOvTc2+h3Nbn8L4208cpcxftPTY4vPaNNXu+gZkQz723wFI67Ojh60P?= =?us-ascii?Q?Dlp3QjoZPNgNy/VfB0iUk+s9wrPan9Pc0VdzUW+ZOjo3m0qnl5DyyBBLv1Jl?= =?us-ascii?Q?CNDOB0LsrWkFxaRxQt+oOMrMVSIJSJaB2VOQ1VmgCu5V9alnMGMRJU3tEVf0?= =?us-ascii?Q?q0IsCLKM5dxO8pmhXtIwoanQ34gfzo3atLYO6bhWQGxeq+bNp92pHcfI7omR?= =?us-ascii?Q?U0spybJhB1e+iYT3HOVmYC4ivh8o2gYhWxKcmUZV2qyAwyCbh/v9IX/XlDyA?= =?us-ascii?Q?NldEbXX+Z1xDQ2HZEpuVao0s+bGtnULbjn9VadoMoMzuCmiMmfOCRoNY405P?= =?us-ascii?Q?JLnuiadND9jiZxv060kgihn4is5xE/XkHwlQ6+E2GwEatiOZwKJvIQ1cy4Nh?= =?us-ascii?Q?YuIRhFDqH+/YqiMed282zDoaXgVmCSZhjNPDwCJQ8xkMEa8ABiA3xYMesSTt?= =?us-ascii?Q?MQNI3gV2SQp0y6JYNrqhjiprPlWNRxGLBCNB242L5pw5rkaNQYPtMsdFz3/4?= =?us-ascii?Q?VWoITanHl0Jzqr7Sc7pQc77c09IKRmxAHgv2DDzjWhAA+9Uiz20AiSp0RkuY?= =?us-ascii?Q?mJQs6U6khSv0RmzqpZSRPHsnU0vkFjd9RBpQVbfcG6VRbaNJ/O1NK6KWjoaP?= =?us-ascii?Q?P0BoBJocKc9Vx4EXKg7WQDle6kSJFgAPERa4xx2b0LKiuO4Ew7uMJ54BHRLC?= =?us-ascii?Q?1xOCAZznlVDmMoRqpsfXhlUIJ36HiN5bXFcXK+UxF93r42rlkEA8jQ=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(376014)(366016)(7053199007)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Z2b7s3PWzriYS5u3WpKjzkiGgs7JdCXKWc7MuMWX4jeiwo30Lst6F1QzMoQA?= =?us-ascii?Q?LAgQblwThHLMzt3tVmIMi6andHCXSS/QNmjM7WXxh8GLkFQXCqSswQ5kyWGg?= =?us-ascii?Q?KBa/ho2kpTHUf3QKfNkzFX7AnJICSycW8kPngvKXK9Oi9dkKjRrI3qh065aq?= =?us-ascii?Q?5B0t8Dvh8AUZTF6IYNByb7ZrpwWBkrnSCBiUIseP8UHoV3KZbPpN+CwGuEsx?= =?us-ascii?Q?u/2zFl8r7AFQKLny8evl+UU4zU9yIaD4bGlPyEeC/jujRpTf04CyR559wBwO?= =?us-ascii?Q?5YRGEPKQz8q0QJL79JNA7a77EuNL0GICNCAqqjQf7j+nF+Q1sdxy2BWJyJZp?= =?us-ascii?Q?wlKFbP0cJZcs8+BQD7KqWhjhpkv8VH4yw+rXh4GN3Nmp/n45gzi3tl0970cG?= =?us-ascii?Q?Ps8SgkDQIi49BL0cms8wJZWmnhlrxk8tWIyjm4XJ7wBLhTVS7VA6EBR5Tgyb?= =?us-ascii?Q?LXCXRGdH0m/jgG0k9zNMV1IKLggeoFzIHLTvMKtF+PxLOvQHjSXNuEldrEe/?= =?us-ascii?Q?ckLP4Z9PJMwAAR+T2K5XYFFU5SgfDxrGnAYTJ4Ga9ufKrLov1pIKPxdhhdOi?= =?us-ascii?Q?Vho1N7uY0RTma9UdGPmsqnkY+6EtGLakRPv0iNxozc/t7ZxE7MpTCNr9f7ix?= =?us-ascii?Q?uFGjO4HvC6KVWdAuazpYK3DD1C88/4/QKowZKJ9/ExzOq4mZQRXkBwM4GlPo?= =?us-ascii?Q?yZcsjfMrevK4mJ9mqmTNhXaNoghNcd9t7Nlaxjhh6TBtDggIu2vvAVh3Wa1g?= =?us-ascii?Q?1WKZfbB2lYxReaZ9vHeuQsDEXMlx6xoK2yfVAkiubzbDWzcsFXL82pxqpCPd?= =?us-ascii?Q?uam7UwkFau5l9oWrfGFPjTD+FmI12ovviDhSxOK0idYgFl5/lDoH9p8f7v6b?= =?us-ascii?Q?fiJQxFZ3r9nbYFuwD+/VrWoJILcjO21cnyBi6qfRQFNOB8+bSVkEW8PCH7KB?= =?us-ascii?Q?wlq6SG5rD9lvdlYc0RJ8Ud9SssNeqpAA7a67xNyGV9NIOujL+EoBUANvVsK+?= =?us-ascii?Q?xuOodobrg3a2wRtYajyQYpzZQIh1fKCO0Oqt74CjItRMkWhpWliMM51EoO+R?= =?us-ascii?Q?CwYUrBZ8a2q7iFNVxRZxiUYfudAiCEXcdLBwZ5+9crffbz9c8kszjYnpVaDV?= =?us-ascii?Q?W6gGcwkAY8Z9y/t4GUxbhKFIGnWkHrxR+4gGs2J/6v4HoewovBsU7wvzOb4z?= =?us-ascii?Q?7Y481s7eFxO+CgjDFiYBs3wOsky4E1jQ0ssgtPO7nqf+Z5hLMHZ9VR2zSsoj?= =?us-ascii?Q?Qxdg+2Vc/NAqaGUOzgjRA6dl0CrrH5DcxYHQRPQhREKvBdDjOvK0TWmrbdch?= =?us-ascii?Q?da+ybL9Y679mBF+UwBMjvF4isCTo62NL45AEXsPZQCOHQrmbJ0sj8QpiHH2C?= =?us-ascii?Q?iQhLZeIdBnQjXdSkS8xKq+MiAzrjwwetx8ZgW9yHYyfwZ1t05wTw35Tq9mOJ?= =?us-ascii?Q?Y46GhL/WhBzkJctf1f4+NlayGQBLgEhQV1OjhJK6r6bUPS6iE0QozOvmyp28?= =?us-ascii?Q?lddoYAEzm9ARmkiUiX+dnSRWjr/S1DN4olcG1Ai0T9Lf/2Q6of6yWOi1CVRJ?= =?us-ascii?Q?SYxkeLLHZcgiGPQXmgUPf092QAxQ6DA4URRYN+onpNQzTw11vwDDx5me6e2/?= =?us-ascii?Q?RDviQ0kCHSnMWu8vQ/Fsv4o=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 81789021-3abc-4c1a-194c-08dd9fed5402 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2025 02:46:22.3086 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: v3oyDGfGCtJcda1k1hmw0hmizSVtzcLxl9Eqts09X4NWbiQ4wIe01TlX1mAk5ztPSNi/tB2vAMS4SSkaE7X0dO9jG7cxAlgs7wIt2dmnfsU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7551 Content-Type: text/plain; charset="utf-8" The later patch will enhance __create_pgd_mapping() and related helpers to split kernel linear mapping, it requires have return value. So make __create_pgd_mapping() and helpers non-void functions. And move the BUG_ON() out of page table alloc helper since failing splitting kernel linear mapping is not fatal and can be handled by the callers in the later patch. Have BUG_ON() after __create_pgd_mapping_locked() returns to keep the current callers behavior intact. Suggested-by: Ryan Roberts Signed-off-by: Yang Shi Reviewed-by: Ryan Roberts --- arch/arm64/kernel/cpufeature.c | 10 ++- arch/arm64/mm/mmu.c | 130 +++++++++++++++++++++++---------- 2 files changed, 99 insertions(+), 41 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 25e1fbfab6a3..e879bfcf853b 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1933,9 +1933,9 @@ static bool has_pmuv3(const struct arm64_cpu_capabili= ties *entry, int scope) #define KPTI_NG_TEMP_VA (-(1UL << PMD_SHIFT)) =20 extern -void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long= virt, - phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags); +int create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long = virt, + phys_addr_t size, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags); =20 static phys_addr_t __initdata kpti_ng_temp_alloc; =20 @@ -1957,6 +1957,7 @@ static int __init __kpti_install_ng_mappings(void *__= unused) u64 kpti_ng_temp_pgd_pa =3D 0; pgd_t *kpti_ng_temp_pgd; u64 alloc =3D 0; + int err; =20 if (levels =3D=3D 5 && !pgtable_l5_enabled()) levels =3D 4; @@ -1986,9 +1987,10 @@ static int __init __kpti_install_ng_mappings(void *_= _unused) // covers the PTE[] page itself, the remaining entries are free // to be used as a ad-hoc fixmap. // - create_kpti_ng_temp_pgd(kpti_ng_temp_pgd, __pa(alloc), + err =3D create_kpti_ng_temp_pgd(kpti_ng_temp_pgd, __pa(alloc), KPTI_NG_TEMP_VA, PAGE_SIZE, PAGE_KERNEL, kpti_ng_pgd_alloc, 0); + BUG_ON(err); } =20 cpu_install_idmap(); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ea6695d53fb9..775c0536b194 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -189,15 +189,16 @@ static void init_pte(pte_t *ptep, unsigned long addr,= unsigned long end, } while (ptep++, addr +=3D PAGE_SIZE, addr !=3D end); } =20 -static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; pmd_t pmd =3D READ_ONCE(*pmdp); pte_t *ptep; + int ret =3D 0; =20 BUG_ON(pmd_sect(pmd)); if (pmd_none(pmd)) { @@ -208,6 +209,10 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned = long addr, pmdval |=3D PMD_TABLE_PXN; BUG_ON(!pgtable_alloc); pte_phys =3D pgtable_alloc(PAGE_SHIFT); + if (pte_phys =3D=3D -1) { + ret =3D -ENOMEM; + goto out; + } ptep =3D pte_set_fixmap(pte_phys); init_clear_pgtable(ptep); ptep +=3D pte_index(addr); @@ -239,13 +244,17 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned= long addr, * walker. */ pte_clear_fixmap(); + +out: + return ret; } =20 -static void init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) +static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; + int ret =3D 0; =20 do { pmd_t old_pmd =3D READ_ONCE(*pmdp); @@ -264,22 +273,27 @@ static void init_pmd(pmd_t *pmdp, unsigned long addr,= unsigned long end, BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), READ_ONCE(pmd_val(*pmdp)))); } else { - alloc_init_cont_pte(pmdp, addr, next, phys, prot, + ret =3D alloc_init_cont_pte(pmdp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; =20 BUG_ON(pmd_val(old_pmd) !=3D 0 && pmd_val(old_pmd) !=3D READ_ONCE(pmd_val(*pmdp))); } phys +=3D next - addr; } while (pmdp++, addr =3D next, addr !=3D end); + + return ret; } =20 -static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) +static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; + int ret =3D 0; pud_t pud =3D READ_ONCE(*pudp); pmd_t *pmdp; =20 @@ -295,6 +309,10 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned = long addr, pudval |=3D PUD_TABLE_PXN; BUG_ON(!pgtable_alloc); pmd_phys =3D pgtable_alloc(PMD_SHIFT); + if (pmd_phys =3D=3D -1) { + ret =3D -ENOMEM; + goto out; + } pmdp =3D pmd_set_fixmap(pmd_phys); init_clear_pgtable(pmdp); pmdp +=3D pmd_index(addr); @@ -314,21 +332,27 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned= long addr, (flags & NO_CONT_MAPPINGS) =3D=3D 0) __prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); =20 - init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); + ret =3D init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); + if (ret) + break; =20 pmdp +=3D pmd_index(next) - pmd_index(addr); phys +=3D next - addr; } while (addr =3D next, addr !=3D end); =20 pmd_clear_fixmap(); + +out: + return ret; } =20 -static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long = end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long e= nd, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; + int ret =3D 0; p4d_t p4d =3D READ_ONCE(*p4dp); pud_t *pudp; =20 @@ -340,6 +364,10 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long = addr, unsigned long end, p4dval |=3D P4D_TABLE_PXN; BUG_ON(!pgtable_alloc); pud_phys =3D pgtable_alloc(PUD_SHIFT); + if (pud_phys =3D=3D -1) { + ret =3D -ENOMEM; + goto out; + } pudp =3D pud_set_fixmap(pud_phys); init_clear_pgtable(pudp); pudp +=3D pud_index(addr); @@ -369,8 +397,10 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long = addr, unsigned long end, BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), READ_ONCE(pud_val(*pudp)))); } else { - alloc_init_cont_pmd(pudp, addr, next, phys, prot, + ret =3D alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; =20 BUG_ON(pud_val(old_pud) !=3D 0 && pud_val(old_pud) !=3D READ_ONCE(pud_val(*pudp))); @@ -379,14 +409,18 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long= addr, unsigned long end, } while (pudp++, addr =3D next, addr !=3D end); =20 pud_clear_fixmap(); + +out: + return ret; } =20 -static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long = end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long e= nd, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; + int ret =3D 0; pgd_t pgd =3D READ_ONCE(*pgdp); p4d_t *p4dp; =20 @@ -398,6 +432,10 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long = addr, unsigned long end, pgdval |=3D PGD_TABLE_PXN; BUG_ON(!pgtable_alloc); p4d_phys =3D pgtable_alloc(P4D_SHIFT); + if (p4d_phys =3D=3D -1) { + ret =3D -ENOMEM; + goto out; + } p4dp =3D p4d_set_fixmap(p4d_phys); init_clear_pgtable(p4dp); p4dp +=3D p4d_index(addr); @@ -412,8 +450,10 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long = addr, unsigned long end, =20 next =3D p4d_addr_end(addr, end); =20 - alloc_init_pud(p4dp, addr, next, phys, prot, + ret =3D alloc_init_pud(p4dp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; =20 BUG_ON(p4d_val(old_p4d) !=3D 0 && p4d_val(old_p4d) !=3D READ_ONCE(p4d_val(*p4dp))); @@ -422,23 +462,27 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long= addr, unsigned long end, } while (p4dp++, addr =3D next, addr !=3D end); =20 p4d_clear_fixmap(); + +out: + return ret; } =20 -static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long addr, end, next; pgd_t *pgdp =3D pgd_offset_pgd(pgdir, virt); + int ret =3D 0; =20 /* * If the virtual and physical address don't have the same offset * within a page, we cannot map the region as the caller expects. */ if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) - return; + return -EINVAL; =20 phys &=3D PAGE_MASK; addr =3D virt & PAGE_MASK; @@ -446,10 +490,14 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir,= phys_addr_t phys, =20 do { next =3D pgd_addr_end(addr, end); - alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, + ret =3D alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; phys +=3D next - addr; } while (pgdp++, addr =3D next, addr !=3D end); + + return ret; } =20 static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, @@ -458,17 +506,20 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_a= ddr_t phys, phys_addr_t (*pgtable_alloc)(int), int flags) { + int err; + mutex_lock(&fixmap_lock); - __create_pgd_mapping_locked(pgdir, phys, virt, size, prot, - pgtable_alloc, flags); + err =3D __create_pgd_mapping_locked(pgdir, phys, virt, size, prot, + pgtable_alloc, flags); + BUG_ON(err); mutex_unlock(&fixmap_lock); } =20 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 extern __alias(__create_pgd_mapping_locked) -void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long= virt, - phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags); +int create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long = virt, + phys_addr_t size, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags); #endif =20 static phys_addr_t __pgd_pgtable_alloc(int shift) @@ -476,13 +527,17 @@ static phys_addr_t __pgd_pgtable_alloc(int shift) /* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ void *ptr =3D (void *)__get_free_page(GFP_PGTABLE_KERNEL & ~__GFP_ZERO); =20 - BUG_ON(!ptr); + if (!ptr) + return -1; + return __pa(ptr); } =20 static phys_addr_t pgd_pgtable_alloc(int shift) { phys_addr_t pa =3D __pgd_pgtable_alloc(shift); + if (pa =3D=3D -1) + goto out; struct ptdesc *ptdesc =3D page_ptdesc(phys_to_page(pa)); =20 /* @@ -498,6 +553,7 @@ static phys_addr_t pgd_pgtable_alloc(int shift) else if (shift =3D=3D PMD_SHIFT) BUG_ON(!pagetable_pmd_ctor(ptdesc)); =20 +out: return pa; } =20 --=20 2.48.1 From nobody Tue Dec 16 22:14:32 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2125.outbound.protection.outlook.com [40.107.223.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0E64149E13 for ; Sat, 31 May 2025 02:46:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.125 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659592; cv=fail; b=X9mvb28J0ZC1inv4EY030P29jH/BtQR5pdo7Z46n85Ds8gjbN81cL0fQ6IdN6G/QuFbAPTSvnBVEJVTNGQPWNyg7HFETdRX73E6N9+HwRDIKnLlRGp3+H/1TW+10OzoVwKHIZGXI8NXgaP0dUBkANBZCe1pfPE4iTXULApi4DIM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659592; c=relaxed/simple; bh=GtvEoCr1F7n64w/1oLGWgvENVSoOl92GrxtJJD2zofQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=stBYtZGV2OzWOGSMn5IYM87qc5+bHRYvKQWfOvEHC/sEMd70UZDYJ+DNjN2WQVTIxBVlFimlX04UqCEeGqUr+Kf/vHNcW2vOIDVnpfoRz5Cw9omlZluBBaXRvmEGNmiVkJCN/vP7Aow1YiUGC2O3Ey+34jdBwgr+GB6Gei/mUno= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=sHOxMZYL; arc=fail smtp.client-ip=40.107.223.125 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="sHOxMZYL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=csHDjY/WGMcfhms7tZNFKsEeHABDOOx191SwQ49SngGklU3qytJUEg50R1PDuGOzmfyGFCCeJNykVHFnaxILplZ3mXPCWaU6slpXWitr4ptpD/ZaxVj0yzilVTirn4dKeiTIzwd7/YRBaZDT0KJAm7hGG7296CoB1Y7lDUc1SN0MiDuyDf8vZ5XvDCK2Q0su7ALkQEMn74RoaNWupvejXKYnuNG7RmvuIWUNTI1x2ndWxygV3sXn+PFDFPmxU87q1FDEFydtDB5A6aTRooecoPK/ChxLI/YLCAcbgBp+/ZP5cbVWAci3Aubdq7FKqWTEsA03q2mT8Md7OlSoOLoiEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JCvwe2kD5x8NGvCLpvQJMVYq++qUWCiwYy9RkEaD1zY=; b=PC9ej7MgqRnmSAFzoKoHC6uyJ95Awtnd2pAax1sRNu/sH5sgdiXVsyZHY9r0INBSdWYTAoCViFFE2T8gGdKPT+t53Cag9NbxWNVYensa0c/jrA1HDv0y02NBYLCbsSHqYJnPAuEQ6lYKg0fM8UPcJ3mkXVZeCEYZeJaF65lGcwGbClZOsENBiyDcUL7onBChONofYW24mu+CbKkdQvW6ba9B4ZQg5mQFd179SZI1SWwtfy2Q4dGgEQWHkWcpnS+1NV9gDKVvdoqSGDbiEW1Qu5N6jJcChqYZRnc4/b+fC/Fa6BHWzSltGAsaZqhDmH2CIcjTHphUpLeivgdgS6hJBQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JCvwe2kD5x8NGvCLpvQJMVYq++qUWCiwYy9RkEaD1zY=; b=sHOxMZYLM7ueQMHKr7r46FJ8Zu4Ts2PZQ3H13KxTRQ3Kc0SYRZaK3I30e0YpRvwIqSrAZJkJend7Y9fFBE5hTyKivt80/GbrCi7dbDu08KEZFqu6QwVSM9EwYO84huIBuJrW8We3TMUb2jIAAk6dKvxcIsaopbv8kITK8SW7UHA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7551.prod.exchangelabs.com (2603:10b6:408:17d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8769.26; Sat, 31 May 2025 02:46:23 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%6]) with mapi id 15.20.8769.029; Sat, 31 May 2025 02:46:23 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] arm64: mm: support large block mapping when rodata=full Date: Fri, 30 May 2025 19:41:53 -0700 Message-ID: <20250531024545.1101304-4-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250531024545.1101304-1-yang@os.amperecomputing.com> References: <20250531024545.1101304-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR22CA0079.namprd22.prod.outlook.com (2603:10b6:930:80::26) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7551:EE_ X-MS-Office365-Filtering-Correlation-Id: 1c2d0aae-5ad5-4dbc-de30-08dd9fed548f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|376014|366016|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?hNt4nIa3Nkb/9Wev7nMmNIs3c/qxkZDUX7dVDuUBHCPHt8G97jdKaWjBy7f6?= =?us-ascii?Q?ggOosGNwqzwlEC+RUebE9N4ZFWcB0QMK+j0c0R7NtA/VwcO7zo9lJWWkKvpS?= =?us-ascii?Q?S+DPvIO8Z8b80XCOuPDVm5StMBk5SNPyCrFL8QudvhmPx4KKzvs9nc0BQb+V?= =?us-ascii?Q?euEbSP3MN2tdmyH1vjaUFt59MOfAkOLU8pEYEKQ6KFrq81NjsJwyCsNHM6Up?= =?us-ascii?Q?6OKbgy/6uPr92FI2XXz4Wjo8jrm9t7aE+hgcIxYhs+ASZrmqnE6+EHaqlZCU?= =?us-ascii?Q?HgcB81DajY1cZqLTVdLgHY9dzhkB4IPQEnsrNIUodpZQL+OS5Ayd9C+U8fCo?= =?us-ascii?Q?kezLmVWShJBxakx7bOPEtjJP6imLq/Yk2ND4YZ9CZ3SXF/0SIK72JBHUaU4l?= =?us-ascii?Q?EobNdA8qLyi5ZXlCXvzTnegYYThEAO2rqNmAHtcCgj4M35rIR3p6UBtCIUtS?= =?us-ascii?Q?zQ9cHXhCZ69nap/6GSeyutFLqMlI94ZaX8qPe3EMcTFwzkgDjWIj9U847Aez?= =?us-ascii?Q?k8xTPrQa6jj5u4oLHZxS8Dw5Azb+HoakhodokwIQUJmRwRZxNDB981EJUNFG?= =?us-ascii?Q?4Et/dybrQrQPzfWcWrtevvrU7W8RzsqRRVBHedaT+GfNFcN8TZxQYHbwNi25?= =?us-ascii?Q?powiGDr6Rw0RU9F3slggbwqYbd91SXk1Jv/vppImWsc9zYl7uN+yWW+v+gRB?= =?us-ascii?Q?pGjnkEz1Oq75M9bJgwrZlgxww7No7hGS3bs7fi5o70orE9BTD8IHucvehq1Z?= =?us-ascii?Q?925fSoXH/h1LRDH04T6d/x90sMwX6zro1BeZr3ZAFPPuOjmwE4825nBCRj6F?= =?us-ascii?Q?kCEHxAe68+KmiS8x01WcPmB+mUmtmD49V3eZuGCgbwBBznCfmrBqtUwpBnG/?= =?us-ascii?Q?//LBw3eXgxIDG8a+DaGASizmovCl4UGWJ5Vva05FuPDodejyP8RSYwoRo6iZ?= =?us-ascii?Q?/YGGEfI0tV4WtRnE4e3/p6lvrKX1sY5TaI9QNaz9KYe0ehRPQYKplnqzcA2/?= =?us-ascii?Q?y0E5nHc+OXDFah/7ObmsAwWX20kq/B6nfvJvAvoKmf1PpLOJIZbRxJW0jiYM?= =?us-ascii?Q?LXB7oInqwtLORvAKh8blr7jjJnijgBl7I9ZDgLtz5IH8MTlJ8GbSLryAsyCj?= =?us-ascii?Q?RMAA4LdoQZJinrbRBxtGJtIaXR4ZXz10608ScqjdORqVCUuK+6XOrJWjy4yB?= =?us-ascii?Q?4CwRRZUEaTKftf1odF2z6I/OB6eY0YqWVgPjxCIasQ6H/mVlBpqWvh6KQluu?= =?us-ascii?Q?Lh9n+VnuWDfnNByoSeDItnvXolg11UrTuUji31fsfx3dgwwsNzX4QW1RU8YS?= =?us-ascii?Q?lneNIr3olLfYruNLUK8qSSSaHDyKSaNAsl2A8aCCvFBPhDOnCLiHMGXyCmsN?= =?us-ascii?Q?o1Ukp2RiKBJfjSKNJJK74xTM/IIThx/9722BNJdC2zfqDzBalUy08NpgV7Al?= =?us-ascii?Q?c7u+bNJ4AY+eXYuUvgoj73OTNNk+mbMTAb4mtEW8yECD7WkG4JRexA=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(376014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?G/7MH3NBAm8fRCiNEIY0svWZtR6CPrt4SDguwJWEE6mB3mx674eiDnJLV/VQ?= =?us-ascii?Q?fkQHFzL7gFmoSgghweFSkqrsgS9yF68UbjC6CJvrJi2GtL2DR8zA5f8sP1YF?= =?us-ascii?Q?7Uu2JlbSvmgKzHhYxZ7TYvNzQ/i85Rs1biR6Vc1gUw2P9jNJm1JlUd6Pfdrg?= =?us-ascii?Q?7on7Ic5oMUBQhfSzR7yoZwWKDTJalZ2F2ou8aDOgL93EZKwA8Y1IobdTlVgr?= =?us-ascii?Q?CV3xzCN36HYLS8HSayw9TzMGlRqReHGaTTmDgdsuLNxKBVWMMrdzRnO8PcwL?= =?us-ascii?Q?EUEieXIbeKWpfov/If+Bn3zpURDWJg31+yHaLCOIMRwsLYLCIUsFSrQiqen5?= =?us-ascii?Q?C4ddedOU+FqPb16KMS1LPpfsFZJm1Jz/nJC2hznpnHcA9GU5NQJ9ZFbYHrYG?= =?us-ascii?Q?PojEv1zgZr7ezRBn8LI/vurpuDjJwXuNbM0NV3TBezoqxDwvi6E/lJb5VJuz?= =?us-ascii?Q?3OtDexyHDZ7qPJhxqURoBpOXZyV8ffJrbHQ0IibYwOw9s0BxYghno88xqBaa?= =?us-ascii?Q?p2bow/tojHb4nGTfxvl1NiXS3FdSmTGgbDrddkIY9GoZD6fptiCfPnkq4HOP?= =?us-ascii?Q?RsEY5RPXF2JHy4oHhOlyd8mJzwXXYCpEbCrqEbC5e+Y23Wahr3IvkrOnPWQl?= =?us-ascii?Q?mXQPlJzRcu3q8ZbZ+5rClO7Oguye1ntaPrRC8VzVpyjhQI0mRRI37yw3UBqB?= =?us-ascii?Q?+RmgcLA5lE9+ISSYSCsTvXkFcrBW9EY61jflyvcoTS1oWQbhS60Rbq2Z6Oyx?= =?us-ascii?Q?5qP4L7o2ij/ALayZMaW2Kpk6iNJ560rPTPUp5KZTiwtpZKRNgzxtJKqV7LQd?= =?us-ascii?Q?GIRGpX0gFED8Cy9MNBC6QhDXxmBuzQmR17l3PpadklU+vX04u2DG3pMG2BU6?= =?us-ascii?Q?wKPVlW1hkiMoTIWJCYCTMKmUc4F2GBQpNn2KG2yfOh5iyj2X/BPluh/C/B9f?= =?us-ascii?Q?MKQWCH1464XDpSJRb+G290GjNHvNlObWGfZ7p4UUzV8uxCWn/M/FAiressJR?= =?us-ascii?Q?qT1xXauSqXyVbRH+M7w2ZWCIQNF9vcGAvWRyQO0pA5mw0VnFnWf1Tw6ehpfu?= =?us-ascii?Q?Y1QVviV/OwneDFjqtb4aMyaXepDWpfmksi7kcAY4tvpATyL1IPRkV+27Rkan?= =?us-ascii?Q?+3R87jwnqq9YzwrPJtHp5QINw1UEX7h3M0TUNIXRKY17Ltjqy3HELHRpRyw9?= =?us-ascii?Q?9l8YP/uPCU0VB5ywVL3t9NlBTVE7IBPEmHA+HXtUWMT8LswjSSRUCijYckVR?= =?us-ascii?Q?aAl0O0iBEEl9zfLj8S7Q/SM+1u8+UuthNz280d2hQH1Saii6jqiHfhFmZOrU?= =?us-ascii?Q?8s8LqdCFBaJBwXtCI9vVFtiOcO2vkfdkTJyAZDszhKbvwQvOKOP0DUpVg+JQ?= =?us-ascii?Q?6pa6+fia3c3Z98lgAPGZgmqjiTp+xfUlrjyJcRRNgTJhpdad9u8S+qjmtdSv?= =?us-ascii?Q?1vIw0D3scVLBUhTG2H75aqfAEDvN6mFNTAgQ4oOUZgfbBmJun9J0QOo9F4sO?= =?us-ascii?Q?OdkbTWoXmjArvEF99LttAOttf5ST7UrkTKGcEAGdOHmrop1dRTKd67mUoSDM?= =?us-ascii?Q?fJk83GSYMNTQ20b50f4XRhwQvcV25WOJoD0reynidF/MFKry/kzcDTblgDBA?= =?us-ascii?Q?7P7ZWnQls3p6Nwi65SFuAw4=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1c2d0aae-5ad5-4dbc-de30-08dd9fed548f X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2025 02:46:23.2932 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wQZqdeK37tDPR3LP0HeBr7gRqHv8q24a4OrOMjUZ+vo7qsee3wvAcalYD4fx34E9LPhmNw45ujZSkOFrZXbMQxAOJzTeY+weg8zbuB0HOU8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7551 Content-Type: text/plain; charset="utf-8" When rodata=3Dfull is specified, kernel linear mapping has to be mapped at PTE level since large page table can't be split due to break-before-make rule on ARM64. This resulted in a couple of problems: - performance degradation - more TLB pressure - memory waste for kernel page table With FEAT_BBM level 2 support, splitting large block page table to smaller ones doesn't need to make the page table entry invalid anymore. This allows kernel split large block mapping on the fly. Add kernel page table split support and use large block mapping by default when FEAT_BBM level 2 is supported for rodata=3Dfull. When changing permissions for kernel linear mapping, the page table will be split to smaller size. The machine without FEAT_BBM level 2 will fallback to have kernel linear mapping PTE-mapped when rodata=3Dfull. With this we saw significant performance boost with some benchmarks and much less memory consumption on my AmpereOne machine (192 cores, 1P) with 256GB memory. * Memory use after boot Before: MemTotal: 258988984 kB MemFree: 254821700 kB After: MemTotal: 259505132 kB MemFree: 255410264 kB Around 500MB more memory are free to use. The larger the machine, the more memory saved. * Memcached We saw performance degradation when running Memcached benchmark with rodata=3Dfull vs rodata=3Don. Our profiling pointed to kernel TLB pressure. With this patchset we saw ops/sec is increased by around 3.5%, P99 latency is reduced by around 9.6%. The gain mainly came from reduced kernel TLB misses. The kernel TLB MPKI is reduced by 28.5%. The benchmark data is now on par with rodata=3Don too. * Disk encryption (dm-crypt) benchmark Ran fio benchmark with the below command on a 128G ramdisk (ext4) with disk encryption (by dm-crypt). fio --directory=3D/data --random_generator=3Dlfsr --norandommap --randrepea= t 1 \ --status-interval=3D999 --rw=3Dwrite --bs=3D4k --loops=3D1 --ioengine= =3Dsync \ --iodepth=3D1 --numjobs=3D1 --fsync_on_close=3D1 --group_reporting --th= read \ --name=3Diops-test-job --eta-newline=3D1 --size 100G The IOPS is increased by 90% - 150% (the variance is high, but the worst number of good case is around 90% more than the best number of bad case). The bandwidth is increased and the avg clat is reduced proportionally. * Sequential file read Read 100G file sequentially on XFS (xfs_io read with page cache populated). The bandwidth is increased by 150%. Signed-off-by: Yang Shi --- arch/arm64/include/asm/cpufeature.h | 26 +++ arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/pgtable.h | 12 +- arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/mm/mmu.c | 269 +++++++++++++++++++++++++--- arch/arm64/mm/pageattr.c | 37 +++- 6 files changed, 319 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/c= pufeature.h index 8f36ffa16b73..a95806980298 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -1053,6 +1053,32 @@ static inline bool cpu_has_lpa2(void) #endif } =20 +bool cpu_has_bbml2_noabort(unsigned int cpu_midr); + +static inline bool has_nobbml2_override(void) +{ + u64 mmfr2; + unsigned int bbm; + + mmfr2 =3D read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &=3D ~id_aa64mmfr2_override.mask; + mmfr2 |=3D id_aa64mmfr2_override.val; + bbm =3D cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_BBM_SHIFT); + return bbm =3D=3D 0; +} + +/* + * Called at early boot stage on boot CPU before cpu info and cpu feature + * are ready. + */ +static inline bool bbml2_noabort_available(void) +{ + return IS_ENABLED(CONFIG_ARM64_BBML2_NOABORT) && + cpu_has_bbml2_noabort(read_cpuid_id()) && + !has_nobbml2_override(); +} + #endif /* __ASSEMBLY__ */ =20 #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 6e8aa8e72601..2693d63bf837 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -71,6 +71,7 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys= _addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t pro= t); extern void mark_linear_text_alias_ro(void); +extern int split_linear_mapping(unsigned long start, unsigned long end); =20 /* * This check is triggered during the early boot before the cpufeature diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index d3b538be1500..bf3cef31d243 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -293,6 +293,11 @@ static inline pmd_t pmd_mkcont(pmd_t pmd) return __pmd(pmd_val(pmd) | PMD_SECT_CONT); } =20 +static inline pmd_t pmd_mknoncont(pmd_t pmd) +{ + return __pmd(pmd_val(pmd) & ~PMD_SECT_CONT); +} + static inline pte_t pte_mkdevmap(pte_t pte) { return set_pte_bit(pte, __pgprot(PTE_DEVMAP | PTE_SPECIAL)); @@ -769,7 +774,7 @@ static inline bool in_swapper_pgdir(void *addr) ((unsigned long)swapper_pg_dir & PAGE_MASK); } =20 -static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) +static inline void __set_pmd_nosync(pmd_t *pmdp, pmd_t pmd) { #ifdef __PAGETABLE_PMD_FOLDED if (in_swapper_pgdir(pmdp)) { @@ -779,6 +784,11 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) #endif /* __PAGETABLE_PMD_FOLDED */ =20 WRITE_ONCE(*pmdp, pmd); +} + +static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) +{ + __set_pmd_nosync(pmdp, pmd); =20 if (pmd_valid(pmd)) { dsb(ishst); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index e879bfcf853b..5fc2a4a804de 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2209,7 +2209,7 @@ static bool hvhe_possible(const struct arm64_cpu_capa= bilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } =20 -static bool cpu_has_bbml2_noabort(unsigned int cpu_midr) +bool cpu_has_bbml2_noabort(unsigned int cpu_midr) { /* * We want to allow usage of bbml2 in as wide a range of kernel contexts diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 775c0536b194..4c5d3aa35d62 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,6 +45,7 @@ #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ +#define SPLIT_MAPPINGS BIT(3) =20 u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); @@ -166,12 +167,91 @@ static void init_clear_pgtable(void *table) dsb(ishst); } =20 +static void split_cont_pte(pte_t *ptep) +{ + pte_t *_ptep =3D PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); + pte_t _pte; + + for (int i =3D 0; i < CONT_PTES; i++, _ptep++) { + _pte =3D READ_ONCE(*_ptep); + _pte =3D pte_mknoncont(_pte); + __set_pte_nosync(_ptep, _pte); + } + + dsb(ishst); + isb(); +} + +static void split_cont_pmd(pmd_t *pmdp) +{ + pmd_t *_pmdp =3D PTR_ALIGN_DOWN(pmdp, sizeof(*pmdp) * CONT_PMDS); + pmd_t _pmd; + + for (int i =3D 0; i < CONT_PMDS; i++, _pmdp++) { + _pmd =3D READ_ONCE(*_pmdp); + _pmd =3D pmd_mknoncont(_pmd); + set_pmd(_pmdp, _pmd); + } +} + +static void split_pmd(pmd_t pmd, phys_addr_t pte_phys, int flags) +{ + pte_t *ptep; + unsigned long pfn; + pgprot_t prot; + + pfn =3D pmd_pfn(pmd); + prot =3D pmd_pgprot(pmd); + prot =3D __pgprot((pgprot_val(prot) & ~PMD_TYPE_MASK) | PTE_TYPE_PAGE); + + ptep =3D (pte_t *)phys_to_virt(pte_phys); + + /* It must be naturally aligned if PMD is leaf */ + if ((flags & NO_CONT_MAPPINGS) =3D=3D 0) + prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + + for (int i =3D 0; i < PTRS_PER_PTE; i++, ptep++, pfn++) + __set_pte_nosync(ptep, pfn_pte(pfn, prot)); + + dsb(ishst); +} + +static void split_pud(pud_t pud, phys_addr_t pmd_phys, int flags) +{ + pmd_t *pmdp; + unsigned long pfn; + pgprot_t prot; + unsigned int step =3D PMD_SIZE >> PAGE_SHIFT; + + pfn =3D pud_pfn(pud); + prot =3D pud_pgprot(pud); + pmdp =3D (pmd_t *)phys_to_virt(pmd_phys); + + /* It must be naturally aligned if PUD is leaf */ + if ((flags & NO_CONT_MAPPINGS) =3D=3D 0) + prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + + for (int i =3D 0; i < PTRS_PER_PMD; i++, pmdp++) { + __set_pmd_nosync(pmdp, pfn_pmd(pfn, prot)); + pfn +=3D step; + } + + dsb(ishst); +} + static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) + phys_addr_t phys, pgprot_t prot, int flags) { do { pte_t old_pte =3D __ptep_get(ptep); =20 + if (flags & SPLIT_MAPPINGS) { + if (pte_cont(old_pte)) + split_cont_pte(ptep); + + continue; + } + /* * Required barriers to make this visible to the table walker * are deferred to the end of alloc_init_cont_pte(). @@ -199,11 +279,20 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned = long addr, pmd_t pmd =3D READ_ONCE(*pmdp); pte_t *ptep; int ret =3D 0; + bool split =3D flags & SPLIT_MAPPINGS; + pmdval_t pmdval; + phys_addr_t pte_phys; =20 - BUG_ON(pmd_sect(pmd)); - if (pmd_none(pmd)) { - pmdval_t pmdval =3D PMD_TYPE_TABLE | PMD_TABLE_UXN | PMD_TABLE_AF; - phys_addr_t pte_phys; + if (!split) + BUG_ON(pmd_sect(pmd)); + + if (pmd_none(pmd) && split) { + ret =3D -EINVAL; + goto out; + } + + if (pmd_none(pmd) || (split && pmd_leaf(pmd))) { + pmdval =3D PMD_TYPE_TABLE | PMD_TABLE_UXN | PMD_TABLE_AF; =20 if (flags & NO_EXEC_MAPPINGS) pmdval |=3D PMD_TABLE_PXN; @@ -213,6 +302,18 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned l= ong addr, ret =3D -ENOMEM; goto out; } + } + + if (split) { + if (pmd_leaf(pmd)) { + split_pmd(pmd, pte_phys, flags); + __pmd_populate(pmdp, pte_phys, pmdval); + } + ptep =3D pte_offset_kernel(pmdp, addr); + goto split_pgtable; + } + + if (pmd_none(pmd)) { ptep =3D pte_set_fixmap(pte_phys); init_clear_pgtable(ptep); ptep +=3D pte_index(addr); @@ -222,17 +323,28 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned = long addr, ptep =3D pte_set_fixmap_offset(pmdp, addr); } =20 +split_pgtable: do { pgprot_t __prot =3D prot; =20 next =3D pte_cont_addr_end(addr, end); =20 + if (split) { + pte_t pteval =3D READ_ONCE(*ptep); + bool cont =3D pte_cont(pteval); + + if (cont && + ((addr | next) & ~CONT_PTE_MASK) =3D=3D 0 && + (flags & NO_CONT_MAPPINGS) =3D=3D 0) + continue; + } + /* use a contiguous mapping if the range is suitably aligned */ if ((((addr | next | phys) & ~CONT_PTE_MASK) =3D=3D 0) && (flags & NO_CONT_MAPPINGS) =3D=3D 0) __prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); =20 - init_pte(ptep, addr, next, phys, __prot); + init_pte(ptep, addr, next, phys, __prot, flags); =20 ptep +=3D pte_index(next) - pte_index(addr); phys +=3D next - addr; @@ -243,7 +355,8 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned lo= ng addr, * ensure that all previous pgtable writes are visible to the table * walker. */ - pte_clear_fixmap(); + if (!split) + pte_clear_fixmap(); =20 out: return ret; @@ -255,15 +368,29 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, = unsigned long end, { unsigned long next; int ret =3D 0; + bool split =3D flags & SPLIT_MAPPINGS; + bool cont; =20 do { pmd_t old_pmd =3D READ_ONCE(*pmdp); =20 next =3D pmd_addr_end(addr, end); =20 + if (split && pmd_leaf(old_pmd)) { + cont =3D pgprot_val(pmd_pgprot(old_pmd)) & PTE_CONT; + if (cont) + split_cont_pmd(pmdp); + + /* The PMD is fully contained in the range */ + if (((addr | next) & ~PMD_MASK) =3D=3D 0 && + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) + continue; + } + /* try section mapping first */ if (((addr | next | phys) & ~PMD_MASK) =3D=3D 0 && - (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) { + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0 && + (flags & SPLIT_MAPPINGS) =3D=3D 0) { pmd_set_huge(pmdp, phys, prot); =20 /* @@ -278,7 +405,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, un= signed long end, if (ret) break; =20 - BUG_ON(pmd_val(old_pmd) !=3D 0 && + BUG_ON(!split && pmd_val(old_pmd) !=3D 0 && pmd_val(old_pmd) !=3D READ_ONCE(pmd_val(*pmdp))); } phys +=3D next - addr; @@ -296,14 +423,23 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned = long addr, int ret =3D 0; pud_t pud =3D READ_ONCE(*pudp); pmd_t *pmdp; + bool split =3D flags & SPLIT_MAPPINGS; + pudval_t pudval; + phys_addr_t pmd_phys; =20 /* * Check for initial section mappings in the pgd/pud. */ - BUG_ON(pud_sect(pud)); - if (pud_none(pud)) { - pudval_t pudval =3D PUD_TYPE_TABLE | PUD_TABLE_UXN | PUD_TABLE_AF; - phys_addr_t pmd_phys; + if (!split) + BUG_ON(pud_sect(pud)); + + if (pud_none(pud) && split) { + ret =3D -EINVAL; + goto out; + } + + if (pud_none(pud) || (split && pud_leaf(pud))) { + pudval =3D PUD_TYPE_TABLE | PUD_TABLE_UXN | PUD_TABLE_AF; =20 if (flags & NO_EXEC_MAPPINGS) pudval |=3D PUD_TABLE_PXN; @@ -313,6 +449,18 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned l= ong addr, ret =3D -ENOMEM; goto out; } + } + + if (split) { + if (pud_leaf(pud)) { + split_pud(pud, pmd_phys, flags); + __pud_populate(pudp, pmd_phys, pudval); + } + pmdp =3D pmd_offset(pudp, addr); + goto split_pgtable; + } + + if (pud_none(pud)) { pmdp =3D pmd_set_fixmap(pmd_phys); init_clear_pgtable(pmdp); pmdp +=3D pmd_index(addr); @@ -322,11 +470,22 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned = long addr, pmdp =3D pmd_set_fixmap_offset(pudp, addr); } =20 +split_pgtable: do { pgprot_t __prot =3D prot; =20 next =3D pmd_cont_addr_end(addr, end); =20 + if (split) { + pmd_t pmdval =3D READ_ONCE(*pmdp); + bool cont =3D pgprot_val(pmd_pgprot(pmdval)) & PTE_CONT; + + if (cont && + ((addr | next) & ~CONT_PMD_MASK) =3D=3D 0 && + (flags & NO_CONT_MAPPINGS) =3D=3D 0) + continue; + } + /* use a contiguous mapping if the range is suitably aligned */ if ((((addr | next | phys) & ~CONT_PMD_MASK) =3D=3D 0) && (flags & NO_CONT_MAPPINGS) =3D=3D 0) @@ -340,7 +499,8 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned lo= ng addr, phys +=3D next - addr; } while (addr =3D next, addr !=3D end); =20 - pmd_clear_fixmap(); + if (!split) + pmd_clear_fixmap(); =20 out: return ret; @@ -355,6 +515,16 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long a= ddr, unsigned long end, int ret =3D 0; p4d_t p4d =3D READ_ONCE(*p4dp); pud_t *pudp; + bool split =3D flags & SPLIT_MAPPINGS; + + if (split) { + if (p4d_none(p4d)) { + ret=3D -EINVAL; + goto out; + } + pudp =3D pud_offset(p4dp, addr); + goto split_pgtable; + } =20 if (p4d_none(p4d)) { p4dval_t p4dval =3D P4D_TYPE_TABLE | P4D_TABLE_UXN | P4D_TABLE_AF; @@ -377,17 +547,26 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long = addr, unsigned long end, pudp =3D pud_set_fixmap_offset(p4dp, addr); } =20 +split_pgtable: do { pud_t old_pud =3D READ_ONCE(*pudp); =20 next =3D pud_addr_end(addr, end); =20 + if (split && pud_leaf(old_pud)) { + /* The PUD is fully contained in the range */ + if (((addr | next) & ~PUD_MASK) =3D=3D 0 && + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) + continue; + } + /* * For 4K granule only, attempt to put down a 1GB block */ if (pud_sect_supported() && ((addr | next | phys) & ~PUD_MASK) =3D=3D 0 && - (flags & NO_BLOCK_MAPPINGS) =3D=3D 0) { + (flags & NO_BLOCK_MAPPINGS) =3D=3D 0 && + (flags & SPLIT_MAPPINGS) =3D=3D 0) { pud_set_huge(pudp, phys, prot); =20 /* @@ -402,13 +581,14 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long = addr, unsigned long end, if (ret) break; =20 - BUG_ON(pud_val(old_pud) !=3D 0 && + BUG_ON(!split && pud_val(old_pud) !=3D 0 && pud_val(old_pud) !=3D READ_ONCE(pud_val(*pudp))); } phys +=3D next - addr; } while (pudp++, addr =3D next, addr !=3D end); =20 - pud_clear_fixmap(); + if (!split) + pud_clear_fixmap(); =20 out: return ret; @@ -423,6 +603,16 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long a= ddr, unsigned long end, int ret =3D 0; pgd_t pgd =3D READ_ONCE(*pgdp); p4d_t *p4dp; + bool split =3D flags & SPLIT_MAPPINGS; + + if (split) { + if (pgd_none(pgd)) { + ret =3D -EINVAL; + goto out; + } + p4dp =3D p4d_offset(pgdp, addr); + goto split_pgtable; + } =20 if (pgd_none(pgd)) { pgdval_t pgdval =3D PGD_TYPE_TABLE | PGD_TABLE_UXN | PGD_TABLE_AF; @@ -445,6 +635,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long ad= dr, unsigned long end, p4dp =3D p4d_set_fixmap_offset(pgdp, addr); } =20 +split_pgtable: do { p4d_t old_p4d =3D READ_ONCE(*p4dp); =20 @@ -461,7 +652,8 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long ad= dr, unsigned long end, phys +=3D next - addr; } while (p4dp++, addr =3D next, addr !=3D end); =20 - p4d_clear_fixmap(); + if (!split) + p4d_clear_fixmap(); =20 out: return ret; @@ -557,6 +749,25 @@ static phys_addr_t pgd_pgtable_alloc(int shift) return pa; } =20 +int split_linear_mapping(unsigned long start, unsigned long end) +{ + int ret =3D 0; + + if (!system_supports_bbml2_noabort()) + return 0; + + mmap_write_lock(&init_mm); + /* NO_EXEC_MAPPINGS is needed when splitting linear map */ + ret =3D __create_pgd_mapping_locked(init_mm.pgd, virt_to_phys((void *)sta= rt), + start, (end - start), __pgprot(0), + __pgd_pgtable_alloc, + NO_EXEC_MAPPINGS | SPLIT_MAPPINGS); + mmap_write_unlock(&init_mm); + flush_tlb_kernel_range(start, end); + + return ret; +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -676,6 +887,24 @@ static inline void arm64_kfence_map_pool(phys_addr_t k= fence_pool, pgd_t *pgdp) { =20 #endif /* CONFIG_KFENCE */ =20 +static inline bool force_pte_mapping(void) +{ + /* + * Can't use cpufeature API to determine whether BBML2 supported + * or not since cpufeature have not been finalized yet. + * + * Checking the boot CPU only for now. If the boot CPU has + * BBML2, paint linear mapping with block mapping. If it turns + * out the secondary CPUs don't support BBML2 once cpufeature is + * fininalized, the linear mapping will be repainted with PTE + * mapping. + */ + return (rodata_full && !bbml2_noabort_available()) || + debug_pagealloc_enabled() || + arm64_kfence_can_set_direct_map() || + is_realm_world(); +} + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end =3D _PAGE_END(VA_BITS_MIN); @@ -701,7 +930,7 @@ static void __init map_mem(pgd_t *pgdp) =20 early_kfence_pool =3D arm64_kfence_alloc_pool(); =20 - if (can_set_direct_map()) + if (force_pte_mapping()) flags |=3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; =20 /* @@ -1402,7 +1631,7 @@ int arch_add_memory(int nid, u64 start, u64 size, =20 VM_BUG_ON(!mhp_range_allowed(start, size, true)); =20 - if (can_set_direct_map()) + if (force_pte_mapping()) flags |=3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; =20 __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 39fd1f7ff02a..25c068712cb5 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -10,6 +10,7 @@ #include =20 #include +#include #include #include #include @@ -42,6 +43,8 @@ static int change_page_range(pte_t *ptep, unsigned long a= ddr, void *data) struct page_change_data *cdata =3D data; pte_t pte =3D __ptep_get(ptep); =20 + BUG_ON(pte_cont(pte)); + pte =3D clear_pte_bit(pte, cdata->clear_mask); pte =3D set_pte_bit(pte, cdata->set_mask); =20 @@ -80,8 +83,9 @@ static int change_memory_common(unsigned long addr, int n= umpages, unsigned long start =3D addr; unsigned long size =3D PAGE_SIZE * numpages; unsigned long end =3D start + size; + unsigned long l_start; struct vm_struct *area; - int i; + int i, ret; =20 if (!PAGE_ALIGNED(addr)) { start &=3D PAGE_MASK; @@ -118,7 +122,12 @@ static int change_memory_common(unsigned long addr, in= t numpages, if (rodata_full && (pgprot_val(set_mask) =3D=3D PTE_RDONLY || pgprot_val(clear_mask) =3D=3D PTE_RDONLY)) { for (i =3D 0; i < area->nr_pages; i++) { - __change_memory_common((u64)page_address(area->pages[i]), + l_start =3D (u64)page_address(area->pages[i]); + ret =3D split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + + __change_memory_common(l_start, PAGE_SIZE, set_mask, clear_mask); } } @@ -174,6 +183,9 @@ int set_memory_valid(unsigned long addr, int numpages, = int enable) =20 int set_direct_map_invalid_noflush(struct page *page) { + unsigned long l_start; + int ret; + struct page_change_data data =3D { .set_mask =3D __pgprot(0), .clear_mask =3D __pgprot(PTE_VALID), @@ -182,13 +194,21 @@ int set_direct_map_invalid_noflush(struct page *page) if (!can_set_direct_map()) return 0; =20 + l_start =3D (unsigned long)page_address(page); + ret =3D split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + l_start, PAGE_SIZE, change_page_range, + &data); } =20 int set_direct_map_default_noflush(struct page *page) { + unsigned long l_start; + int ret; + struct page_change_data data =3D { .set_mask =3D __pgprot(PTE_VALID | PTE_WRITE), .clear_mask =3D __pgprot(PTE_RDONLY), @@ -197,9 +217,14 @@ int set_direct_map_default_noflush(struct page *page) if (!can_set_direct_map()) return 0; =20 + l_start =3D (unsigned long)page_address(page); + ret =3D split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + l_start, PAGE_SIZE, change_page_range, + &data); } =20 static int __set_memory_enc_dec(unsigned long addr, --=20 2.48.1 From nobody Tue Dec 16 22:14:32 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2125.outbound.protection.outlook.com [40.107.223.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27909151985 for ; Sat, 31 May 2025 02:46:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.125 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659594; cv=fail; b=s4uhdfWmRbYnC2mFuU6Rz0JK81mwFVYnXstPayGhLC6yc3eG7w1b05UPQHs1s4yNGiE9/RaPsxpRknw5ggdIdulGJFi7Ua32OXDl0GoCxArJZr/Cm77qAp7mx1QaWXoK1V4C6nAOBtt3IycmdSrx3AhNfemjlovOzmuSl+RJuiA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748659594; c=relaxed/simple; bh=/a+woRAUa/adTLeD95LlY3jIcFpAfpjKgBdw28QCASM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=JrhEfQJpkiKBHwEoD6PPYyHOrejMlJSKB29pXgokLPhZ3Igjv9cu4o68xs0LKdfLj/EgG0cyA+fFfOkHXm/sbSqCTkEEcFMJYisRypwWM09sa58P/yjaoe9cyxQtwM1EH4PpY8gH69Q3rWsqjUOva93z9dPlJVzmQegYakDvlSQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com; spf=pass smtp.mailfrom=os.amperecomputing.com; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b=BaUjgI8y; arc=fail smtp.client-ip=40.107.223.125 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=os.amperecomputing.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=os.amperecomputing.com header.i=@os.amperecomputing.com header.b="BaUjgI8y" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yU3SSl9nAp55XUU8t7m/CuNOtrn+h1O+zc6rllRi4jGJLlp5P9wmAXmpuRU4Zk+EsQb05LedgyM0CAruTMzzu7T5bPOsjMVw7wx/EbmLHc653OizVpSOWSdjE6oRpATCMLV2F96LPDcgkMaxJqfpnf5gqkXr6oAfkL6P/LQ9+pEAfy0c7A4XxH2u74Hp9nljwMU85UynskEFu+z2uWQThc9qT8Ae7J9c6F5jhYppK7QPvV5Ga6uRi/2chH3V6PB3nPpYc/mhroveMJMnhwhK0QPzMloJGdi2YboaOczmszlQ17Pwrl4CkcDlA0lAU12lczkq4nv8CIjZs8NFhattVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/hdaU4qAj7KfXSEvI3ZktRssDqTg9Rc3Fto0Nrb6oW4=; b=jfwCkvw/srPFc+0aeE0D6eHIFlsd+pBlHgA7rd+WMAVKR7u1obGdC/vjSwhWoQ0NSPuKc0GfGAGDMmV4hQ2wYPImryNHxqx236AqPmvgzOQFIErxsw4vkIeP+/Ju/JSl6NDIxpT4xC6TtycgsqLxO9mZfyioIJTqsPvikKEQLmsPU1QK56mD24wKru5GF12f+Tzf8P4T12TNDMmEMUNeHJKKxwUG/CwKoykDjOB+uUHCprRp0ATQK3Y8y4Lab4n7eNN5KMEOu2cdiWNDLwKcyWoolk5NXxLUXU3RtPN7XQo3fROPyyHODowVY9jtggzg41amE+z3tc3Yl/ebKbQ8+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/hdaU4qAj7KfXSEvI3ZktRssDqTg9Rc3Fto0Nrb6oW4=; b=BaUjgI8yOU8u8bS0Bv1z2h8UywNHBzFPmWfIRHIfnX07wnNh+MZh3WJf7q1C7bSVilgZOVzJlGoXCgjuwcP1fFt0og1vEB0c033qXG0xoX68k8CUvQxJBS/pqKaz7FPzl0EyyEFwRe8rndTPzMaapeBojwp+uXjGoUFv0bxiEFo= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by LV2PR01MB7551.prod.exchangelabs.com (2603:10b6:408:17d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8769.26; Sat, 31 May 2025 02:46:24 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%6]) with mapi id 15.20.8769.029; Sat, 31 May 2025 02:46:24 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, dev.jain@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] arm64: mm: split linear mapping if BBML2 is not supported on secondary CPUs Date: Fri, 30 May 2025 19:41:54 -0700 Message-ID: <20250531024545.1101304-5-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250531024545.1101304-1-yang@os.amperecomputing.com> References: <20250531024545.1101304-1-yang@os.amperecomputing.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: CY5PR22CA0079.namprd22.prod.outlook.com (2603:10b6:930:80::26) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|LV2PR01MB7551:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d347729-a9e7-4bb8-6471-08dd9fed5523 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|376014|366016|38350700014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?85oMLblO6J0Q1pNV7Z3MDlB+6ygpFhAm68XwLa2j5H0l0uha5V9qSi1z3ihi?= =?us-ascii?Q?Dfg5/wDPSIFLk59E7xK8wXAwGb4RJ4NwgHG110pdWo2m+c3eKiFqAkiYdaE0?= =?us-ascii?Q?6DMaGLeN3OfR/DIyzyirocn+6qLmnQdSn0LNlGVzka3db/w+jAmx7LWSwXPJ?= =?us-ascii?Q?j/1e4NB2hrsLq8a9rhXVb4Ym3cEhRiRUDL2vIJaDoph6OIeIZC6Q8vfbl6AD?= =?us-ascii?Q?jOm4d4oM7LLnnMeOOY4wTzoU4VfpCK+IjzwpnnxAl0rFoZWYYZbyBwdfoO+D?= =?us-ascii?Q?FFQvAp6rCnrO1FKmRP+p4nEb68XDua8Ic9X/IqokQo+ghYGRGmkLeRBRG0YC?= =?us-ascii?Q?MV+CjujZgXnvHo3FT4nrUL/7SVdyzlqRwhqWp6SVFOpe9Glge0AmGE7xSk8/?= =?us-ascii?Q?KjptOoP0WqZbDzTIV3vUdMxqI7Okfnr6PwsJu2hPb4slRPVS9RVzU+XtDdMV?= =?us-ascii?Q?BI9RNE2DvZlchTRpwd5zIxqspnGBw4kX8/B51YAfm/ShT1TOauusDPPg3EZV?= =?us-ascii?Q?x7DGUf4PFJVR6E+QT4EbW713Jc0M+VEqgDdX03m6y//SfGUj3ylI/IKcLAHa?= =?us-ascii?Q?ZUh/235TL/ZW7jvmk6G/TpbKl6BKNicm21kQrrZ66K3iohCV/dC+5caUeeAe?= =?us-ascii?Q?LlbRGmWgeFI7IO3t8CFfYnkE1SCDArC5lE16cXpMYzXisLdFQEPz9VDo7bYN?= =?us-ascii?Q?mdWeiFg7S2wUyWXERcKIQei/GSiCcwRSUSiymzp8oZemddSEmDeB7D9KqDGW?= =?us-ascii?Q?TlIbQolqbAjZRUKCnib/b6s1HEpk0pRoBvXT+PmCq9fKCUgOScU0OnioJuBr?= =?us-ascii?Q?m/xUHJl26L3CIpm35/sU5hh20HNI4Ocix4kKacFy0d4Gs4JbgWd6aTshhSWG?= =?us-ascii?Q?bCbyOnizpnXOpborg2GsSEFdyk3AXNHBvuPOjzQQMJPPcHmIhySkPZD2xxzB?= =?us-ascii?Q?3Af/eX4kbyUiZbqGQv2CkDSVpGWnDVuoihVofux17jnhV0XHyCpd60xQn1od?= =?us-ascii?Q?RLhWx15Z6G/KBE8Oqh6TzR8UQN5fQlajEAZ9E0sA88vukf1/0vyJCJcgWv32?= =?us-ascii?Q?wxYwn5BYKlKBoj4iBaZBew3hKYwdmeV3nunAm6oJvVQGZBkZWU6IOvLNQHFS?= =?us-ascii?Q?Uw9/pRG061KQRdqF4aoJ8Xd1BkoZFXK1VAVYQhlYNnvBRPZ5aXMtscxVljt7?= =?us-ascii?Q?SSXEy6XkaEFCS/37tqgJ6efmjrZ9anEsybvaS8wk7amzs6CIq5aZlYVj0JKs?= =?us-ascii?Q?jRPmprcmcmrjKel4GMxy8r9HTTskLsGUIrhUSPs6Q9wNCKBdJ0hDy9XX/cex?= =?us-ascii?Q?rWhIdAT+5QmbWvPnl3Qmyk4L+8QzYskhkEZD3IzM5x2sRxwB8DuYpRSnDkey?= =?us-ascii?Q?ZMr5uNw+CS8t2AHs/QldbVXGK1Z1riY8orZfW7UFVamEInwbz+i9Bez4V6RR?= =?us-ascii?Q?2eDggmMc6tfLWVnx1xOXCVXn+F9qQxGlnb0FGlbudaL0wl5Tj2rl9Q=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(376014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1Ds1ETskWE6WB1jhKlHTboyL0muCWEZG143C27w2XPWY4WMPcFZgjRQDyN3a?= =?us-ascii?Q?siyPKV77iK420kq3i9HS+zFdZKmpQVQFmL/FkRkOZNLuEhxqVt//tkb0HpiI?= =?us-ascii?Q?kgJkVC84gpL6W7TXQ6pj2JuWWaDfLEadsXIwgIy4+F1ZzJL12vTubfYGQ3h9?= =?us-ascii?Q?uWrP8xy05bBkL7zV0D1Ox/n2pI7nqPQCK/AB022IlYfEwuXHzliJWlQOdP38?= =?us-ascii?Q?vALxRN5jGr5Wb3E7jVx1iwXl+u8nwC41H+qhk52/8/L/QAR0uGKCMCZYsZtO?= =?us-ascii?Q?4d2NQGV4k69RdKNMyn0CQ6J15LuXurZJCg9reaGz0I8L5vE25j1cyWLr4I0k?= =?us-ascii?Q?kchr4ISHTJgTBT9i9oREgj8TdaM4poUqXDk2STx4TBAv1Cb/c3K3D7JNuz3j?= =?us-ascii?Q?O1M38MZTFDmpd+jhTqzxLSExCr4vmJk3pkeus0K+GeYfczuvTzKBnNB62WwP?= =?us-ascii?Q?3XlWZGjGUMfEMc2iSoExLHUM6uBWZhnjN1lSm3mtX0j5Ume82CCCu1NTnVmW?= =?us-ascii?Q?+tB3R3S6CpERHCqBkqVAQL+7YyPwLmKguMlMrPo+CTHhkcU+jzAl6hFOTerr?= =?us-ascii?Q?KuehVLPnk6v2VjiL419LtDdWShbHxEaHWCNcD7bYbDRJd6cs4QytzPB8mH1y?= =?us-ascii?Q?mKxamwplPmfaNveaRp1XLoiEjqxQLB5zS/5Ih9N34qFYzjNMMu/Ymr0yv2IF?= =?us-ascii?Q?c0pzQ4aBOGPCvtRLSZO2NS7zexk6h/ESSSsgnO0I0J1ERor456uAqannYm1s?= =?us-ascii?Q?h2Tu+EpN7BBs8GaNDLL9cTxDUO5KXDMpQvF+7O7FsPRaZ4ZW4bjo9xLtWyQv?= =?us-ascii?Q?FDteSb44I9YR36Yc7XvYQWsj5DaLoouD96ANYl8nQQKYPpU8RFBAocYa9zN0?= =?us-ascii?Q?6YuZMus63VMoT45acwUtaXud8pbuXF+vmZ6NKIoP4AvnRJcQCbVI2jktWDV1?= =?us-ascii?Q?nTfhoGzYlQ8SndQWHu02UnXGzhY0DsagPYdry2i8izQUltFLQzsDAKU3pzFW?= =?us-ascii?Q?bQUChUvyBM8belIR3Fd1BGSUWjtaSp+Jp6MUv1fVgEBfPQ+R4oz0JxUBrBko?= =?us-ascii?Q?9+oFfSKj40YMtVZuAmIS/h7xTXyzQOUV1NEChIVV1JZy+umKGBu3cS+Ak/Lw?= =?us-ascii?Q?m/Q+1xR6flSMctWx9K9QSW1uOb12ccUxLK5y+0ZyfNLxbU9HiNGPXiTs2cUu?= =?us-ascii?Q?VCje49zFohCYF0db0fpvlaBCBMaqpDhQiqSBDPolcOt+f6E6DZR+QF+IjUHb?= =?us-ascii?Q?JnvdNCIq+0MKyEqKD5VxB/J35cV4mrXvUe1rEHOngEuAgc9WA4B+bmIP3mZ1?= =?us-ascii?Q?Mons76ShxYAqCO1H/gdqRDyPsx2nVAm6Cb++tjEzhKCzNK6JpyN1S3rr8ivH?= =?us-ascii?Q?fBJ1aE3biw9D0X9k187dlDsErmbYOY1X/7N3NF377z1hK9VNjf0Op8ziMlyY?= =?us-ascii?Q?2lKiTEnd6heDeUFMLHFP0D3T5QL5JGwSSH62as5SQdi1vh7g7fYZ1DntKVZO?= =?us-ascii?Q?8o1wqUSnw9ZIvpJJh56Rso6Dm82gN25K5+fWOCa71tNnAlFFqPE26SWivFct?= =?us-ascii?Q?VKCvp9guFsSJJMYEXLymQiHgD65MqYrpIEumA5J+PjwcIAtMcZXLroTTAzTm?= =?us-ascii?Q?NU5yLiW5YTc8XCxLKe5uMMk=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d347729-a9e7-4bb8-6471-08dd9fed5523 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2025 02:46:24.2000 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2+Gy52ZumMt24AdoXMwzU5oqf6JvNnI59w73sNe4gS3CceQKn1RMfbzTmII7/Q/UPYmyA9TZvFZm7XDCg9rN+bV4vvPabSzjlvJTiZp64Xg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR01MB7551 Content-Type: text/plain; charset="utf-8" The kernel linear mapping is painted in very early stage of system boot. The cpufeature has not been finalized yet at this point. So the linear mapping is determined by the capability of boot CPU. If the boot CPU supports BBML2, large block mapping will be used for linear mapping. But the secondary CPUs may not support BBML2, so repaint the linear mapping if large block mapping is used and the secondary CPUs don't support BBML2 once cpufeature is finalized on all CPUs. If the boot CPU doesn't support BBML2 or the secondary CPUs have the same BBML2 capability with the boot CPU, repainting the linear mapping is not needed. Signed-off-by: Yang Shi --- arch/arm64/include/asm/mmu.h | 3 + arch/arm64/kernel/cpufeature.c | 16 +++++ arch/arm64/mm/mmu.c | 108 ++++++++++++++++++++++++++++++++- arch/arm64/mm/proc.S | 41 +++++++++++++ 4 files changed, 166 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 2693d63bf837..ad38135d1aa1 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -56,6 +56,8 @@ typedef struct { */ #define ASID(mm) (atomic64_read(&(mm)->context.id) & 0xffff) =20 +extern bool block_mapping; + static inline bool arm64_kernel_unmapped_at_el0(void) { return alternative_has_cap_unlikely(ARM64_UNMAP_KERNEL_AT_EL0); @@ -72,6 +74,7 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys= _addr_t phys, extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t pro= t); extern void mark_linear_text_alias_ro(void); extern int split_linear_mapping(unsigned long start, unsigned long end); +extern int __repaint_linear_mappings(void *__unused); =20 /* * This check is triggered during the early boot before the cpufeature diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 5fc2a4a804de..5151c101fbaf 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -2005,6 +2006,20 @@ static int __init __kpti_install_ng_mappings(void *_= _unused) return 0; } =20 +static void __init repaint_linear_mappings(void) +{ + if (!block_mapping) + return; + + if (!rodata_full) + return; + + if (system_supports_bbml2_noabort()) + return; + + stop_machine(__repaint_linear_mappings, NULL, cpu_online_mask); +} + static void __init kpti_install_ng_mappings(void) { /* Check whether KPTI is going to be used */ @@ -3868,6 +3883,7 @@ void __init setup_system_features(void) { setup_system_capabilities(); =20 + repaint_linear_mappings(); kpti_install_ng_mappings(); =20 sve_setup(); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 4c5d3aa35d62..3922af89abbb 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -209,6 +209,8 @@ static void split_pmd(pmd_t pmd, phys_addr_t pte_phys, = int flags) /* It must be naturally aligned if PMD is leaf */ if ((flags & NO_CONT_MAPPINGS) =3D=3D 0) prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + else + prot =3D __pgprot(pgprot_val(prot) & ~PTE_CONT); =20 for (int i =3D 0; i < PTRS_PER_PTE; i++, ptep++, pfn++) __set_pte_nosync(ptep, pfn_pte(pfn, prot)); @@ -230,6 +232,8 @@ static void split_pud(pud_t pud, phys_addr_t pmd_phys, = int flags) /* It must be naturally aligned if PUD is leaf */ if ((flags & NO_CONT_MAPPINGS) =3D=3D 0) prot =3D __pgprot(pgprot_val(prot) | PTE_CONT); + else + prot =3D __pgprot(pgprot_val(prot) & ~PTE_CONT); =20 for (int i =3D 0; i < PTRS_PER_PMD; i++, pmdp++) { __set_pmd_nosync(pmdp, pfn_pmd(pfn, prot)); @@ -833,6 +837,86 @@ void __init mark_linear_text_alias_ro(void) PAGE_KERNEL_RO); } =20 +static phys_addr_t repaint_pgtable_alloc(int shift) +{ + void *ptr; + + ptr =3D (void *)__get_free_page(GFP_ATOMIC); + if (!ptr) + return -1; + + return __pa(ptr); +} + +extern u32 repaint_done; + +int __init __repaint_linear_mappings(void *__unused) +{ + typedef void (repaint_wait_fn)(void); + extern repaint_wait_fn bbml2_wait_for_repainting; + repaint_wait_fn *wait_fn; + + phys_addr_t kernel_start =3D __pa_symbol(_stext); + phys_addr_t kernel_end =3D __pa_symbol(__init_begin); + phys_addr_t start, end; + unsigned long vstart, vend; + u64 i; + int ret; + int flags =3D NO_EXEC_MAPPINGS | NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS | + SPLIT_MAPPINGS; + int cpu =3D smp_processor_id(); + + wait_fn =3D (void *)__pa_symbol(bbml2_wait_for_repainting); + + /* + * Repainting just can be run on CPU 0 because we just can be sure + * CPU 0 supports BBML2. + */ + if (!cpu) { + /* + * Wait for all secondary CPUs get prepared for repainting + * the linear mapping. + */ +wait_for_secondary: + if (READ_ONCE(repaint_done) !=3D num_online_cpus()) + goto wait_for_secondary; + + memblock_mark_nomap(kernel_start, kernel_end - kernel_start); + /* Split the whole linear mapping */ + for_each_mem_range(i, &start, &end) { + if (start >=3D end) + return -EINVAL; + + vstart =3D __phys_to_virt(start); + vend =3D __phys_to_virt(end); + ret =3D __create_pgd_mapping_locked(init_mm.pgd, start, + vstart, (end - start), __pgprot(0), + repaint_pgtable_alloc, flags); + if (ret) + panic("Failed to split linear mappings\n"); + + flush_tlb_kernel_range(vstart, vend); + } + memblock_clear_nomap(kernel_start, kernel_end - kernel_start); + + WRITE_ONCE(repaint_done, 0); + } else { + /* + * The secondary CPUs can't run in the same address space + * with CPU 0 because accessing the linear mapping address + * when CPU 0 is repainting it is not safe. + * + * Let the secondary CPUs run busy loop in idmap address + * space when repainting is ongoing. + */ + cpu_install_idmap(); + wait_fn(); + cpu_uninstall_idmap(); + } + + return 0; +} + #ifdef CONFIG_KFENCE =20 bool __ro_after_init kfence_early_init =3D !!CONFIG_KFENCE_SAMPLE_INTERVAL; @@ -887,6 +971,8 @@ static inline void arm64_kfence_map_pool(phys_addr_t kf= ence_pool, pgd_t *pgdp) { =20 #endif /* CONFIG_KFENCE */ =20 +bool block_mapping; + static inline bool force_pte_mapping(void) { /* @@ -915,6 +1001,8 @@ static void __init map_mem(pgd_t *pgdp) int flags =3D NO_EXEC_MAPPINGS; u64 i; =20 + block_mapping =3D true; + /* * Setting hierarchical PXNTable attributes on table entries covering * the linear region is only possible if it is guaranteed that no table @@ -930,8 +1018,10 @@ static void __init map_mem(pgd_t *pgdp) =20 early_kfence_pool =3D arm64_kfence_alloc_pool(); =20 - if (force_pte_mapping()) + if (force_pte_mapping()) { + block_mapping =3D false; flags |=3D NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; + } =20 /* * Take care not to create a writable alias for the @@ -1063,7 +1153,8 @@ void __pi_map_range(u64 *pgd, u64 start, u64 end, u64= pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); =20 static u8 idmap_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __r= o_after_init, - kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_= init; + kpti_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after_= init, + bbml2_ptes[IDMAP_LEVELS - 1][PAGE_SIZE] __aligned(PAGE_SIZE) __ro_after= _init; =20 static void __init create_idmap(void) { @@ -1088,6 +1179,19 @@ static void __init create_idmap(void) IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, __phys_to_virt(ptep) - ptep); } + + /* + * Setup idmap mapping for repaint_done flag. It will be used if + * repainting the linear mapping is needed later. + */ + if (block_mapping) { + u64 pa =3D __pa_symbol(&repaint_done); + ptep =3D __pa_symbol(bbml2_ptes); + + __pi_map_range(&ptep, pa, pa + sizeof(u32), pa, PAGE_KERNEL, + IDMAP_ROOT_LEVEL, (pte_t *)idmap_pg_dir, false, + __phys_to_virt(ptep) - ptep); + } } =20 void __init paging_init(void) diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index fb30c8804f87..c40e6126c093 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -440,6 +440,47 @@ SYM_FUNC_END(idmap_kpti_install_ng_mappings) .popsection #endif =20 +/* + * Wait for repainting is done. Run on secondary CPUs + * only. + */ + .pushsection ".data", "aw", %progbits +SYM_DATA(repaint_done, .long 1) + .popsection + + .pushsection ".idmap.text", "a" +SYM_TYPED_FUNC_START(bbml2_wait_for_repainting) + swapper_ttb .req x0 + flag_ptr .req x1 + + mrs swapper_ttb, ttbr1_el1 + adr_l flag_ptr, repaint_done + + /* Uninstall swapper before surgery begins */ + __idmap_cpu_set_reserved_ttbr1 x16, x17 + + /* Increment the flag to let the boot CPU we're ready */ +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] + cbnz w17, 1b + + /* Wait for the boot CPU to finish repainting */ + sevl +1: wfe + ldxr w16, [flag_ptr] + cbnz w16, 1b + + /* All done, act like nothing happened */ + msr ttbr1_el1, swapper_ttb + isb + ret + + .unreq swapper_ttb + .unreq flag_ptr +SYM_FUNC_END(bbml2_wait_for_repainting) + .popsection + /* * __cpu_setup * --=20 2.48.1