From nobody Fri Dec 19 19:04:48 2025 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2068.outbound.protection.outlook.com [40.107.95.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C01D20371E for ; Wed, 8 Jan 2025 20:25:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.68 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736367956; cv=fail; b=s4JnUEZfmhh2SxKeQhdHIwxtLZeOp8iAwNr+PtwXObT2IGTelEEVuFyXa8JGubSp9jpIyvZl2WbHCqLr0LpOaMTPbCMnw+Ijp/dSeF1xI+yzZZ6322bx5GLs/Lc/x91gp9LRnya/M/XY8LVGep+J7R9FClyY26vlxNm2rfgmEGc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736367956; c=relaxed/simple; bh=tanGygiVgOBjaLjNZw96wp0lRwp6ANls7ltUNnAiilg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=L33bFqR0CGxuj0mRme/qek19X/uZ5SORaq2NyB+6GtpF5mlbmv37uFRxHG675oRK/iDqeDb+G9UCgmwM6QwlUZF/999PDKDVSRTDv2bO73n10nB3NgNmHGEJQjMm0H0Ou2hEBDNv9VetW4ZezA3refmKpxryuxqLpCEvXCK5qBw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=JF7rvzVn; arc=fail smtp.client-ip=40.107.95.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="JF7rvzVn" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nuAP7jZKPSkgp4Oca8QD/ZtMIS4jzv4a2oslwFI/zVve8KLAna4jL/ME3EaNmv8du6cLwpArKqQB8siL9ID6+IEYnCiUQC4q5s0i86e1b3wMCmheFVKMLxQhp0DbcB80vMYDHlGW6LqXz53UuYNBf1tPLLSgFHQoSPz/fAp92GnvsbXPA4iqk25YAGQjiy4DJmFRXjT5BXLErdghvFWwOkQQmYSMUuk/HqX8oBmmzkFlYKGkg/03XxYcYuNLyc3/feqZn/bcTb3L1OIDVPAU47mk9G7rVVmP1ZW0dha+FJkhtfnryOrkBtA+iUbB4RXqWT4+QDPFInooAZZ4CiFQQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Whd+Q/b7Y/Ez9j46JZT7Sw33QoCr1QMINHHDlEsYcvE=; b=SYvCnyoJY/KXrjEc6fPPSScMz3OWtO94beCeBFdOwKb+GBATL48RmX0EHjePbClfO6jszSyLs/AppjJFSg9vKt+HFyAQlrsGePxT7HoH+wwueKe3tWEQhzbpTeSZ4ZpBaDLlTGXl2gmUtDyZbpHWz4xXtLNVLfuo5Bk1lsIBqacYhlgqm9Voau3DG1z1GHbxI+0oBkR9IkSFIFXjR3CAnRCFimTJou3q6VvYAp4s3GoxU9Sq6OOHqDct57A9p+ZD1ag5RvSLBku8Um6RYZwFNjCT1Z6o1uQcH38dx3ypH1l7DpFXD0wAbS60v+b9PDX0OOvja0E708xVCqB6PTQR7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linutronix.de smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Whd+Q/b7Y/Ez9j46JZT7Sw33QoCr1QMINHHDlEsYcvE=; b=JF7rvzVnTTdLRq/1YFXk0MMzbHgNn4h8trVvJpO7i9U6ez/K+177EAN7/qj+IoZipqtVhLNmiHpf40oAxgCdTU70sCesS+fjolFBsklb9SkE1DA1DNwjiqgsRXN8r+2Obc+EBDx8mpLAmNJ4ocMsdpJF7T7ibcp82hx1++oDLYs= Received: from SA9PR11CA0019.namprd11.prod.outlook.com (2603:10b6:806:6e::24) by IA1PR12MB8496.namprd12.prod.outlook.com (2603:10b6:208:446::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.18; Wed, 8 Jan 2025 20:25:47 +0000 Received: from DS3PEPF000099E2.namprd04.prod.outlook.com (2603:10b6:806:6e:cafe::95) by SA9PR11CA0019.outlook.office365.com (2603:10b6:806:6e::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8335.11 via Frontend Transport; Wed, 8 Jan 2025 20:25:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099E2.mail.protection.outlook.com (10.167.17.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8335.7 via Frontend Transport; Wed, 8 Jan 2025 20:25:46 +0000 Received: from tiny.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 8 Jan 2025 14:25:45 -0600 From: David Kaplan To: Thomas Gleixner , Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Pawan Gupta , Ingo Molnar , Dave Hansen , , "H . Peter Anvin" CC: Subject: [PATCH v3 18/35] x86/bugs: Restructure srso mitigation Date: Wed, 8 Jan 2025 14:24:58 -0600 Message-ID: <20250108202515.385902-19-david.kaplan@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250108202515.385902-1-david.kaplan@amd.com> References: <20250108202515.385902-1-david.kaplan@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099E2:EE_|IA1PR12MB8496:EE_ X-MS-Office365-Filtering-Correlation-Id: 847f98c9-a941-4df1-8673-08dd3022a272 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?HYkhV35k02COKLzZ2D15kHNgx8Y3CUZHpWw1Q6W7iKHJ5FaRMa1vB67jtKvq?= =?us-ascii?Q?gEF+OP5JN0b/LVvQbprV/7nAHUMG571ESHC5kVutoicpDcrZROKgkTfYxwoZ?= =?us-ascii?Q?WepQqD2VS3fcT8k82QwCafVGLCRkjbFo3oZCef14Pe6bRsw0T7y62QXZsIKt?= =?us-ascii?Q?mwRMaodw9KLBOrrgre7/WWlFy+iY2YAboN8AMYax4PlzuiNM/2hjKc2Oy5Zf?= =?us-ascii?Q?HPc5zXksVxmj/js6ALP0MxywAwfgPLWhz2Z/DxXDkyrcEhfcGtpwoB3FkKsx?= =?us-ascii?Q?XkbovxmHpdIFZnY9orNeYL9Zyv50g19zLJd9kU8k+3kvi3S2sxVei11SauIg?= =?us-ascii?Q?YK4rA81v6uCd6JXTZCcpPOYKTPBLMSLjv2+RjhSeBEErAOp74SFaHRy9LXyQ?= =?us-ascii?Q?C3S45v+Uit/mggA6Y7oqROs91uVd/k7l3x7kzz3+bIjATLxppaLtFxIEqK5n?= =?us-ascii?Q?NZbMoNXdPrSfy8deROiRiu2vYnFnSaAMMDQiQGzPh93C96EcoezOTb020ptO?= =?us-ascii?Q?yJbTrW57Qkl5ML1xgw47Li1D3GD+vYO3c+AiiFljxeDUnTxHrEP4lxCC29RY?= =?us-ascii?Q?Hm7zYJHyyXgu3SEXKvSsHJQLbr/iJ2ykbxjMyKt+gGly/7i6XVgHYcGtcFh6?= =?us-ascii?Q?Pkcf5IVuSRgMyX5T19TgTg0LPaSlmhUyrbTeeeyPoARCEFTdKl7j7I4JfDGu?= =?us-ascii?Q?e09UBBraczm/mkkPCQR/oNyYHERuNHaGAcwr+Lb7k4hNL/DI76u6uCB1AiZ9?= =?us-ascii?Q?Cr5HPkSykgtqZ64NeV2qLgRLsCHbF4JTrNdxgcWUR4+T4+xOgejdiIdEbZ+0?= =?us-ascii?Q?edUZ9MTpKd0AuyuZ8hMvP/o/fQtQkfCDp/LVEgxT00wKOm+oZyqfNEu4YyuI?= =?us-ascii?Q?GG4wYynDDPIrRhnpiipli3tocRj1vNpZ4o+nrnGUo7DV9+x8x/VWQKPJlYjs?= =?us-ascii?Q?ejSsuW7vKBUlr1ENJdruWVqyxI3M730qEoS6tcfjCoxeuR3RI9hIxAMG3KYx?= =?us-ascii?Q?MQ3qqJMEB66cpRJ7IsiLZ5TUBlnpemBqDUlFZvrZR/OEM0zgXQMqTMlgKh07?= =?us-ascii?Q?QWGHfELqB/maJ2e/QHHTNcLukdRGdX+PhTJI40xnOG08PBPq+lBtM4UZC832?= =?us-ascii?Q?0yRj2MWq0gxwGPKsqy4PMW1mslj8RKnbtUjmBsNEQjqPYQi7YdEnri86Z2xk?= =?us-ascii?Q?RGqFImEX15ztysSUSUGVIGS7IaofnCHisTyKMU7ZCgLu40FqYVF5Uvvkvtg0?= =?us-ascii?Q?B674Zg7eK5U6qX4In4wCYspf0ABbq3wi8ModkzJWKQA0WiVqTYUtE2fGdlkP?= =?us-ascii?Q?7eYavGpQ/rBqaBy6Aw1RelR67aVQ6WJ+p3lvS5y51K+GndhdqgmacxhqAqjX?= =?us-ascii?Q?NFSFZeYHhYA8J0x0WBsfM0KhsP8FYg/7UAQAqbcDxamwO2EtDAtjCBIyVMqU?= =?us-ascii?Q?7k3DKogni2LhIqneHA1eNfhzx74H45wTAdYo9MA2Lncfufc66QoDL1wke23e?= =?us-ascii?Q?9ObS734dJK/bCnU=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2025 20:25:46.7820 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 847f98c9-a941-4df1-8673-08dd3022a272 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099E2.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8496 Content-Type: text/plain; charset="utf-8" Restructure srso to use select/update/apply functions to create consistent vulnerability handling. Like with retbleed, the command line options directly select mitigations which can later be modified. Signed-off-by: David Kaplan --- arch/x86/kernel/cpu/bugs.c | 188 ++++++++++++++++++------------------- 1 file changed, 90 insertions(+), 98 deletions(-) diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 08ac515df888..aee2945bdef9 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -84,6 +84,8 @@ static void __init srbds_select_mitigation(void); static void __init srbds_apply_mitigation(void); static void __init l1d_flush_select_mitigation(void); static void __init srso_select_mitigation(void); +static void __init srso_update_mitigation(void); +static void __init srso_apply_mitigation(void); static void __init gds_select_mitigation(void); static void __init gds_apply_mitigation(void); static void __init bhi_select_mitigation(void); @@ -200,11 +202,6 @@ void __init cpu_select_mitigations(void) rfds_select_mitigation(); srbds_select_mitigation(); l1d_flush_select_mitigation(); - - /* - * srso_select_mitigation() depends and must run after - * retbleed_select_mitigation(). - */ srso_select_mitigation(); gds_select_mitigation(); bhi_select_mitigation(); @@ -220,6 +217,7 @@ void __init cpu_select_mitigations(void) taa_update_mitigation(); mmio_update_mitigation(); rfds_update_mitigation(); + srso_update_mitigation(); =20 spectre_v1_apply_mitigation(); spectre_v2_apply_mitigation(); @@ -232,6 +230,7 @@ void __init cpu_select_mitigations(void) mmio_apply_mitigation(); rfds_apply_mitigation(); srbds_apply_mitigation(); + srso_apply_mitigation(); gds_apply_mitigation(); bhi_apply_mitigation(); } @@ -2673,6 +2672,7 @@ early_param("l1tf", l1tf_cmdline); =20 enum srso_mitigation { SRSO_MITIGATION_NONE, + SRSO_MITIGATION_AUTO, SRSO_MITIGATION_UCODE_NEEDED, SRSO_MITIGATION_SAFE_RET_UCODE_NEEDED, SRSO_MITIGATION_MICROCODE, @@ -2681,14 +2681,6 @@ enum srso_mitigation { SRSO_MITIGATION_IBPB_ON_VMEXIT, }; =20 -enum srso_mitigation_cmd { - SRSO_CMD_OFF, - SRSO_CMD_MICROCODE, - SRSO_CMD_SAFE_RET, - SRSO_CMD_IBPB, - SRSO_CMD_IBPB_ON_VMEXIT, -}; - static const char * const srso_strings[] =3D { [SRSO_MITIGATION_NONE] =3D "Vulnerable", [SRSO_MITIGATION_UCODE_NEEDED] =3D "Vulnerable: No microcode", @@ -2699,8 +2691,7 @@ static const char * const srso_strings[] =3D { [SRSO_MITIGATION_IBPB_ON_VMEXIT] =3D "Mitigation: IBPB on VMEXIT only" }; =20 -static enum srso_mitigation srso_mitigation __ro_after_init =3D SRSO_MITIG= ATION_NONE; -static enum srso_mitigation_cmd srso_cmd __ro_after_init =3D SRSO_CMD_SAFE= _RET; +static enum srso_mitigation srso_mitigation __ro_after_init =3D SRSO_MITIG= ATION_AUTO; =20 static int __init srso_parse_cmdline(char *str) { @@ -2708,15 +2699,15 @@ static int __init srso_parse_cmdline(char *str) return -EINVAL; =20 if (!strcmp(str, "off")) - srso_cmd =3D SRSO_CMD_OFF; + srso_mitigation =3D SRSO_MITIGATION_NONE; else if (!strcmp(str, "microcode")) - srso_cmd =3D SRSO_CMD_MICROCODE; + srso_mitigation =3D SRSO_MITIGATION_MICROCODE; else if (!strcmp(str, "safe-ret")) - srso_cmd =3D SRSO_CMD_SAFE_RET; + srso_mitigation =3D SRSO_MITIGATION_SAFE_RET; else if (!strcmp(str, "ibpb")) - srso_cmd =3D SRSO_CMD_IBPB; + srso_mitigation =3D SRSO_MITIGATION_IBPB; else if (!strcmp(str, "ibpb-vmexit")) - srso_cmd =3D SRSO_CMD_IBPB_ON_VMEXIT; + srso_mitigation =3D SRSO_MITIGATION_IBPB_ON_VMEXIT; else pr_err("Ignoring unknown SRSO option (%s).", str); =20 @@ -2730,13 +2721,14 @@ static void __init srso_select_mitigation(void) { bool has_microcode =3D boot_cpu_has(X86_FEATURE_IBPB_BRTYPE); =20 - if (!boot_cpu_has_bug(X86_BUG_SRSO) || - cpu_mitigations_off() || - srso_cmd =3D=3D SRSO_CMD_OFF) { - if (boot_cpu_has(X86_FEATURE_SBPB)) - x86_pred_cmd =3D PRED_CMD_SBPB; + if (!boot_cpu_has_bug(X86_BUG_SRSO) || cpu_mitigations_off()) + srso_mitigation =3D SRSO_MITIGATION_NONE; + + if (srso_mitigation =3D=3D SRSO_MITIGATION_NONE) return; - } + + if (srso_mitigation =3D=3D SRSO_MITIGATION_AUTO) + srso_mitigation =3D SRSO_MITIGATION_SAFE_RET; =20 if (has_microcode) { /* @@ -2749,98 +2741,98 @@ static void __init srso_select_mitigation(void) setup_force_cpu_cap(X86_FEATURE_SRSO_NO); return; } - - if (retbleed_mitigation =3D=3D RETBLEED_MITIGATION_IBPB) { - srso_mitigation =3D SRSO_MITIGATION_IBPB; - goto out; - } } else { pr_warn("IBPB-extending microcode not applied!\n"); pr_warn(SRSO_NOTICE); =20 - /* may be overwritten by SRSO_CMD_SAFE_RET below */ - srso_mitigation =3D SRSO_MITIGATION_UCODE_NEEDED; + /* Fall-back to Safe-RET */ + srso_mitigation =3D SRSO_MITIGATION_SAFE_RET_UCODE_NEEDED; } =20 - switch (srso_cmd) { - case SRSO_CMD_MICROCODE: - if (has_microcode) { - srso_mitigation =3D SRSO_MITIGATION_MICROCODE; - pr_warn(SRSO_NOTICE); - } + switch (srso_mitigation) { + case SRSO_MITIGATION_MICROCODE: break; =20 - case SRSO_CMD_SAFE_RET: - if (boot_cpu_has(X86_FEATURE_SRSO_USER_KERNEL_NO)) - goto ibpb_on_vmexit; - - if (IS_ENABLED(CONFIG_MITIGATION_SRSO)) { - /* - * Enable the return thunk for generated code - * like ftrace, static_call, etc. - */ - setup_force_cpu_cap(X86_FEATURE_RETHUNK); - setup_force_cpu_cap(X86_FEATURE_UNRET); - - if (boot_cpu_data.x86 =3D=3D 0x19) { - setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); - x86_return_thunk =3D srso_alias_return_thunk; - } else { - setup_force_cpu_cap(X86_FEATURE_SRSO); - x86_return_thunk =3D srso_return_thunk; - } - if (has_microcode) - srso_mitigation =3D SRSO_MITIGATION_SAFE_RET; - else - srso_mitigation =3D SRSO_MITIGATION_SAFE_RET_UCODE_NEEDED; - } else { + case SRSO_MITIGATION_SAFE_RET: + case SRSO_MITIGATION_SAFE_RET_UCODE_NEEDED: + if (!IS_ENABLED(CONFIG_MITIGATION_SRSO)) pr_err("WARNING: kernel not compiled with MITIGATION_SRSO.\n"); - } + else if (boot_cpu_has(X86_FEATURE_SRSO_USER_KERNEL_NO)) + srso_mitigation =3D SRSO_MITIGATION_IBPB_ON_VMEXIT; break; =20 - case SRSO_CMD_IBPB: - if (IS_ENABLED(CONFIG_MITIGATION_IBPB_ENTRY)) { - if (has_microcode) { - setup_force_cpu_cap(X86_FEATURE_ENTRY_IBPB); - srso_mitigation =3D SRSO_MITIGATION_IBPB; - - /* - * IBPB on entry already obviates the need for - * software-based untraining so clear those in case some - * other mitigation like Retbleed has selected them. - */ - setup_clear_cpu_cap(X86_FEATURE_UNRET); - setup_clear_cpu_cap(X86_FEATURE_RETHUNK); - } - } else { + case SRSO_MITIGATION_IBPB: + if (!IS_ENABLED(CONFIG_MITIGATION_IBPB_ENTRY)) pr_err("WARNING: kernel not compiled with MITIGATION_IBPB_ENTRY.\n"); - } break; =20 -ibpb_on_vmexit: - case SRSO_CMD_IBPB_ON_VMEXIT: - if (IS_ENABLED(CONFIG_MITIGATION_SRSO)) { - if (!boot_cpu_has(X86_FEATURE_ENTRY_IBPB) && has_microcode) { - setup_force_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT); - srso_mitigation =3D SRSO_MITIGATION_IBPB_ON_VMEXIT; - - /* - * There is no need for RSB filling: entry_ibpb() ensures - * all predictions, including the RSB, are invalidated, - * regardless of IBPB implementation. - */ - setup_clear_cpu_cap(X86_FEATURE_RSB_VMEXIT); - } - } else { + case SRSO_MITIGATION_IBPB_ON_VMEXIT: + if (!IS_ENABLED(CONFIG_MITIGATION_SRSO)) pr_err("WARNING: kernel not compiled with MITIGATION_SRSO.\n"); - } + break; + default: + break; + } +} + +static void __init srso_update_mitigation(void) +{ + /* If retbleed is using IBPB, that works for SRSO as well */ + if (retbleed_mitigation =3D=3D RETBLEED_MITIGATION_IBPB) + srso_mitigation =3D SRSO_MITIGATION_IBPB; + + if (srso_mitigation !=3D SRSO_MITIGATION_NONE) + pr_info("%s\n", srso_strings[srso_mitigation]); +} + +static void __init srso_apply_mitigation(void) +{ + if (srso_mitigation =3D=3D SRSO_MITIGATION_NONE) { + if (boot_cpu_has(X86_FEATURE_SBPB)) + x86_pred_cmd =3D PRED_CMD_SBPB; + return; + } + switch (srso_mitigation) { + case SRSO_MITIGATION_SAFE_RET: + case SRSO_MITIGATION_SAFE_RET_UCODE_NEEDED: + /* + * Enable the return thunk for generated code + * like ftrace, static_call, etc. + */ + setup_force_cpu_cap(X86_FEATURE_RETHUNK); + setup_force_cpu_cap(X86_FEATURE_UNRET); + + if (boot_cpu_data.x86 =3D=3D 0x19) { + setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); + x86_return_thunk =3D srso_alias_return_thunk; + } else { + setup_force_cpu_cap(X86_FEATURE_SRSO); + x86_return_thunk =3D srso_return_thunk; + } + break; + case SRSO_MITIGATION_IBPB: + setup_force_cpu_cap(X86_FEATURE_ENTRY_IBPB); + /* + * IBPB on entry already obviates the need for + * software-based untraining so clear those in case some + * other mitigation like Retbleed has selected them. + */ + setup_clear_cpu_cap(X86_FEATURE_UNRET); + setup_clear_cpu_cap(X86_FEATURE_RETHUNK); + break; + case SRSO_MITIGATION_IBPB_ON_VMEXIT: + setup_force_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT); + /* + * There is no need for RSB filling: entry_ibpb() ensures + * all predictions, including the RSB, are invalidated, + * regardless of IBPB implementation. + */ + setup_clear_cpu_cap(X86_FEATURE_RSB_VMEXIT); break; default: break; } =20 -out: - pr_info("%s\n", srso_strings[srso_mitigation]); } =20 #undef pr_fmt --=20 2.34.1