From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8832C07E8E for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234701AbjHNMUO (ORCPT ); Mon, 14 Aug 2023 08:20:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232533AbjHNMTi (ORCPT ); Mon, 14 Aug 2023 08:19:38 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 157B818F for ; Mon, 14 Aug 2023 05:19:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=1dQK7+y7wnp9bbFjSoE9Iy4vHQneof7CDIZ0OzqGKqo=; b=EmAL5xkzB4R4Wn54xCVDxw3dYJ Cw+bD4PkfUiVSxEhEJ2xpv2l0ajq5oN1kJMNFEiEj6yZQi6RXkjR2NYaAkKw5cuPL0UVsSNulyL08 Fgs3aTm5wWEBKA7RKmixSK2VCYIeYrAlMakjfnJv7vanbiJxPjFvNjg4tIR//OvWQQaHW7ZxTLJb+ yy8LXCWZIdvfLk0917bJyR71+0sxHW4N0C9iIeylLvBYioXWvtopOvnNGyf8TuRe6wDDmSYIfCmwG SU9VV7SamP+Prh+2eZmF63CEc/4Hu4uMCZDwHIX5cOjLzgGLIHHk41AWxRQK1QLSTFxPbbYjvdsxT woIZvXwg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVWXY-001t43-A1; Mon, 14 Aug 2023 12:19:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id C8C7F3004AF; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B0D7921C1D1B2; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121148.571027074@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:27 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 01/11] x86/cpu: Fixup __x86_return_thunk References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Patch fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation") reimplemented __x86_return_thunk with a mix of SYM_FUNC_START and SYM_CODE_END, this is not a sane combination. Since nothing should ever actually 'CALL' this, make it consistently CODE. Fixes: fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation") Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/lib/retpoline.S | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -264,7 +264,9 @@ SYM_CODE_END(srso_safe_ret) SYM_FUNC_END(srso_untrain_ret) __EXPORT_THUNK(srso_untrain_ret) =20 -SYM_FUNC_START(__x86_return_thunk) +SYM_CODE_START(__x86_return_thunk) + UNWIND_HINT_FUNC + ANNOTATE_NOENDBR ALTERNATIVE_2 "jmp __ret", "call srso_safe_ret", X86_FEATURE_SRSO, \ "call srso_safe_ret_alias", X86_FEATURE_SRSO_ALIAS int3 From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9474FC0729B for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234214AbjHNMUK (ORCPT ); Mon, 14 Aug 2023 08:20:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233940AbjHNMTe (ORCPT ); Mon, 14 Aug 2023 08:19:34 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E681910F4 for ; Mon, 14 Aug 2023 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=UW8swBZKEShmVwzbkW/tIP2uQ5C31tN5ZsH9IAoW9A8=; b=TzWAzfoGgbzh3Rm3v0WnarL5Wb bGs0mnFhkKcvmw+DdE3Qp6GUYX6CNTyQT2QjKvZr6W9+mJYwUYsF/PCcMDQGaw8lPMXMSz8MsJJEb 1tOL63HgJFmbicRrFFs/y9+HdnSwCGG/G1vx8z9mazau8ccqNgSxVletEkL5mbudSEg9QYwGzmBIf LKSA/nbxpc3nyFzqSOionVx+Hnnuxq/GHKMNt9oV+LMFxPY2hzGT9TFdDTuftp8KPQub2cHqXqv2V hHTSlpHAJU38WkKGaZDfu5/oQcLlqkkxOcAy8p8vtFqZXYOIYNhHwI3nY09J/qzO6ln2p8kv5Vl8z y74lTWdw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qVWXY-00AaKX-0y; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id CCA5E300599; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B50BC21C1D1B3; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121148.637802730@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:28 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 02/11] x86/cpu: Fix up srso_safe_ret() and __x86_return_thunk() References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vmlinux.o: warning: objtool: srso_untrain_ret() falls through to next funct= ion __x86_return_skl() vmlinux.o: warning: objtool: __x86_return_thunk() falls through to next fun= ction __x86_return_skl() This is because these functions (can) end with CALL, which objtool does not consider a terminating instruction. Therefore replace the INT3 instruction (which is a non-fatal trap) with UD2 (which is a fatal-trap). This indicates execution will not continue past this point. Fixes: fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation") Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/lib/retpoline.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -259,7 +259,7 @@ SYM_INNER_LABEL(srso_safe_ret, SYM_L_GLO int3 lfence call srso_safe_ret - int3 + ud2 SYM_CODE_END(srso_safe_ret) SYM_FUNC_END(srso_untrain_ret) __EXPORT_THUNK(srso_untrain_ret) @@ -269,7 +269,7 @@ SYM_CORE_START(__x86_return_thunk) ANNOTATE_NOENDBR ALTERNATIVE_2 "jmp __ret", "call srso_safe_ret", X86_FEATURE_SRSO, \ "call srso_safe_ret_alias", X86_FEATURE_SRSO_ALIAS - int3 + ud2 SYM_CODE_END(__x86_return_thunk) EXPORT_SYMBOL(__x86_return_thunk) From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C2C8C04E69 for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232770AbjHNMUA (ORCPT ); Mon, 14 Aug 2023 08:20:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231527AbjHNMTc (ORCPT ); Mon, 14 Aug 2023 08:19:32 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2196E18B for ; Mon, 14 Aug 2023 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=MSbD0mrcSH8dRdmwVMp2ZpkoVj+73cEbOOROFTSZaOE=; b=Xc+7xuyes5BTnEkvrWq6InpvIq /prorkXQbTxexs20t3eoQnFS29yNZJHIJ3jMndXHLqmOABHqCSAkjZ6nio6Y2taGz1z+zLTkf9d3r EhPqEKEC61VDJuTm/PFMSWoB+m/PX2Sh0ei+VWOSXecg9PejGXkUpbdwFV20Ywcyd0nxVE2FHH9az hWPvXI5ymDemSilOCjTtysItSkd3/+wobA6r3Itf9g2h/i3eH/2dVksonDGUDXYrU0Rjip0E6mJB7 5OBryalYhiIH8HncfxWThUUADUx2Y/qRctwvt8IOw0L2KGvWvmbFbUdZN0f/lzY0VnJK9KnGAIcB+ y7tkk2RA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qVWXY-00AaKY-14; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id CF9EE3005E3; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id BA11C20167A97; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121148.704502245@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:29 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 03/11] objtool/x86: Fix SRSO mess References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Objtool --rethunk does two things: - it collects all (tail) call's of __x86_return_thunk and places them into .return_sites. These are typically compiler generated, but RET also emits this same. - it fudges the validation of the __x86_return_thunk symbol; because this symbol is inside another instruction, it can't actually find the instruction pointed to by the symbol offset and gets upset. Because these two things pertained to the same symbol, there was no pressing need to separate these two separate things. However, alas, along comes SRSO and we get more crazy things to deal with. The SRSO patch itself added the following symbol names to identify as rethunk: 'srso_untrain_ret', 'srso_safe_ret' and '__ret' Where '__ret' is the old retbleed return thunk, 'srso_safe_ret' is a new similarly embedded return thunk, and 'srso_untrain_ret' is completely unrelated to anything the above does (and was only included because of that INT3 vs UD2 issue fixed previous). Clear things up by adding a second category for the embedded instruction thing. Fixes: fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation") Signed-off-by: Peter Zijlstra (Intel) --- tools/objtool/arch/x86/decode.c | 11 +++++++---- tools/objtool/check.c | 24 ++++++++++++++++++++++-- tools/objtool/include/objtool/arch.h | 1 + tools/objtool/include/objtool/elf.h | 1 + 4 files changed, 31 insertions(+), 6 deletions(-) --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -824,8 +824,11 @@ bool arch_is_retpoline(struct symbol *sy =20 bool arch_is_rethunk(struct symbol *sym) { - return !strcmp(sym->name, "__x86_return_thunk") || - !strcmp(sym->name, "srso_untrain_ret") || - !strcmp(sym->name, "srso_safe_ret") || - !strcmp(sym->name, "__ret"); + return !strcmp(sym->name, "__x86_return_thunk"); +} + +bool arch_is_embedded_insn(struct symbol *sym) +{ + return !strcmp(sym->name, "__ret") || + !strcmp(sym->name, "srso_safe_ret"); } --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -455,7 +455,7 @@ static int decode_instructions(struct ob return -1; } =20 - if (func->return_thunk || func->alias !=3D func) + if (func->embedded_insn || func->alias !=3D func) continue; =20 if (!find_insn(file, sec, func->offset)) { @@ -1288,16 +1288,33 @@ static int add_ignore_alternatives(struc return 0; } =20 +/* + * Symbols that replace INSN_CALL_DYNAMIC, every (tail) call to such a sym= bol + * will be added to the .retpoline_sites section. + */ __weak bool arch_is_retpoline(struct symbol *sym) { return false; } =20 +/* + * Symbols that replace INSN_RETURN, every (tail) call to such a symbol + * will be added to the .return_sites section. + */ __weak bool arch_is_rethunk(struct symbol *sym) { return false; } =20 +/* + * Symbols that are embedded inside other instructions, because sometimes = crazy + * code exists. These are mostly ignored for validation purposes. + */ +__weak bool arch_is_embedded_insn(struct symbol *sym) +{ + return false; +} + static struct reloc *insn_reloc(struct objtool_file *file, struct instruct= ion *insn) { struct reloc *reloc; @@ -1583,7 +1600,7 @@ static int add_jump_destinations(struct * middle of another instruction. Objtool only * knows about the outer instruction. */ - if (sym && sym->return_thunk) { + if (sym && sym->embedded_insn) { add_return_call(file, insn, false); continue; } @@ -2502,6 +2519,9 @@ static int classify_symbols(struct objto if (arch_is_rethunk(func)) func->return_thunk =3D true; =20 + if (arch_is_embedded_insn(func)) + func->embedded_insn =3D true; + if (arch_ftrace_match(func->name)) func->fentry =3D true; =20 --- a/tools/objtool/include/objtool/arch.h +++ b/tools/objtool/include/objtool/arch.h @@ -90,6 +90,7 @@ int arch_decode_hint_reg(u8 sp_reg, int =20 bool arch_is_retpoline(struct symbol *sym); bool arch_is_rethunk(struct symbol *sym); +bool arch_is_embedded_insn(struct symbol *sym); =20 int arch_rewrite_retpolines(struct objtool_file *file); =20 --- a/tools/objtool/include/objtool/elf.h +++ b/tools/objtool/include/objtool/elf.h @@ -66,6 +66,7 @@ struct symbol { u8 fentry : 1; u8 profiling_func : 1; u8 warned : 1; + u8 embedded_insn : 1; struct list_head pv_target; struct reloc *relocs; }; From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C25A3C07E8B for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234476AbjHNMUN (ORCPT ); Mon, 14 Aug 2023 08:20:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234082AbjHNMTj (ORCPT ); Mon, 14 Aug 2023 08:19:39 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86790F5 for ; Mon, 14 Aug 2023 05:19:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=4YdyLh/jaFymiTs1kquLvymfFdQhLGTp8bGZNgL90Dw=; b=tKK6amFPdJKTLwe2ykNtNstSJF k2tmEmP4HYagy/5XVSUh8PfKrR3gKbUR/9YhsIo464YO8FQEGmq1pzROaNH8UliZRqLBu333dsWgg oDTJU7QC66IRzJ2UJuWbxSuCArNiuptO7SpjyAYIQb/ljVJVYfUOi24qf14QsibDeAh098bHZXi3B y0fiiOOdGwM1szmOIUrhUESFMGT4tODVRBqjKcaY4NVJPTO/aNrar76F+DNARe7PbLaPxa8wwZQRg RDS3XE80CC8sC3X0pvnxpM+n52rEzA+Kxc7gsccyi/Vl76+vy9Ed1MXeAwQZMODWowA9A4ej+0AMZ kTMc/+sg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVWXY-001t45-Bh; Mon, 14 Aug 2023 12:19:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D4A1030075E; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id BEB3121C1D181; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121148.775293785@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:30 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 04/11] x86/alternative: Make custom return thunk unconditional References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is infrastructure to rewrite return thunks to point to any random thunk one desires, unwrap that from CALL_THUNKS, which up to now was the sole user of that. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 4 ---- arch/x86/kernel/alternative.c | 2 -- 2 files changed, 6 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -347,11 +347,7 @@ extern void srso_untrain_ret(void); extern void srso_untrain_ret_alias(void); extern void entry_ibpb(void); =20 -#ifdef CONFIG_CALL_THUNKS extern void (*x86_return_thunk)(void); -#else -#define x86_return_thunk (&__x86_return_thunk) -#endif =20 #ifdef CONFIG_CALL_DEPTH_TRACKING extern void __x86_return_skl(void); --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -687,9 +687,7 @@ void __init_or_module noinline apply_ret =20 #ifdef CONFIG_RETHUNK =20 -#ifdef CONFIG_CALL_THUNKS void (*x86_return_thunk)(void) __ro_after_init =3D &__x86_return_thunk; -#endif =20 /* * Rewrite the compiler generated return thunk tail-calls. From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B8ECC04FE1 for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233289AbjHNMUC (ORCPT ); Mon, 14 Aug 2023 08:20:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233689AbjHNMTc (ORCPT ); Mon, 14 Aug 2023 08:19:32 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07BF0133 for ; Mon, 14 Aug 2023 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=V9fM2o7mhVY5C/pJpd3IZUYsrW+NYZNMXL2ceFzfrn8=; b=NwWe9FkE7X3q+hcfXBWMgR5r8t ezINDHcVB1OaVd5oGEY+P+0vCU5fxn6e6SO2CwOdXM5OHjMJEZToxhvQeCK166rj+0WNTLzjB86Xa dtCumrecatJMivYZJkgvhNBOpRJjpUWYOi8mRKPlLva3nAHAGTqaeUU4e00GJ+eZxKsK11sUQ/9/z 2lnvNbki6pBcjWUPbx1jwBuXaaLLt95S1tSrqOdMJFWcUKrgjQpvywxj4Q1X8GRSQOszGRVM6LB6x pM9BaUCGnVmp2SJi6IxpAltSmB715Fm3HeZ/ux7TF/b/L+yiAVj3+kun1Ozd7kBwn8fLe0va5CVWX Mn/FUObQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qVWXY-00AaKc-38; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E4FB43008CD; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C318721C1D1B4; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121148.842775684@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:31 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 05/11] x86/cpu: Clean up SRSO return thunk mess References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the existing configurable return thunk. There is absolute no justification for having created this __x86_return_thunk alternative. To clarify, the whole thing looks like: Zen3/4 does: srso_alias_untrain_ret: nop2 lfence jmp srso_alias_return_thunk int3 srso_alias_safe_ret: // aliasses srso_alias_untrain_ret just so add $8, %rsp ret int3 srso_alias_return_thunk: call srso_alias_safe_ret ud2 While Zen1/2 does: srso_untrain_ret: movabs $foo, %rax lfence call srso_safe_ret (jmp srso_return_thunk ?) int3 srso_safe_ret: // embedded in movabs instruction add $8,%rsp ret int3 srso_return_thunk: call srso_safe_ret ud2 While retbleed does: zen_untrain_ret: test $0xcc, %bl lfence jmp zen_return_thunk int3 zen_return_thunk: // embedded in the test instruction ret int3 Where Zen1/2 flush the BTB entry using the instruction decoder trick (test,movabs) Zen3/4 use instruction aliasing. SRSO adds RSB (RAP in AMD speak) stuffing to force speculation into a trap an cause a mis-predict. Pick one of three options at boot (evey function can only ever return once). Fixes: fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation") Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 6 ++++ arch/x86/kernel/cpu/bugs.c | 8 ++++-- arch/x86/kernel/vmlinux.lds.S | 2 - arch/x86/lib/retpoline.S | 45 ++++++++++++++++++++++--------= ----- tools/objtool/arch/x86/decode.c | 2 - 5 files changed, 43 insertions(+), 20 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -342,9 +342,15 @@ extern retpoline_thunk_t __x86_indirect_ extern retpoline_thunk_t __x86_indirect_jump_thunk_array[]; =20 extern void __x86_return_thunk(void); + +extern void zen_return_thunk(void); +extern void srso_return_thunk(void); +extern void srso_alias_return_thunk(void); + extern void zen_untrain_ret(void); extern void srso_untrain_ret(void); extern void srso_untrain_ret_alias(void); + extern void entry_ibpb(void); =20 extern void (*x86_return_thunk)(void); --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1034,6 +1034,7 @@ static void __init retbleed_select_mitig case RETBLEED_MITIGATION_UNRET: setup_force_cpu_cap(X86_FEATURE_RETHUNK); setup_force_cpu_cap(X86_FEATURE_UNRET); + x86_return_thunk =3D zen_return_thunk; =20 if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) @@ -2451,10 +2452,13 @@ static void __init srso_select_mitigatio */ setup_force_cpu_cap(X86_FEATURE_RETHUNK); =20 - if (boot_cpu_data.x86 =3D=3D 0x19) + if (boot_cpu_data.x86 =3D=3D 0x19) { setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); - else + x86_return_thunk =3D srso_alias_return_thunk; + } else { setup_force_cpu_cap(X86_FEATURE_SRSO); + x86_return_thunk =3D srso_return_thunk; + } srso_mitigation =3D SRSO_MITIGATION_SAFE_RET; } else { pr_err("WARNING: kernel not compiled with CPU_SRSO.\n"); --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -521,7 +521,7 @@ INIT_PER_CPU(irq_stack_backing_store); #endif =20 #ifdef CONFIG_RETHUNK -. =3D ASSERT((__ret & 0x3f) =3D=3D 0, "__ret not cacheline-aligned"); +. =3D ASSERT((zen_return_thunk & 0x3f) =3D=3D 0, "zen_return_thunk not cac= heline-aligned"); . =3D ASSERT((srso_safe_ret & 0x3f) =3D=3D 0, "srso_safe_ret not cacheline= -aligned"); #endif =20 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -151,22 +151,20 @@ SYM_CODE_END(__x86_indirect_jump_thunk_a .section .text..__x86.rethunk_untrain =20 SYM_START(srso_untrain_ret_alias, SYM_L_GLOBAL, SYM_A_NONE) + UNWIND_HINT_FUNC ANNOTATE_NOENDBR ASM_NOP2 lfence - jmp __x86_return_thunk + jmp srso_alias_return_thunk SYM_FUNC_END(srso_untrain_ret_alias) __EXPORT_THUNK(srso_untrain_ret_alias) =20 .section .text..__x86.rethunk_safe #endif =20 -/* Needs a definition for the __x86_return_thunk alternative below. */ SYM_START(srso_safe_ret_alias, SYM_L_GLOBAL, SYM_A_NONE) -#ifdef CONFIG_CPU_SRSO lea 8(%_ASM_SP), %_ASM_SP UNWIND_HINT_FUNC -#endif ANNOTATE_UNRET_SAFE ret int3 @@ -174,9 +172,16 @@ SYM_FUNC_END(srso_safe_ret_alias) =20 .section .text..__x86.return_thunk =20 +SYM_CODE_START(srso_alias_return_thunk) + UNWIND_HINT_FUNC + ANNOTATE_NOENDBR + call srso_safe_ret_alias + ud2 +SYM_CODE_END(srso_alias_return_thunk) + /* * Safety details here pertain to the AMD Zen{1,2} microarchitecture: - * 1) The RET at __x86_return_thunk must be on a 64 byte boundary, for + * 1) The RET at zen_return_thunk must be on a 64 byte boundary, for * alignment within the BTB. * 2) The instruction at zen_untrain_ret must contain, and not * end with, the 0xc3 byte of the RET. @@ -184,7 +189,7 @@ SYM_FUNC_END(srso_safe_ret_alias) * from re-poisioning the BTB prediction. */ .align 64 - .skip 64 - (__ret - zen_untrain_ret), 0xcc + .skip 64 - (zen_return_thunk - zen_untrain_ret), 0xcc SYM_START(zen_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) ANNOTATE_NOENDBR /* @@ -192,16 +197,16 @@ SYM_START(zen_untrain_ret, SYM_L_GLOBAL, * * TEST $0xcc, %bl * LFENCE - * JMP __x86_return_thunk + * JMP zen_return_thunk * * Executing the TEST instruction has a side effect of evicting any BTB * prediction (potentially attacker controlled) attached to the RET, as - * __x86_return_thunk + 1 isn't an instruction boundary at the moment. + * zen_return_thunk + 1 isn't an instruction boundary at the moment. */ .byte 0xf6 =20 /* - * As executed from __x86_return_thunk, this is a plain RET. + * As executed from zen_return_thunk, this is a plain RET. * * As part of the TEST above, RET is the ModRM byte, and INT3 the imm8. * @@ -213,13 +218,13 @@ SYM_START(zen_untrain_ret, SYM_L_GLOBAL, * With SMT enabled and STIBP active, a sibling thread cannot poison * RET's prediction to a type of its choice, but can evict the * prediction due to competitive sharing. If the prediction is - * evicted, __x86_return_thunk will suffer Straight Line Speculation + * evicted, zen_return_thunk will suffer Straight Line Speculation * which will be contained safely by the INT3. */ -SYM_INNER_LABEL(__ret, SYM_L_GLOBAL) +SYM_INNER_LABEL(zen_return_thunk, SYM_L_GLOBAL) ret int3 -SYM_CODE_END(__ret) +SYM_CODE_END(zen_return_thunk) =20 /* * Ensure the TEST decoding / BTB invalidation is complete. @@ -230,7 +235,7 @@ SYM_CODE_END(__ret) * Jump back and execute the RET in the middle of the TEST instruction. * INT3 is for SLS protection. */ - jmp __ret + jmp zen_return_thunk int3 SYM_FUNC_END(zen_untrain_ret) __EXPORT_THUNK(zen_untrain_ret) @@ -256,6 +261,7 @@ SYM_INNER_LABEL(srso_safe_ret, SYM_L_GLO ret int3 int3 + /* end of movabs */ lfence call srso_safe_ret ud2 @@ -263,12 +269,19 @@ SYM_CODE_END(srso_safe_ret) SYM_FUNC_END(srso_untrain_ret) __EXPORT_THUNK(srso_untrain_ret) =20 -SYM_CODE_START(__x86_return_thunk) +SYM_CODE_START(srso_return_thunk) UNWIND_HINT_FUNC ANNOTATE_NOENDBR - ALTERNATIVE_2 "jmp __ret", "call srso_safe_ret", X86_FEATURE_SRSO, \ - "call srso_safe_ret_alias", X86_FEATURE_SRSO_ALIAS + call srso_safe_ret ud2 +SYM_CODE_END(srso_return_thunk) + +SYM_CODE_START(__x86_return_thunk) + UNWIND_HINT_FUNC + ANNOTATE_NOENDBR + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(__x86_return_thunk) EXPORT_SYMBOL(__x86_return_thunk) =20 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -829,6 +829,6 @@ bool arch_is_rethunk(struct symbol *sym) =20 bool arch_is_embedded_insn(struct symbol *sym) { - return !strcmp(sym->name, "__ret") || + return !strcmp(sym->name, "zen_return_thunk") || !strcmp(sym->name, "srso_safe_ret"); } From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76C0DC04FE2 for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233998AbjHNMUH (ORCPT ); Mon, 14 Aug 2023 08:20:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233958AbjHNMTe (ORCPT ); Mon, 14 Aug 2023 08:19:34 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AD78195 for ; Mon, 14 Aug 2023 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=d6SegL+nYoLh8jug65feyPeZBS++RFW4LBO1LVgSiZQ=; b=u9PDo1PR8M5ZlIrVBEE8OU3DVa 6uG13G2dAgcE8HtCx9/e+Wugl7jpbBFStGt2VECnTnmgqF0A+2GnQWuOxR/9SoumImJmFkxkpsk3E TtlMNxALBA9f9vtc/PrSoi2QGLGWlL438U9DElHZWlkv0Mxo4fMh3Rb43dUoemYs6QowKX24R+TZ8 FJiSvlZNJUWDUn+gOGejGwqTlDGSUuwSw+Kfbs3QZyewwWAfWdtR8/twjj6EVUBJQOH07in5Kause Z54HfX+c8rX8yNVL28kIkToH0AJqCJeS+Y9cMtuEr3USGHIoJW4Dws55/vDFWJBIdt5Jgc5NlwTlX FnMLuzGQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVWXY-001t4D-VY; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E4DF43007FB; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C74CB21C1D1B5; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121148.909378169@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:32 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 06/11] x86/cpu: Rename original retbleed methods References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename the original retbleed return thunk and untrain_ret to retbleed_return_thunk and retbleed_untrain_ret. Andrew wants to call this btc_*, do we have a poll? Suggested-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 8 ++++---- arch/x86/kernel/cpu/bugs.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 2 +- arch/x86/lib/retpoline.S | 30 +++++++++++++++--------------- tools/objtool/arch/x86/decode.c | 2 +- tools/objtool/check.c | 2 +- 6 files changed, 23 insertions(+), 23 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -272,7 +272,7 @@ .endm =20 #ifdef CONFIG_CPU_UNRET_ENTRY -#define CALL_ZEN_UNTRAIN_RET "call zen_untrain_ret" +#define CALL_ZEN_UNTRAIN_RET "call retbleed_untrain_ret" #else #define CALL_ZEN_UNTRAIN_RET "" #endif @@ -282,7 +282,7 @@ * return thunk isn't mapped into the userspace tables (then again, AMD * typically has NO_MELTDOWN). * - * While zen_untrain_ret() doesn't clobber anything but requires stack, + * While retbleed_untrain_ret() doesn't clobber anything but requires stac= k, * entry_ibpb() will clobber AX, CX, DX. * * As such, this must be placed after every *SWITCH_TO_KERNEL_CR3 at a poi= nt @@ -343,11 +343,11 @@ extern retpoline_thunk_t __x86_indirect_ =20 extern void __x86_return_thunk(void); =20 -extern void zen_return_thunk(void); +extern void retbleed_return_thunk(void); extern void srso_return_thunk(void); extern void srso_alias_return_thunk(void); =20 -extern void zen_untrain_ret(void); +extern void retbleed_untrain_ret(void); extern void srso_untrain_ret(void); extern void srso_untrain_ret_alias(void); =20 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1034,7 +1034,7 @@ static void __init retbleed_select_mitig case RETBLEED_MITIGATION_UNRET: setup_force_cpu_cap(X86_FEATURE_RETHUNK); setup_force_cpu_cap(X86_FEATURE_UNRET); - x86_return_thunk =3D zen_return_thunk; + x86_return_thunk =3D retbleed_return_thunk; =20 if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -521,7 +521,7 @@ INIT_PER_CPU(irq_stack_backing_store); #endif =20 #ifdef CONFIG_RETHUNK -. =3D ASSERT((zen_return_thunk & 0x3f) =3D=3D 0, "zen_return_thunk not cac= heline-aligned"); +. =3D ASSERT((retbleed_return_thunk & 0x3f) =3D=3D 0, "retbleed_return_thu= nk not cacheline-aligned"); . =3D ASSERT((srso_safe_ret & 0x3f) =3D=3D 0, "srso_safe_ret not cacheline= -aligned"); #endif =20 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -181,32 +181,32 @@ SYM_CODE_END(srso_alias_return_thunk) =20 /* * Safety details here pertain to the AMD Zen{1,2} microarchitecture: - * 1) The RET at zen_return_thunk must be on a 64 byte boundary, for + * 1) The RET at retbleed_return_thunk must be on a 64 byte boundary, for * alignment within the BTB. - * 2) The instruction at zen_untrain_ret must contain, and not + * 2) The instruction at retbleed_untrain_ret must contain, and not * end with, the 0xc3 byte of the RET. * 3) STIBP must be enabled, or SMT disabled, to prevent the sibling thread * from re-poisioning the BTB prediction. */ .align 64 - .skip 64 - (zen_return_thunk - zen_untrain_ret), 0xcc -SYM_START(zen_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) + .skip 64 - (retbleed_return_thunk - retbleed_untrain_ret), 0xcc +SYM_START(retbleed_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) ANNOTATE_NOENDBR /* - * As executed from zen_untrain_ret, this is: + * As executed from retbleed_untrain_ret, this is: * * TEST $0xcc, %bl * LFENCE - * JMP zen_return_thunk + * JMP retbleed_return_thunk * * Executing the TEST instruction has a side effect of evicting any BTB * prediction (potentially attacker controlled) attached to the RET, as - * zen_return_thunk + 1 isn't an instruction boundary at the moment. + * retbleed_return_thunk + 1 isn't an instruction boundary at the moment. */ .byte 0xf6 =20 /* - * As executed from zen_return_thunk, this is a plain RET. + * As executed from retbleed_return_thunk, this is a plain RET. * * As part of the TEST above, RET is the ModRM byte, and INT3 the imm8. * @@ -218,13 +218,13 @@ SYM_START(zen_untrain_ret, SYM_L_GLOBAL, * With SMT enabled and STIBP active, a sibling thread cannot poison * RET's prediction to a type of its choice, but can evict the * prediction due to competitive sharing. If the prediction is - * evicted, zen_return_thunk will suffer Straight Line Speculation + * evicted, retbleed_return_thunk will suffer Straight Line Speculation * which will be contained safely by the INT3. */ -SYM_INNER_LABEL(zen_return_thunk, SYM_L_GLOBAL) +SYM_INNER_LABEL(retbleed_return_thunk, SYM_L_GLOBAL) ret int3 -SYM_CODE_END(zen_return_thunk) +SYM_CODE_END(retbleed_return_thunk) =20 /* * Ensure the TEST decoding / BTB invalidation is complete. @@ -235,13 +235,13 @@ SYM_CODE_END(zen_return_thunk) * Jump back and execute the RET in the middle of the TEST instruction. * INT3 is for SLS protection. */ - jmp zen_return_thunk + jmp retbleed_return_thunk int3 -SYM_FUNC_END(zen_untrain_ret) -__EXPORT_THUNK(zen_untrain_ret) +SYM_FUNC_END(retbleed_untrain_ret) +__EXPORT_THUNK(retbleed_untrain_ret) =20 /* - * SRSO untraining sequence for Zen1/2, similar to zen_untrain_ret() + * SRSO untraining sequence for Zen1/2, similar to retbleed_untrain_ret() * above. On kernel entry, srso_untrain_ret() is executed which is a * * movabs $0xccccc30824648d48,%rax --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -829,6 +829,6 @@ bool arch_is_rethunk(struct symbol *sym) =20 bool arch_is_embedded_insn(struct symbol *sym) { - return !strcmp(sym->name, "zen_return_thunk") || + return !strcmp(sym->name, "retbleed_return_thunk") || !strcmp(sym->name, "srso_safe_ret"); } --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1593,7 +1593,7 @@ static int add_jump_destinations(struct struct symbol *sym =3D find_symbol_by_offset(dest_sec, dest_off); =20 /* - * This is a special case for zen_untrain_ret(). + * This is a special case for retbleed_untrain_ret(). * It jumps to __x86_return_thunk(), but objtool * can't find the thunk's starting RET * instruction, because the RET is also in the From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C18EC04FDF for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231851AbjHNMUA (ORCPT ); Mon, 14 Aug 2023 08:20:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233583AbjHNMTb (ORCPT ); Mon, 14 Aug 2023 08:19:31 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF327107 for ; Mon, 14 Aug 2023 05:19:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LK3mbjd5nWgwqe+uxKwpv04Jq+49BgYgVpVgFwn6JuQ=; b=TjcWH19Wbdlq1otV+nW5r+zJIS G+7p/8cl28HWJ+h1f8JjsYGpZbyPTXAujzq/PweWRTFLns2tF3kcOS2hpsWtvCMEm6mbWq/SVwjuw jX6Xnlke8QA1fAzRe5Bpq5CaMLjs3mwx/D/I/IeKmfCRVEQ92Oq9jgkacKYJsdMS07SXGxk1xC4Fg VDAlhf49lS1tqz7rAG6E9rxGsDKBiOJw/Lv2JQe1Ol80pY3h1fDsBBaSx6I+Ua/03eJwfJ/1ENTBO I3i670hFxZL7mgFMFOaPMyy9BmzFGsoxGAgFCGoq6vm//flo/uylreNSyRdnOQB1kHYg1QRR1kmjS U89TyJSQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qVWXY-00AaKb-39; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E4FC6300F59; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id CB56721C1D1B6; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121148.976236447@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:33 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 07/11] x86/cpu: Rename srso_(.*)_alias to srso_alias_\1 References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For a more consistent namespace. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 6 +++--- arch/x86/kernel/vmlinux.lds.S | 8 ++++---- arch/x86/lib/retpoline.S | 22 +++++++++++----------- 3 files changed, 18 insertions(+), 18 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -300,7 +300,7 @@ =20 #ifdef CONFIG_CPU_SRSO ALTERNATIVE_2 "", "call srso_untrain_ret", X86_FEATURE_SRSO, \ - "call srso_untrain_ret_alias", X86_FEATURE_SRSO_ALIAS + "call srso_alias_untrain_ret", X86_FEATURE_SRSO_ALIAS #endif .endm =20 @@ -316,7 +316,7 @@ =20 #ifdef CONFIG_CPU_SRSO ALTERNATIVE_2 "", "call srso_untrain_ret", X86_FEATURE_SRSO, \ - "call srso_untrain_ret_alias", X86_FEATURE_SRSO_ALIAS + "call srso_alias_untrain_ret", X86_FEATURE_SRSO_ALIAS #endif .endm =20 @@ -349,7 +349,7 @@ extern void srso_alias_return_thunk(void =20 extern void retbleed_untrain_ret(void); extern void srso_untrain_ret(void); -extern void srso_untrain_ret_alias(void); +extern void srso_alias_untrain_ret(void); =20 extern void entry_ibpb(void); =20 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -147,10 +147,10 @@ SECTIONS =20 #ifdef CONFIG_CPU_SRSO /* - * See the comment above srso_untrain_ret_alias()'s + * See the comment above srso_alias_untrain_ret()'s * definition. */ - . =3D srso_untrain_ret_alias | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 2= 0); + . =3D srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 2= 0); *(.text..__x86.rethunk_safe) #endif ALIGN_ENTRY_TEXT_END @@ -536,8 +536,8 @@ INIT_PER_CPU(irq_stack_backing_store); * Instead do: (A | B) - (A & B) in order to compute the XOR * of the two function addresses: */ -. =3D ASSERT(((ABSOLUTE(srso_untrain_ret_alias) | srso_safe_ret_alias) - - (ABSOLUTE(srso_untrain_ret_alias) & srso_safe_ret_alias)) =3D=3D ((1 << = 2) | (1 << 8) | (1 << 14) | (1 << 20)), +. =3D ASSERT(((ABSOLUTE(srso_alias_untrain_ret) | srso_alias_safe_ret) - + (ABSOLUTE(srso_alias_untrain_ret) & srso_alias_safe_ret)) =3D=3D ((1 << = 2) | (1 << 8) | (1 << 14) | (1 << 20)), "SRSO function pair won't alias"); #endif =20 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -133,49 +133,49 @@ SYM_CODE_END(__x86_indirect_jump_thunk_a #ifdef CONFIG_RETHUNK =20 /* - * srso_untrain_ret_alias() and srso_safe_ret_alias() are placed at + * srso_alias_untrain_ret() and srso_alias_safe_ret() are placed at * special addresses: * - * - srso_untrain_ret_alias() is 2M aligned - * - srso_safe_ret_alias() is also in the same 2M page but bits 2, 8, 14 + * - srso_alias_untrain_ret() is 2M aligned + * - srso_alias_safe_ret() is also in the same 2M page but bits 2, 8, 14 * and 20 in its virtual address are set (while those bits in the - * srso_untrain_ret_alias() function are cleared). + * srso_alias_untrain_ret() function are cleared). * * This guarantees that those two addresses will alias in the branch * target buffer of Zen3/4 generations, leading to any potential * poisoned entries at that BTB slot to get evicted. * - * As a result, srso_safe_ret_alias() becomes a safe return. + * As a result, srso_alias_safe_ret() becomes a safe return. */ #ifdef CONFIG_CPU_SRSO .section .text..__x86.rethunk_untrain =20 -SYM_START(srso_untrain_ret_alias, SYM_L_GLOBAL, SYM_A_NONE) +SYM_START(srso_alias_untrain_ret, SYM_L_GLOBAL, SYM_A_NONE) UNWIND_HINT_FUNC ANNOTATE_NOENDBR ASM_NOP2 lfence jmp srso_alias_return_thunk -SYM_FUNC_END(srso_untrain_ret_alias) -__EXPORT_THUNK(srso_untrain_ret_alias) +SYM_FUNC_END(srso_alias_untrain_ret) +__EXPORT_THUNK(srso_alias_untrain_ret) =20 .section .text..__x86.rethunk_safe #endif =20 -SYM_START(srso_safe_ret_alias, SYM_L_GLOBAL, SYM_A_NONE) +SYM_START(srso_alias_safe_ret, SYM_L_GLOBAL, SYM_A_NONE) lea 8(%_ASM_SP), %_ASM_SP UNWIND_HINT_FUNC ANNOTATE_UNRET_SAFE ret int3 -SYM_FUNC_END(srso_safe_ret_alias) +SYM_FUNC_END(srso_alias_safe_ret) =20 .section .text..__x86.return_thunk =20 SYM_CODE_START(srso_alias_return_thunk) UNWIND_HINT_FUNC ANNOTATE_NOENDBR - call srso_safe_ret_alias + call srso_alias_safe_ret ud2 SYM_CODE_END(srso_alias_return_thunk) From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8575FC07E8A for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234100AbjHNMUJ (ORCPT ); Mon, 14 Aug 2023 08:20:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233941AbjHNMTe (ORCPT ); Mon, 14 Aug 2023 08:19:34 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC59D10E5 for ; Mon, 14 Aug 2023 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=k+jdCnSGLNegRrt+BRPd/E4RcrAcPDHt8sE5EB9kWss=; b=WL1A9g4qII0nPzV+/E+nJYbGFz XkOqXa3fNsZJEWtZj8B+YdPZqCGN7rXz7y8nZ4vq6dhUrHdfo05HodWty1zpRVQs0CrZCCSfW0IRz SmsFp4yx9BaqUrX6k+18jXGDmFjx7HDhEPBMmyAVvnb44Xcs1lZpRVa3tERp0qyXQPzE7Da7PST1T PZQ/tUJxIrjl0vrcWU8PWNPaT6u8mJIYGI2bXmj7CY4c1Oe081rXTFwK6VJzsTm/1B/ZHkei4zszS 7rjnkgfAhx4SRFGDnAlLR1FT0KmwAAVdFa8MhDZpAuEMcuMQFfKcmnyy0pT6fA6x7SRXnJUM0DApA /4IfhMmg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qVWXY-00AaKZ-2z; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E4F723008AC; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id CF53921C1D1B7; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121149.042774962@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:34 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 08/11] x86/cpu: Cleanup the untrain mess References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since there can only be one active return_thunk, there only needs be one (matching) untrain_ret. It fundamentally doesn't make sense to allow multiple untrain_ret at the same time. Fold all the 3 different untrain methods into a single (temporary) helper stub. Fixes: fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation") Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 19 +++++-------------- arch/x86/kernel/cpu/bugs.c | 1 + arch/x86/lib/retpoline.S | 7 +++++++ 3 files changed, 13 insertions(+), 14 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -272,9 +272,9 @@ .endm =20 #ifdef CONFIG_CPU_UNRET_ENTRY -#define CALL_ZEN_UNTRAIN_RET "call retbleed_untrain_ret" +#define CALL_UNTRAIN_RET "call entry_untrain_ret" #else -#define CALL_ZEN_UNTRAIN_RET "" +#define CALL_UNTRAIN_RET "" #endif =20 /* @@ -293,15 +293,10 @@ defined(CONFIG_CALL_DEPTH_TRACKING) || defined(CONFIG_CPU_SRSO) VALIDATE_UNRET_END ALTERNATIVE_3 "", \ - CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ + CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ __stringify(RESET_CALL_DEPTH), X86_FEATURE_CALL_DEPTH #endif - -#ifdef CONFIG_CPU_SRSO - ALTERNATIVE_2 "", "call srso_untrain_ret", X86_FEATURE_SRSO, \ - "call srso_alias_untrain_ret", X86_FEATURE_SRSO_ALIAS -#endif .endm =20 .macro UNTRAIN_RET_FROM_CALL @@ -309,15 +304,10 @@ defined(CONFIG_CALL_DEPTH_TRACKING) VALIDATE_UNRET_END ALTERNATIVE_3 "", \ - CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ + CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ __stringify(RESET_CALL_DEPTH_FROM_CALL), X86_FEATURE_CALL_DEPTH #endif - -#ifdef CONFIG_CPU_SRSO - ALTERNATIVE_2 "", "call srso_untrain_ret", X86_FEATURE_SRSO, \ - "call srso_alias_untrain_ret", X86_FEATURE_SRSO_ALIAS -#endif .endm =20 =20 @@ -351,6 +341,7 @@ extern void retbleed_untrain_ret(void); extern void srso_untrain_ret(void); extern void srso_alias_untrain_ret(void); =20 +extern void entry_untrain_ret(void); extern void entry_ibpb(void); =20 extern void (*x86_return_thunk)(void); --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -2449,6 +2449,7 @@ static void __init srso_select_mitigatio * like ftrace, static_call, etc. */ setup_force_cpu_cap(X86_FEATURE_RETHUNK); + setup_force_cpu_cap(X86_FEATURE_UNRET); =20 if (boot_cpu_data.x86 =3D=3D 0x19) { setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -277,6 +277,13 @@ SYM_CODE_START(srso_return_thunk) ud2 SYM_CODE_END(srso_return_thunk) =20 +SYM_FUNC_START(entry_untrain_ret) + ALTERNATIVE_2 "jmp retbleed_untrain_ret", \ + "jmp srso_untrain_ret", X86_FEATURE_SRSO, \ + "jmp srso_alias_untrain_ret", X86_FEATURE_SRSO_ALIAS +SYM_FUNC_END(entry_untrain_ret) +__EXPORT_THUNK(entry_untrain_ret) + SYM_CODE_START(__x86_return_thunk) UNWIND_HINT_FUNC ANNOTATE_NOENDBR From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F00BCC001DB for ; Mon, 14 Aug 2023 12:20:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230320AbjHNMT6 (ORCPT ); Mon, 14 Aug 2023 08:19:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231704AbjHNMTc (ORCPT ); Mon, 14 Aug 2023 08:19:32 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23A3018F for ; Mon, 14 Aug 2023 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=RWAKOPXL75r/HZ4tdiHieOe7R8DXm6PJWs2Op5/I0MI=; b=at9vLP/NyIwieP1QiMvha26Uib K79BfN17+B5So+qNaPaY5zvn9gonO3E31gWwhm3AxBTmngCg1haw8dHEd+xVXDPva8GzEhZn8+7fh DjyP6/CL1P8EbjRHY57hx0kfHFEpBL9rf/Y9qbG/d9RB5LA/MTvKixgRocNlbvs8froSt9BSmxnHN 1EARSZHPQgheljCEkHAZKoA1rjkSgX0jubJ28uQN3bJyL4cyYV6iVcVLDnVL9CjHSi4hOEA7suWtu jD8WAj+oQc9m2PQxRauiWLCoJ1JvRRbBDsOFbx+CyQ9JJ2BPl4t/zCQ9acugRYMW2dbR/GwgDs2Wl HxTtbIfA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qVWXY-00AaKa-37; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E4EF0300886; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D32B821C1D1B8; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121149.109557833@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:35 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 09/11] x86/cpu/kvm: Provide UNTRAIN_RET_VM References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similar to how it doesn't make sense to have UNTRAIN_RET have two untrain calls, it also doesn't make sense for VMEXIT to have an extra IBPB call. This cures VMEXIT doing potentially unret+IBPB or double IBPB. Also, the (SEV) VMEXIT case seems to have been overlooked. Redefine the meaning of the synthetic IBPB flags to: - ENTRY_IBPB -- issue IBPB on entry (was: entry + VMEXIT) - IBPB_ON_VMEXIT -- issue IBPB on VMEXIT And have 'retbleed=3Dibpb' set *BOTH* feature flags to ensure it retains the previous behaviour and issues IBPB on entry+VMEXIT. The new 'srso=3Dibpb_vmexit' option only sets IBPB_ON_VMEXIT. Create UNTRAIN_RET_VM specifically for the VMEXIT case, and have that check IBPB_ON_VMEXIT. All this avoids having the VMEXIT case having to check both ENTRY_IBPB and IBPB_ON_VMEXIT and simplifies the alternatives. Fixes: fb3bd914b3ec ("x86/srso: Add a Speculative RAS Overflow mitigation") Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 11 +++++++++++ arch/x86/kernel/cpu/bugs.c | 1 + arch/x86/kvm/svm/vmenter.S | 7 ++----- 3 files changed, 14 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -299,6 +299,17 @@ #endif .endm =20 +.macro UNTRAIN_RET_VM +#if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ + defined(CONFIG_CALL_DEPTH_TRACKING) || defined(CONFIG_CPU_SRSO) + VALIDATE_UNRET_END + ALTERNATIVE_3 "", \ + CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ + "call entry_ibpb", X86_FEATURE_IBPB_ON_VMEXIT, \ + __stringify(RESET_CALL_DEPTH), X86_FEATURE_CALL_DEPTH +#endif +.endm + .macro UNTRAIN_RET_FROM_CALL #if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ defined(CONFIG_CALL_DEPTH_TRACKING) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1045,6 +1045,7 @@ static void __init retbleed_select_mitig =20 case RETBLEED_MITIGATION_IBPB: setup_force_cpu_cap(X86_FEATURE_ENTRY_IBPB); + setup_force_cpu_cap(X86_FEATURE_IBPB_ON_VMEXIT); mitigate_smt =3D true; break; =20 --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -222,10 +222,7 @@ SYM_FUNC_START(__svm_vcpu_run) * because interrupt handlers won't sanitize 'ret' if the return is * from the kernel. */ - UNTRAIN_RET - - /* SRSO */ - ALTERNATIVE "", "call entry_ibpb", X86_FEATURE_IBPB_ON_VMEXIT + UNTRAIN_RET_VM =20 /* * Clear all general purpose registers except RSP and RAX to prevent @@ -362,7 +359,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) * because interrupt handlers won't sanitize RET if the return is * from the kernel. */ - UNTRAIN_RET + UNTRAIN_RET_VM =20 /* "Pop" @spec_ctrl_intercepted. */ pop %_ASM_BX From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A817C04FE0 for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233645AbjHNMUD (ORCPT ); Mon, 14 Aug 2023 08:20:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234035AbjHNMTe (ORCPT ); Mon, 14 Aug 2023 08:19:34 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E902183 for ; Mon, 14 Aug 2023 05:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=eJYoBaLwFQ8HTmoWBxbdGh7SPizbmjSWCfqJNKOHlJE=; b=wSuvFG5dvqrJ3V91LlLYHPDbe4 xZ/FoW4lC7dEuUcC8LH7SSQg9vPawMOX3gSvi08wLz1a4NvggtsSgWQAaphXCGMXoHbv7snU2LDtr GR6JKJQgKMYbWlLafWZNi5kih0up+MbR0DhSPUUjX9forqUEa9BF+svSCw/8G46xEV3/59SdKKdYW cjlCSWsM8c2dIfb7TyXP+Cv1V/BlawCOqHm/6czpBMoVxxnvVy5UsV90Em45uqo5gqdhZVeb6thWm kWUcmf1Dqu7QD4T0NRwZfewjCI1f34PCq06F1oj+eJR248ePcX6Vj+hOvJa4m0uZpHcslfU8AhSro AMZLNpvQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qVWXY-001t4E-Vp; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E9FF03010E0; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D74FA21C1D1B9; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121149.176244760@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:36 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 10/11] x86/alternatives: Simplify ALTERNATIVE_n() References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of making increasingly complicated ALTERNATIVE_n() implementations, use a nested alternative expression. The only difference between: ALTERNATIVE_2(oldinst, newinst1, flag1, newinst2, flag2) and ALTERNATIVE(ALTERNATIVE(oldinst, newinst1, flag1), newinst2, flag2) is that the outer alternative can add additional padding when the inner alternative is the shorter one, which then results in alt_instr::instrlen being inconsistent. However, this is easily remedied since the alt_instr entries will be consecutive and it is trivial to compute the max(alt_instr::instrlen) at runtime while patching. Specifically, after this patch the ALTERNATIVE_2 macro, after CPP expansion (and manual layout), looks like this: .macro ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 140: 140: \oldinstr ; 141: .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)= ),0x90 ; 142: .pushsection .altinstructions,"a" ; altinstr_entry 140b,143f,\ft_flags1,142b-140b,144f-143f ; .popsection ; .pushsection .altinstr_replacement,"ax" ; 143: \newinstr1 ; 144: .popsection ; ; 141: .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),= 0x90 ; 142: .pushsection .altinstructions,"a" ; altinstr_entry 140b,143f,\ft_flags2,142b-140b,144f-143f ; .popsection ; .pushsection .altinstr_replacement,"ax" ; 143: \newinstr2 ; 144: .popsection ; .endm The only label that is ambiguous is 140, however they all reference the same spot, so that doesn't matter. NOTE: obviously only @oldinstr may be an alternative; making @newinstr an alternative would mean patching .altinstr_replacement which very likely isn't what is intended, also the labels will be confused in that case. Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20230628104952.GA2439977@hirez.programming.= kicks-ass.net Reviewed-by: Nikolay Borisov --- arch/x86/include/asm/alternative.h | 206 ++++++++++----------------------= ----- arch/x86/kernel/alternative.c | 13 ++ tools/objtool/arch/x86/special.c | 23 ++++ tools/objtool/special.c | 16 +- 4 files changed, 100 insertions(+), 158 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -150,102 +150,60 @@ static inline int alternatives_text_rese } #endif /* CONFIG_SMP */ =20 -#define b_replacement(num) "664"#num -#define e_replacement(num) "665"#num - -#define alt_end_marker "663" #define alt_slen "662b-661b" -#define alt_total_slen alt_end_marker"b-661b" -#define alt_rlen(num) e_replacement(num)"f-"b_replacement(num)"f" +#define alt_total_slen "663b-661b" +#define alt_rlen "665f-664f" =20 -#define OLDINSTR(oldinstr, num) \ +#define OLDINSTR(oldinstr) \ "# ALT: oldnstr\n" \ "661:\n\t" oldinstr "\n662:\n" \ "# ALT: padding\n" \ - ".skip -(((" alt_rlen(num) ")-(" alt_slen ")) > 0) * " \ - "((" alt_rlen(num) ")-(" alt_slen ")),0x90\n" \ - alt_end_marker ":\n" - -/* - * gas compatible max based on the idea from: - * http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax - * - * The additional "-" is needed because gas uses a "true" value of -1. - */ -#define alt_max_short(a, b) "((" a ") ^ (((" a ") ^ (" b ")) & -(-((" a ")= < (" b ")))))" + ".skip -(((" alt_rlen ")-(" alt_slen ")) > 0) * " \ + "((" alt_rlen ")-(" alt_slen ")),0x90\n" \ + "663:\n" =20 -/* - * Pad the second replacement alternative with additional NOPs if it is - * additionally longer than the first replacement alternative. - */ -#define OLDINSTR_2(oldinstr, num1, num2) \ - "# ALT: oldinstr2\n" \ - "661:\n\t" oldinstr "\n662:\n" \ - "# ALT: padding2\n" \ - ".skip -((" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen= ")) > 0) * " \ - "(" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen ")), 0= x90\n" \ - alt_end_marker ":\n" - -#define OLDINSTR_3(oldinsn, n1, n2, n3) \ - "# ALT: oldinstr3\n" \ - "661:\n\t" oldinsn "\n662:\n" \ - "# ALT: padding3\n" \ - ".skip -((" alt_max_short(alt_max_short(alt_rlen(n1), alt_rlen(n2)), alt_= rlen(n3)) \ - " - (" alt_slen ")) > 0) * " \ - "(" alt_max_short(alt_max_short(alt_rlen(n1), alt_rlen(n2)), alt_rlen(n3= )) \ - " - (" alt_slen ")), 0x90\n" \ - alt_end_marker ":\n" - -#define ALTINSTR_ENTRY(ft_flags, num) \ +#define ALTINSTR_ENTRY(ft_flags) \ + ".pushsection .altinstructions,\"a\"\n" \ " .long 661b - .\n" /* label */ \ - " .long " b_replacement(num)"f - .\n" /* new instruction */ \ + " .long 664f - .\n" /* new instruction */ \ " .4byte " __stringify(ft_flags) "\n" /* feature + flags */ \ " .byte " alt_total_slen "\n" /* source len */ \ - " .byte " alt_rlen(num) "\n" /* replacement len */ - -#define ALTINSTR_REPLACEMENT(newinstr, num) /* replacement */ \ - "# ALT: replacement " #num "\n" \ - b_replacement(num)":\n\t" newinstr "\n" e_replacement(num) ":\n" - -/* alternative assembly primitive: */ -#define ALTERNATIVE(oldinstr, newinstr, ft_flags) \ - OLDINSTR(oldinstr, 1) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags, 1) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinstr, 1) \ + " .byte " alt_rlen "\n" /* replacement len */ \ ".popsection\n" =20 -#define ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2= ) \ - OLDINSTR_2(oldinstr, 1, 2) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags1, 1) \ - ALTINSTR_ENTRY(ft_flags2, 2) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinstr1, 1) \ - ALTINSTR_REPLACEMENT(newinstr2, 2) \ +#define ALTINSTR_REPLACEMENT(newinstr) /* replacement */ \ + ".pushsection .altinstr_replacement, \"ax\"\n" \ + "# ALT: replacement \n" \ + "664:\n\t" newinstr "\n 665:\n" \ ".popsection\n" =20 +/* + * Define an alternative between two instructions. If @ft_flags is + * present, early code in apply_alternatives() replaces @oldinstr with + * @newinstr. ".skip" directive takes care of proper instruction padding + * in case @newinstr is longer than @oldinstr. + * + * Notably: @oldinstr may be an ALTERNATIVE() itself, also see + * apply_alternatives() + */ +#define ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + OLDINSTR(oldinstr) \ + ALTINSTR_ENTRY(ft_flags) \ + ALTINSTR_REPLACEMENT(newinstr) + +#define ALTERNATIVE_2(oldinst, newinst1, flag1, newinst2, flag2) \ + ALTERNATIVE(ALTERNATIVE(oldinst, newinst1, flag1), \ + newinst2, flag2) + /* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY(oldinstr, ft_flags, newinstr_yes, newinstr_no)= \ ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ newinstr_yes, ft_flags) =20 -#define ALTERNATIVE_3(oldinsn, newinsn1, ft_flags1, newinsn2, ft_flags2, \ - newinsn3, ft_flags3) \ - OLDINSTR_3(oldinsn, 1, 2, 3) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags1, 1) \ - ALTINSTR_ENTRY(ft_flags2, 2) \ - ALTINSTR_ENTRY(ft_flags3, 3) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinsn1, 1) \ - ALTINSTR_REPLACEMENT(newinsn2, 2) \ - ALTINSTR_REPLACEMENT(newinsn3, 3) \ - ".popsection\n" +#define ALTERNATIVE_3(oldinst, newinst1, flag1, newinst2, flag2, \ + newinst3, flag3) \ + ALTERNATIVE(ALTERNATIVE_2(oldinst, newinst1, flag1, newinst2, flag2), \ + newinst3, flag3) =20 /* * Alternative instructions for different CPU types or capabilities. @@ -370,6 +328,21 @@ static inline int alternatives_text_rese .byte \alt_len .endm =20 +#define __ALTERNATIVE(oldinst, newinst, flag) \ +140: \ + oldinst ; \ +141: \ + .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 ;\ +142: \ + .pushsection .altinstructions,"a" ; \ + altinstr_entry 140b,143f,flag,142b-140b,144f-143f ; \ + .popsection ; \ + .pushsection .altinstr_replacement,"ax" ; \ +143: \ + newinst ; \ +144: \ + .popsection ; + /* * Define an alternative between two instructions. If @feature is * present, early code in apply_alternatives() replaces @oldinstr with @@ -377,88 +350,23 @@ static inline int alternatives_text_rese * in case @newinstr is longer than @oldinstr. */ .macro ALTERNATIVE oldinstr, newinstr, ft_flags -140: - \oldinstr -141: - .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags,142b-140b,144f-143f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr -144: - .popsection + __ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) .endm =20 -#define old_len 141b-140b -#define new_len1 144f-143f -#define new_len2 145f-144f -#define new_len3 146f-145f - -/* - * gas compatible max based on the idea from: - * http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax - * - * The additional "-" is needed because gas uses a "true" value of -1. - */ -#define alt_max_2(a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) -#define alt_max_3(a, b, c) (alt_max_2(alt_max_2(a, b), c)) - - /* * Same as ALTERNATIVE macro above but for two alternatives. If CPU * has @feature1, it replaces @oldinstr with @newinstr1. If CPU has * @feature2, it replaces @oldinstr with @feature2. */ .macro ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 -140: - \oldinstr -141: - .skip -((alt_max_2(new_len1, new_len2) - (old_len)) > 0) * \ - (alt_max_2(new_len1, new_len2) - (old_len)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags1,142b-140b,144f-143f - altinstr_entry 140b,144f,\ft_flags2,142b-140b,145f-144f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr1 -144: - \newinstr2 -145: - .popsection + __ALTERNATIVE(__ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), + \newinstr2, \ft_flags2) .endm =20 .macro ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2,= newinstr3, ft_flags3 -140: - \oldinstr -141: - .skip -((alt_max_3(new_len1, new_len2, new_len3) - (old_len)) > 0) * \ - (alt_max_3(new_len1, new_len2, new_len3) - (old_len)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags1,142b-140b,144f-143f - altinstr_entry 140b,144f,\ft_flags2,142b-140b,145f-144f - altinstr_entry 140b,145f,\ft_flags3,142b-140b,146f-145f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr1 -144: - \newinstr2 -145: - \newinstr3 -146: - .popsection + __ALTERNATIVE(__ALTERNATIVE(__ALTERNATIVE(\oldinstr, \newinstr1, \ft_flag= s1), + \newinstr2, \ft_flags2), + \newinstr3, \ft_flags3) .endm =20 /* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -398,7 +398,7 @@ apply_relocation(u8 *buf, size_t len, u8 void __init_or_module noinline apply_alternatives(struct alt_instr *start, struct alt_instr *end) { - struct alt_instr *a; + struct alt_instr *a, *b; u8 *instr, *replacement; u8 insn_buff[MAX_PATCH_LEN]; =20 @@ -415,6 +415,17 @@ void __init_or_module noinline apply_alt for (a =3D start; a < end; a++) { int insn_buff_sz =3D 0; =20 + /* + * In case of nested ALTERNATIVE()s the outer alternative might + * add more padding. To ensure consistent patching find the max + * padding for all alt_instr entries for this site (nested + * alternatives result in consecutive entries). + */ + for (b =3D a+1; b < end && b->instr_offset =3D=3D a->instr_offset; b++) { + u8 len =3D max(a->instrlen, b->instrlen); + a->instrlen =3D b->instrlen =3D len; + } + instr =3D (u8 *)&a->instr_offset + a->instr_offset; replacement =3D (u8 *)&a->repl_offset + a->repl_offset; BUG_ON(a->instrlen > sizeof(insn_buff)); --- a/tools/objtool/arch/x86/special.c +++ b/tools/objtool/arch/x86/special.c @@ -9,6 +9,29 @@ =20 void arch_handle_alternative(unsigned short feature, struct special_alt *a= lt) { + static struct special_alt *group, *prev; + + /* + * Recompute orig_len for nested ALTERNATIVE()s. + */ + if (group && group->orig_sec =3D=3D alt->orig_sec && + group->orig_off =3D=3D alt->orig_off) { + + struct special_alt *iter =3D group; + for (;;) { + unsigned int len =3D max(iter->orig_len, alt->orig_len); + iter->orig_len =3D alt->orig_len =3D len; + + if (iter =3D=3D prev) + break; + + iter =3D list_next_entry(iter, list); + } + + } else group =3D alt; + + prev =3D alt; + switch (feature) { case X86_FEATURE_SMAP: /* --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -84,6 +84,14 @@ static int get_alt_entry(struct elf *elf entry->new_len); } =20 + orig_reloc =3D find_reloc_by_dest(elf, sec, offset + entry->orig); + if (!orig_reloc) { + WARN_FUNC("can't find orig reloc", sec, offset + entry->orig); + return -1; + } + + reloc_to_sec_off(orig_reloc, &alt->orig_sec, &alt->orig_off); + if (entry->feature) { unsigned short feature; =20 @@ -94,14 +102,6 @@ static int get_alt_entry(struct elf *elf arch_handle_alternative(feature, alt); } =20 - orig_reloc =3D find_reloc_by_dest(elf, sec, offset + entry->orig); - if (!orig_reloc) { - WARN_FUNC("can't find orig reloc", sec, offset + entry->orig); - return -1; - } - - reloc_to_sec_off(orig_reloc, &alt->orig_sec, &alt->orig_off); - if (!entry->group || alt->new_len) { new_reloc =3D find_reloc_by_dest(elf, sec, offset + entry->new); if (!new_reloc) { From nobody Thu Dec 18 18:40:55 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67CB1C05052 for ; Mon, 14 Aug 2023 12:20:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233817AbjHNMUF (ORCPT ); Mon, 14 Aug 2023 08:20:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233998AbjHNMTe (ORCPT ); Mon, 14 Aug 2023 08:19:34 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9E70F5 for ; Mon, 14 Aug 2023 05:19:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=QKrcKC+fz5pTmgDQnsBjyi5G3CXOC+/dSWDSPAW6gBs=; b=OVg8NKHQtT5H6H9wULx8X+67M9 EjSeq9yBRZs+k7rx8FrbLIIAX9ttDI3xrF7P4TwkL/Jt6nOR3djw1aIO8NaJ1xnaSxhnM6R1chvs/ zUh2UR2+Jy/kGlj4tZ+94FSiVUOBtJp5xFSmD5CIOonvjOt9pheBYYy9SZJgj1TnABeISJ5/p5tiR 3eM3hVJO1cGRPDnaPp/g3xreYscpmJNfiCr+Dz6qBE0IyV4sb91kg8uYhdnVpgaZp0PSL86XzypnE tIM8SKth2CISswD/0AJbTwZwSeQHduHlNggqpRCgK/8NuF1dx+67b4ARCwTfd7yheN3iT0DfoIqxn a53+WypA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qVWXZ-00AaKd-1t; Mon, 14 Aug 2023 12:19:21 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id F0F0A301CA0; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id DC27E20410DA1; Mon, 14 Aug 2023 14:19:19 +0200 (CEST) Message-ID: <20230814121149.243457126@infradead.org> User-Agent: quilt/0.66 Date: Mon, 14 Aug 2023 13:44:37 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, David.Kaplan@amd.com, Andrew.Cooper3@citrix.com, jpoimboe@kernel.org, gregkh@linuxfoundation.org, nik.borisov@suse.com Subject: [PATCH v2 11/11] x86/cpu: Use fancy alternatives to get rid of entry_untrain_ret() References: <20230814114426.057251214@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the new nested alternatives to create what is effectively ALTERNATIVE_5 and merge the dummy entry_untrain_ret stub into UNTRAIN_RET properly. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/nospec-branch.h | 33 ++++++++++++++++++------------= --- arch/x86/kernel/cpu/bugs.c | 1 - arch/x86/lib/retpoline.S | 7 ------- 3 files changed, 18 insertions(+), 23 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -272,11 +272,15 @@ .endm =20 #ifdef CONFIG_CPU_UNRET_ENTRY -#define CALL_UNTRAIN_RET "call entry_untrain_ret" +#define ALT_UNRET(old) \ + __ALTERNATIVE(__ALTERNATIVE(__ALTERNATIVE(old, call retbleed_untrain_ret,= X86_FEATURE_UNRET), \ + call srso_untrain_ret, X86_FEATURE_SRSO), \ + call srso_alias_untrain_ret, X86_FEATURE_SRSO_ALIAS) #else -#define CALL_UNTRAIN_RET "" +#define ALT_UNRET(old) old #endif =20 + /* * Mitigate RETBleed for AMD/Hygon Zen uarch. Requires KERNEL CR3 because = the * return thunk isn't mapped into the userspace tables (then again, AMD @@ -292,10 +296,10 @@ #if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ defined(CONFIG_CALL_DEPTH_TRACKING) || defined(CONFIG_CPU_SRSO) VALIDATE_UNRET_END - ALTERNATIVE_3 "", \ - CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ - "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ - __stringify(RESET_CALL_DEPTH), X86_FEATURE_CALL_DEPTH + + __ALTERNATIVE(__ALTERNATIVE(ALT_UNRET(;), + call entry_ibpb, X86_FEATURE_ENTRY_IBPB), + RESET_CALL_DEPTH, X86_FEATURE_CALL_DEPTH) #endif .endm =20 @@ -303,10 +307,10 @@ #if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ defined(CONFIG_CALL_DEPTH_TRACKING) || defined(CONFIG_CPU_SRSO) VALIDATE_UNRET_END - ALTERNATIVE_3 "", \ - CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ - "call entry_ibpb", X86_FEATURE_IBPB_ON_VMEXIT, \ - __stringify(RESET_CALL_DEPTH), X86_FEATURE_CALL_DEPTH + + __ALTERNATIVE(__ALTERNATIVE(ALT_UNRET(;), + call entry_ibpb, X86_FEATURE_IBPB_ON_VMEXIT), + RESET_CALL_DEPTH, X86_FEATURE_CALL_DEPTH) #endif .endm =20 @@ -314,10 +318,10 @@ #if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) || \ defined(CONFIG_CALL_DEPTH_TRACKING) VALIDATE_UNRET_END - ALTERNATIVE_3 "", \ - CALL_UNTRAIN_RET, X86_FEATURE_UNRET, \ - "call entry_ibpb", X86_FEATURE_ENTRY_IBPB, \ - __stringify(RESET_CALL_DEPTH_FROM_CALL), X86_FEATURE_CALL_DEPTH + + __ALTERNATIVE(__ALTERNATIVE(ALT_UNRET(;), + call entry_ibpb, X86_FEATURE_ENTRY_IBPB), + RESET_CALL_DEPTH_FROM_CALL, X86_FEATURE_CALL_DEPTH) #endif .endm =20 @@ -352,7 +356,6 @@ extern void retbleed_untrain_ret(void); extern void srso_untrain_ret(void); extern void srso_alias_untrain_ret(void); =20 -extern void entry_untrain_ret(void); extern void entry_ibpb(void); =20 extern void (*x86_return_thunk)(void); --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -2450,7 +2450,6 @@ static void __init srso_select_mitigatio * like ftrace, static_call, etc. */ setup_force_cpu_cap(X86_FEATURE_RETHUNK); - setup_force_cpu_cap(X86_FEATURE_UNRET); =20 if (boot_cpu_data.x86 =3D=3D 0x19) { setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -277,13 +277,6 @@ SYM_CODE_START(srso_return_thunk) ud2 SYM_CODE_END(srso_return_thunk) =20 -SYM_FUNC_START(entry_untrain_ret) - ALTERNATIVE_2 "jmp retbleed_untrain_ret", \ - "jmp srso_untrain_ret", X86_FEATURE_SRSO, \ - "jmp srso_alias_untrain_ret", X86_FEATURE_SRSO_ALIAS -SYM_FUNC_END(entry_untrain_ret) -__EXPORT_THUNK(entry_untrain_ret) - SYM_CODE_START(__x86_return_thunk) UNWIND_HINT_FUNC ANNOTATE_NOENDBR