From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 757DBCCA483 for ; Sat, 23 Jul 2022 09:56:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237326AbiGWJ4d (ORCPT ); Sat, 23 Jul 2022 05:56:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230216AbiGWJ42 (ORCPT ); Sat, 23 Jul 2022 05:56:28 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1B891ADBD; Sat, 23 Jul 2022 02:56:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8EBC960C81; Sat, 23 Jul 2022 09:56:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 963D2C341C0; Sat, 23 Jul 2022 09:56:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570186; bh=94KCIESH3xJDnFEmSbONd5NtuihgFvmDI4tfqbS1O0E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0V/Ex/94+ccKQfB0UZ1ULbTr0DELhQzoU6i/6ditc71JGQh7HL0jaqQ1mt4LVK5pW oXI/OZLfTQ75X5yAU3VqKd6O4Woesw8FMu7p4cKnqEVvg5ggnS16Qq6ZHHgTVP/h7d NhTGjs9CpqNOqaj4BLlWSGZjc6nJrfl3VNeAiENE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Paolo Bonzini , Sean Christopherson , Uros Bizjak , Ben Hutchings Subject: [PATCH 5.10 001/148] KVM/VMX: Use TEST %REG,%REG instead of CMP $0,%REG in vmenter.S Date: Sat, 23 Jul 2022 11:53:33 +0200 Message-Id: <20220723095224.714782226@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 X-stable: review X-Patchwork-Hint: ignore MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Uros Bizjak commit 6c44221b05236cc65d76cb5dc2463f738edff39d upstream. Saves one byte in __vmx_vcpu_run for the same functionality. Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Uros Bizjak Message-Id: <20201029140457.126965-1-ubizjak@gmail.com> Signed-off-by: Paolo Bonzini Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/vmx/vmenter.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -132,7 +132,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov (%_ASM_SP), %_ASM_AX =20 /* Check if vmlaunch or vmresume is needed */ - cmpb $0, %bl + testb %bl, %bl =20 /* Load guest registers. Don't clobber flags. */ mov VCPU_RCX(%_ASM_AX), %_ASM_CX From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 198EFC433EF for ; Sat, 23 Jul 2022 09:56:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237356AbiGWJ4g (ORCPT ); Sat, 23 Jul 2022 05:56:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237312AbiGWJ4c (ORCPT ); Sat, 23 Jul 2022 05:56:32 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B4B0FD1F; Sat, 23 Jul 2022 02:56:31 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4FA70B82C1B; Sat, 23 Jul 2022 09:56:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 93E58C341CA; Sat, 23 Jul 2022 09:56:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570189; bh=xgz+phWSzPt31Ph3dA4Jf3eDUzJaXACYi/OnM6XS898=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=xKdtpuExy5uBnx7ze+RV2XlGLtfiVhHT2DaYKAp7Brbn/8a3XusUJ/7Q21i2O8qLd 9u+TqT8vJ6npBFHeIP+fn93VMyC3G6BrR3EMymL4MElpQK82LdaQYe4Tjy1gGr+6zh GjzU6jHBzpUECv6YaSXOpkdoJYhe2s24FE+QBnME= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Paolo Bonzini , Sean Christopherson , Uros Bizjak , Ben Hutchings Subject: [PATCH 5.10 002/148] KVM/nVMX: Use __vmx_vcpu_run in nested_vmx_check_vmentry_hw Date: Sat, 23 Jul 2022 11:53:34 +0200 Message-Id: <20220723095225.031469615@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Uros Bizjak commit 150f17bfab37e981ba03b37440638138ff2aa9ec upstream. Replace inline assembly in nested_vmx_check_vmentry_hw with a call to __vmx_vcpu_run. The function is not performance critical, so (double) GPR save/restore in __vmx_vcpu_run can be tolerated, as far as performance effects are concerned. Cc: Paolo Bonzini Cc: Sean Christopherson Reviewed-and-tested-by: Sean Christopherson Signed-off-by: Uros Bizjak [sean: dropped versioning info from changelog] Signed-off-by: Sean Christopherson Message-Id: <20201231002702.2223707-5-seanjc@google.com> Signed-off-by: Paolo Bonzini Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/vmx/nested.c | 32 +++----------------------------- arch/x86/kvm/vmx/vmenter.S | 2 +- arch/x86/kvm/vmx/vmx.c | 2 -- arch/x86/kvm/vmx/vmx.h | 1 + 4 files changed, 5 insertions(+), 32 deletions(-) --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -12,6 +12,7 @@ #include "nested.h" #include "pmu.h" #include "trace.h" +#include "vmx.h" #include "x86.h" =20 static bool __read_mostly enable_shadow_vmcs =3D 1; @@ -3075,35 +3076,8 @@ static int nested_vmx_check_vmentry_hw(s vmx->loaded_vmcs->host_state.cr4 =3D cr4; } =20 - asm( - "sub $%c[wordsize], %%" _ASM_SP "\n\t" /* temporarily adjust RSP for CAL= L */ - "cmp %%" _ASM_SP ", %c[host_state_rsp](%[loaded_vmcs]) \n\t" - "je 1f \n\t" - __ex("vmwrite %%" _ASM_SP ", %[HOST_RSP]") "\n\t" - "mov %%" _ASM_SP ", %c[host_state_rsp](%[loaded_vmcs]) \n\t" - "1: \n\t" - "add $%c[wordsize], %%" _ASM_SP "\n\t" /* un-adjust RSP */ - - /* Check if vmlaunch or vmresume is needed */ - "cmpb $0, %c[launched](%[loaded_vmcs])\n\t" - - /* - * VMLAUNCH and VMRESUME clear RFLAGS.{CF,ZF} on VM-Exit, set - * RFLAGS.CF on VM-Fail Invalid and set RFLAGS.ZF on VM-Fail - * Valid. vmx_vmenter() directly "returns" RFLAGS, and so the - * results of VM-Enter is captured via CC_{SET,OUT} to vm_fail. - */ - "call vmx_vmenter\n\t" - - CC_SET(be) - : ASM_CALL_CONSTRAINT, CC_OUT(be) (vm_fail) - : [HOST_RSP]"r"((unsigned long)HOST_RSP), - [loaded_vmcs]"r"(vmx->loaded_vmcs), - [launched]"i"(offsetof(struct loaded_vmcs, launched)), - [host_state_rsp]"i"(offsetof(struct loaded_vmcs, host_state.rsp)), - [wordsize]"i"(sizeof(ulong)) - : "memory" - ); + vm_fail =3D __vmx_vcpu_run(vmx, (unsigned long *)&vcpu->arch.regs, + vmx->loaded_vmcs->launched); =20 if (vmx->msr_autoload.host.nr) vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr); --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -44,7 +44,7 @@ * they VM-Fail, whereas a successful VM-Enter + VM-Exit will jump * to vmx_vmexit. */ -SYM_FUNC_START(vmx_vmenter) +SYM_FUNC_START_LOCAL(vmx_vmenter) /* EFLAGS.ZF is set if VMCS.LAUNCHED =3D=3D 0 */ je 2f =20 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6687,8 +6687,6 @@ static fastpath_t vmx_exit_handlers_fast } } =20 -bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, bool launch= ed); - static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) { --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -365,6 +365,7 @@ void vmx_set_virtual_apic_mode(struct kv struct vmx_uret_msr *vmx_find_uret_msr(struct vcpu_vmx *vmx, u32 msr); void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu); void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp); +bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, bool launch= ed); int vmx_find_loadstore_msr_slot(struct vmx_msrs *m, u32 msr); void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA5B6C433EF for ; Sat, 23 Jul 2022 09:56:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237371AbiGWJ4i (ORCPT ); Sat, 23 Jul 2022 05:56:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237328AbiGWJ4e (ORCPT ); Sat, 23 Jul 2022 05:56:34 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6013FD1F; Sat, 23 Jul 2022 02:56:32 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7418F6116A; Sat, 23 Jul 2022 09:56:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6D016C341C0; Sat, 23 Jul 2022 09:56:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570191; bh=7IGIX27Y9atr+FIz4Nekvn8paVwSfv/7bUB0/eaOZqs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ka1AhHqK85xW4IPooEidIpsQv0+z1q9aCNlfaOpB2nGcb/ssRbyR5Me+njUPmX8IA 2zHUm0mp/nK4rL+kwqnB0ejhNNpIUezEcDOdwJP3+DFbiRBhM97Jdozq3EmKMOXIX6 p1KY1YnK0FUfwxUZZgjWPuNWNaAFtE3NMDPcRpZc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 003/148] objtool: Refactor ORC section generation Date: Sat, 23 Jul 2022 11:53:35 +0200 Message-Id: <20220723095225.313623463@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit ab4e0744e99b87e1a223e89fc3c9ae44f727c9a6 upstream. Decouple ORC entries from instructions. This simplifies the control/data flow, and is going to make it easier to support alternative instructions which change the stack layout. Signed-off-by: Josh Poimboeuf Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/Makefile | 4=20 tools/objtool/arch.h | 4=20 tools/objtool/builtin-orc.c | 6=20 tools/objtool/check.h | 3=20 tools/objtool/objtool.h | 3=20 tools/objtool/orc_gen.c | 274 ++++++++++++++++++++++-----------------= ----- tools/objtool/weak.c | 7 - 7 files changed, 141 insertions(+), 160 deletions(-) --- a/tools/objtool/Makefile +++ b/tools/objtool/Makefile @@ -46,10 +46,6 @@ ifeq ($(SRCARCH),x86) SUBCMD_ORC :=3D y endif =20 -ifeq ($(SUBCMD_ORC),y) - CFLAGS +=3D -DINSN_USE_ORC -endif - export SUBCMD_CHECK SUBCMD_ORC export srctree OUTPUT CFLAGS SRCARCH AWK include $(srctree)/tools/build/Makefile.include --- a/tools/objtool/arch.h +++ b/tools/objtool/arch.h @@ -11,10 +11,6 @@ #include "objtool.h" #include "cfi.h" =20 -#ifdef INSN_USE_ORC -#include -#endif - enum insn_type { INSN_JUMP_CONDITIONAL, INSN_JUMP_UNCONDITIONAL, --- a/tools/objtool/builtin-orc.c +++ b/tools/objtool/builtin-orc.c @@ -51,11 +51,7 @@ int cmd_orc(int argc, const char **argv) if (list_empty(&file->insn_list)) return 0; =20 - ret =3D create_orc(file); - if (ret) - return ret; - - ret =3D create_orc_sections(file); + ret =3D orc_create(file); if (ret) return ret; =20 --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -43,9 +43,6 @@ struct instruction { struct symbol *func; struct list_head stack_ops; struct cfi_state cfi; -#ifdef INSN_USE_ORC - struct orc_entry orc; -#endif }; =20 static inline bool is_static_jump(struct instruction *insn) --- a/tools/objtool/objtool.h +++ b/tools/objtool/objtool.h @@ -26,7 +26,6 @@ struct objtool_file *objtool_open_read(c =20 int check(struct objtool_file *file); int orc_dump(const char *objname); -int create_orc(struct objtool_file *file); -int create_orc_sections(struct objtool_file *file); +int orc_create(struct objtool_file *file); =20 #endif /* _OBJTOOL_H */ --- a/tools/objtool/orc_gen.c +++ b/tools/objtool/orc_gen.c @@ -12,89 +12,84 @@ #include "check.h" #include "warn.h" =20 -int create_orc(struct objtool_file *file) +static int init_orc_entry(struct orc_entry *orc, struct cfi_state *cfi) { - struct instruction *insn; + struct instruction *insn =3D container_of(cfi, struct instruction, cfi); + struct cfi_reg *bp =3D &cfi->regs[CFI_BP]; =20 - for_each_insn(file, insn) { - struct orc_entry *orc =3D &insn->orc; - struct cfi_reg *cfa =3D &insn->cfi.cfa; - struct cfi_reg *bp =3D &insn->cfi.regs[CFI_BP]; + memset(orc, 0, sizeof(*orc)); =20 - if (!insn->sec->text) - continue; - - orc->end =3D insn->cfi.end; - - if (cfa->base =3D=3D CFI_UNDEFINED) { - orc->sp_reg =3D ORC_REG_UNDEFINED; - continue; - } - - switch (cfa->base) { - case CFI_SP: - orc->sp_reg =3D ORC_REG_SP; - break; - case CFI_SP_INDIRECT: - orc->sp_reg =3D ORC_REG_SP_INDIRECT; - break; - case CFI_BP: - orc->sp_reg =3D ORC_REG_BP; - break; - case CFI_BP_INDIRECT: - orc->sp_reg =3D ORC_REG_BP_INDIRECT; - break; - case CFI_R10: - orc->sp_reg =3D ORC_REG_R10; - break; - case CFI_R13: - orc->sp_reg =3D ORC_REG_R13; - break; - case CFI_DI: - orc->sp_reg =3D ORC_REG_DI; - break; - case CFI_DX: - orc->sp_reg =3D ORC_REG_DX; - break; - default: - WARN_FUNC("unknown CFA base reg %d", - insn->sec, insn->offset, cfa->base); - return -1; - } + orc->end =3D cfi->end; =20 - switch(bp->base) { - case CFI_UNDEFINED: - orc->bp_reg =3D ORC_REG_UNDEFINED; - break; - case CFI_CFA: - orc->bp_reg =3D ORC_REG_PREV_SP; - break; - case CFI_BP: - orc->bp_reg =3D ORC_REG_BP; - break; - default: - WARN_FUNC("unknown BP base reg %d", - insn->sec, insn->offset, bp->base); - return -1; - } + if (cfi->cfa.base =3D=3D CFI_UNDEFINED) { + orc->sp_reg =3D ORC_REG_UNDEFINED; + return 0; + } + + switch (cfi->cfa.base) { + case CFI_SP: + orc->sp_reg =3D ORC_REG_SP; + break; + case CFI_SP_INDIRECT: + orc->sp_reg =3D ORC_REG_SP_INDIRECT; + break; + case CFI_BP: + orc->sp_reg =3D ORC_REG_BP; + break; + case CFI_BP_INDIRECT: + orc->sp_reg =3D ORC_REG_BP_INDIRECT; + break; + case CFI_R10: + orc->sp_reg =3D ORC_REG_R10; + break; + case CFI_R13: + orc->sp_reg =3D ORC_REG_R13; + break; + case CFI_DI: + orc->sp_reg =3D ORC_REG_DI; + break; + case CFI_DX: + orc->sp_reg =3D ORC_REG_DX; + break; + default: + WARN_FUNC("unknown CFA base reg %d", + insn->sec, insn->offset, cfi->cfa.base); + return -1; + } =20 - orc->sp_offset =3D cfa->offset; - orc->bp_offset =3D bp->offset; - orc->type =3D insn->cfi.type; + switch (bp->base) { + case CFI_UNDEFINED: + orc->bp_reg =3D ORC_REG_UNDEFINED; + break; + case CFI_CFA: + orc->bp_reg =3D ORC_REG_PREV_SP; + break; + case CFI_BP: + orc->bp_reg =3D ORC_REG_BP; + break; + default: + WARN_FUNC("unknown BP base reg %d", + insn->sec, insn->offset, bp->base); + return -1; } =20 + orc->sp_offset =3D cfi->cfa.offset; + orc->bp_offset =3D bp->offset; + orc->type =3D cfi->type; + return 0; } =20 -static int create_orc_entry(struct elf *elf, struct section *u_sec, struct= section *ip_relocsec, - unsigned int idx, struct section *insn_sec, - unsigned long insn_off, struct orc_entry *o) +static int write_orc_entry(struct elf *elf, struct section *orc_sec, + struct section *ip_rsec, unsigned int idx, + struct section *insn_sec, unsigned long insn_off, + struct orc_entry *o) { struct orc_entry *orc; struct reloc *reloc; =20 /* populate ORC data */ - orc =3D (struct orc_entry *)u_sec->data->d_buf + idx; + orc =3D (struct orc_entry *)orc_sec->data->d_buf + idx; memcpy(orc, o, sizeof(*orc)); =20 /* populate reloc for ip */ @@ -114,102 +109,109 @@ static int create_orc_entry(struct elf * =20 reloc->type =3D R_X86_64_PC32; reloc->offset =3D idx * sizeof(int); - reloc->sec =3D ip_relocsec; + reloc->sec =3D ip_rsec; =20 elf_add_reloc(elf, reloc); =20 return 0; } =20 -int create_orc_sections(struct objtool_file *file) +struct orc_list_entry { + struct list_head list; + struct orc_entry orc; + struct section *insn_sec; + unsigned long insn_off; +}; + +static int orc_list_add(struct list_head *orc_list, struct orc_entry *orc, + struct section *sec, unsigned long offset) +{ + struct orc_list_entry *entry =3D malloc(sizeof(*entry)); + + if (!entry) { + WARN("malloc failed"); + return -1; + } + + entry->orc =3D *orc; + entry->insn_sec =3D sec; + entry->insn_off =3D offset; + + list_add_tail(&entry->list, orc_list); + return 0; +} + +int orc_create(struct objtool_file *file) { - struct instruction *insn, *prev_insn; - struct section *sec, *u_sec, *ip_relocsec; - unsigned int idx; + struct section *sec, *ip_rsec, *orc_sec; + unsigned int nr =3D 0, idx =3D 0; + struct orc_list_entry *entry; + struct list_head orc_list; =20 - struct orc_entry empty =3D { - .sp_reg =3D ORC_REG_UNDEFINED, + struct orc_entry null =3D { + .sp_reg =3D ORC_REG_UNDEFINED, .bp_reg =3D ORC_REG_UNDEFINED, .type =3D UNWIND_HINT_TYPE_CALL, }; =20 - sec =3D find_section_by_name(file->elf, ".orc_unwind"); - if (sec) { - WARN("file already has .orc_unwind section, skipping"); - return -1; - } - - /* count the number of needed orcs */ - idx =3D 0; + /* Build a deduplicated list of ORC entries: */ + INIT_LIST_HEAD(&orc_list); for_each_sec(file, sec) { + struct orc_entry orc, prev_orc =3D {0}; + struct instruction *insn; + bool empty =3D true; + if (!sec->text) continue; =20 - prev_insn =3D NULL; sec_for_each_insn(file, sec, insn) { - if (!prev_insn || - memcmp(&insn->orc, &prev_insn->orc, - sizeof(struct orc_entry))) { - idx++; - } - prev_insn =3D insn; + if (init_orc_entry(&orc, &insn->cfi)) + return -1; + if (!memcmp(&prev_orc, &orc, sizeof(orc))) + continue; + if (orc_list_add(&orc_list, &orc, sec, insn->offset)) + return -1; + nr++; + prev_orc =3D orc; + empty =3D false; } =20 - /* section terminator */ - if (prev_insn) - idx++; + /* Add a section terminator */ + if (!empty) { + orc_list_add(&orc_list, &null, sec, sec->len); + nr++; + } } - if (!idx) - return -1; + if (!nr) + return 0; =20 + /* Create .orc_unwind, .orc_unwind_ip and .rela.orc_unwind_ip sections: */ + sec =3D find_section_by_name(file->elf, ".orc_unwind"); + if (sec) { + WARN("file already has .orc_unwind section, skipping"); + return -1; + } + orc_sec =3D elf_create_section(file->elf, ".orc_unwind", 0, + sizeof(struct orc_entry), nr); + if (!orc_sec) + return -1; =20 - /* create .orc_unwind_ip and .rela.orc_unwind_ip sections */ - sec =3D elf_create_section(file->elf, ".orc_unwind_ip", 0, sizeof(int), i= dx); + sec =3D elf_create_section(file->elf, ".orc_unwind_ip", 0, sizeof(int), n= r); if (!sec) return -1; - - ip_relocsec =3D elf_create_reloc_section(file->elf, sec, SHT_RELA); - if (!ip_relocsec) + ip_rsec =3D elf_create_reloc_section(file->elf, sec, SHT_RELA); + if (!ip_rsec) return -1; =20 - /* create .orc_unwind section */ - u_sec =3D elf_create_section(file->elf, ".orc_unwind", 0, - sizeof(struct orc_entry), idx); - - /* populate sections */ - idx =3D 0; - for_each_sec(file, sec) { - if (!sec->text) - continue; - - prev_insn =3D NULL; - sec_for_each_insn(file, sec, insn) { - if (!prev_insn || memcmp(&insn->orc, &prev_insn->orc, - sizeof(struct orc_entry))) { - - if (create_orc_entry(file->elf, u_sec, ip_relocsec, idx, - insn->sec, insn->offset, - &insn->orc)) - return -1; - - idx++; - } - prev_insn =3D insn; - } - - /* section terminator */ - if (prev_insn) { - if (create_orc_entry(file->elf, u_sec, ip_relocsec, idx, - prev_insn->sec, - prev_insn->offset + prev_insn->len, - &empty)) - return -1; - - idx++; - } + /* Write ORC entries to sections: */ + list_for_each_entry(entry, &orc_list, list) { + if (write_orc_entry(file->elf, orc_sec, ip_rsec, idx++, + entry->insn_sec, entry->insn_off, + &entry->orc)) + return -1; } =20 - if (elf_rebuild_reloc_section(file->elf, ip_relocsec)) + if (elf_rebuild_reloc_section(file->elf, ip_rsec)) return -1; =20 return 0; --- a/tools/objtool/weak.c +++ b/tools/objtool/weak.c @@ -25,12 +25,7 @@ int __weak orc_dump(const char *_objname UNSUPPORTED("orc"); } =20 -int __weak create_orc(struct objtool_file *file) -{ - UNSUPPORTED("orc"); -} - -int __weak create_orc_sections(struct objtool_file *file) +int __weak orc_create(struct objtool_file *file) { UNSUPPORTED("orc"); } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7AE1C433EF for ; Sat, 23 Jul 2022 09:56:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237419AbiGWJ4r (ORCPT ); Sat, 23 Jul 2022 05:56:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237374AbiGWJ4p (ORCPT ); Sat, 23 Jul 2022 05:56:45 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D10D3207C; Sat, 23 Jul 2022 02:56:37 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EAD73B82C1B; Sat, 23 Jul 2022 09:56:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B5EFC341C0; Sat, 23 Jul 2022 09:56:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570194; bh=tB83HSTYttxr8QeYTHDKAIaxQNlUjPWiBjASejzuzWM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZhCz+2yS29EZ1nuIXGkT0BT4nPr1dHz5IMyPDnzT6f5WZaywXL6o1bswKVa56/a5J PMcdysnJ5z9zQJ+ikEocEbHW5CJcqKm0Fc3bKlpdfN0i5gZVPvPWzZjOZXUv1P7kxt YgSO1WT/GPtwoZoQj3s6DV0P06LQ7fsQJYUIs0zU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 004/148] objtool: Add alt_group struct Date: Sat, 23 Jul 2022 11:53:36 +0200 Message-Id: <20220723095225.568104100@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit b23cc71c62747f2e4c3e56138872cf47e1294f8a upstream. Create a new struct associated with each group of alternatives instructions. This will help with the removal of fake jumps, and more importantly with adding support for stack layout changes in alternatives. Signed-off-by: Josh Poimboeuf Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 29 +++++++++++++++++++++++------ tools/objtool/check.h | 13 ++++++++++++- 2 files changed, 35 insertions(+), 7 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1012,20 +1012,28 @@ static int handle_group_alt(struct objto struct instruction *orig_insn, struct instruction **new_insn) { - static unsigned int alt_group_next_index =3D 1; struct instruction *last_orig_insn, *last_new_insn, *insn, *fake_jump =3D= NULL; - unsigned int alt_group =3D alt_group_next_index++; + struct alt_group *orig_alt_group, *new_alt_group; unsigned long dest_off; =20 + + orig_alt_group =3D malloc(sizeof(*orig_alt_group)); + if (!orig_alt_group) { + WARN("malloc failed"); + return -1; + } last_orig_insn =3D NULL; insn =3D orig_insn; sec_for_each_insn_from(file, insn) { if (insn->offset >=3D special_alt->orig_off + special_alt->orig_len) break; =20 - insn->alt_group =3D alt_group; + insn->alt_group =3D orig_alt_group; last_orig_insn =3D insn; } + orig_alt_group->orig_group =3D NULL; + orig_alt_group->first_insn =3D orig_insn; + orig_alt_group->last_insn =3D last_orig_insn; =20 if (next_insn_same_sec(file, last_orig_insn)) { fake_jump =3D malloc(sizeof(*fake_jump)); @@ -1056,8 +1064,13 @@ static int handle_group_alt(struct objto return 0; } =20 + new_alt_group =3D malloc(sizeof(*new_alt_group)); + if (!new_alt_group) { + WARN("malloc failed"); + return -1; + } + last_new_insn =3D NULL; - alt_group =3D alt_group_next_index++; insn =3D *new_insn; sec_for_each_insn_from(file, insn) { struct reloc *alt_reloc; @@ -1069,7 +1082,7 @@ static int handle_group_alt(struct objto =20 insn->ignore =3D orig_insn->ignore_alts; insn->func =3D orig_insn->func; - insn->alt_group =3D alt_group; + insn->alt_group =3D new_alt_group; =20 /* * Since alternative replacement code is copy/pasted by the @@ -1118,6 +1131,10 @@ static int handle_group_alt(struct objto return -1; } =20 + new_alt_group->orig_group =3D orig_alt_group; + new_alt_group->first_insn =3D *new_insn; + new_alt_group->last_insn =3D last_new_insn; + if (fake_jump) list_add(&fake_jump->list, &last_new_insn->list); =20 @@ -2440,7 +2457,7 @@ static int validate_return(struct symbol static void fill_alternative_cfi(struct objtool_file *file, struct instruc= tion *insn) { struct instruction *first_insn =3D insn; - int alt_group =3D insn->alt_group; + struct alt_group *alt_group =3D insn->alt_group; =20 sec_for_each_insn_continue(file, insn) { if (insn->alt_group !=3D alt_group) --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -19,6 +19,17 @@ struct insn_state { s8 instr; }; =20 +struct alt_group { + /* + * Pointer from a replacement group to the original group. NULL if it + * *is* the original group. + */ + struct alt_group *orig_group; + + /* First and last instructions in the group */ + struct instruction *first_insn, *last_insn; +}; + struct instruction { struct list_head list; struct hlist_node hash; @@ -34,7 +45,7 @@ struct instruction { s8 instr; u8 visited; u8 ret_offset; - int alt_group; + struct alt_group *alt_group; struct symbol *call_dest; struct instruction *jump_dest; struct instruction *first_jump_src; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2D04C433EF for ; Sat, 23 Jul 2022 09:56:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237451AbiGWJ4z (ORCPT ); Sat, 23 Jul 2022 05:56:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237388AbiGWJ4p (ORCPT ); Sat, 23 Jul 2022 05:56:45 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DBC85205C5; Sat, 23 Jul 2022 02:56:38 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5D89360C81; Sat, 23 Jul 2022 09:56:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 618DEC341C7; Sat, 23 Jul 2022 09:56:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570197; bh=EtTFjG7OnojSH91UoiB/6KIGcEoUlGrFdu4JHLTygf8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vtZD+CztOvBbDTiswstS9pmUSIK41kZCeHvNuAxIwAaKpoVRj2n7VfRMRARlTA3oZ pne6y+t0x5Sx88039ttirOMNVuWtON4rieIh53f+T8vx5OzgjpsXK81ZuXncoyGE5i h4RLFI7ZAVP+v5+gk69dGARwHNYhYfqkm+yNBi9Y= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Shinichiro Kawasaki , Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 005/148] objtool: Support stack layout changes in alternatives Date: Sat, 23 Jul 2022 11:53:37 +0200 Message-Id: <20220723095225.884978511@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit c9c324dc22aab1687da37001b321b6dfa93a0699 upstream. The ORC unwinder showed a warning [1] which revealed the stack layout didn't match what was expected. The problem was that paravirt patching had replaced "CALL *pv_ops.irq.save_fl" with "PUSHF;POP". That changed the stack layout between the PUSHF and the POP, so unwinding from an interrupt which occurred between those two instructions would fail. Part of the agreed upon solution was to rework the custom paravirt patching code to use alternatives instead, since objtool already knows how to read alternatives (and converging runtime patching infrastructure is always a good thing anyway). But the main problem still remains, which is that runtime patching can change the stack layout. Making stack layout changes in alternatives was disallowed with commit 7117f16bf460 ("objtool: Fix ORC vs alternatives"), but now that paravirt is going to be doing it, it needs to be supported. One way to do so would be to modify the ORC table when the code gets patched. But ORC is simple -- a good thing! -- and it's best to leave it alone. Instead, support stack layout changes by "flattening" all possible stack states (CFI) from parallel alternative code streams into a single set of linear states. The only necessary limitation is that CFI conflicts are disallowed at all possible instruction boundaries. For example, this scenario is allowed: Alt1 Alt2 Alt3 0x00 CALL *pv_ops.save_fl CALL xen_save_fl PUSHF 0x01 POP %RAX 0x02 NOP ... 0x05 NOP ... 0x07 The unwind information for offset-0x00 is identical for all 3 alternatives. Similarly offset-0x05 and higher also are identical (and the same as 0x00). However offset-0x01 has deviating CFI, but that is only relevant for Alt3, neither of the other alternative instruction streams will ever hit that offset. This scenario is NOT allowed: Alt1 Alt2 0x00 CALL *pv_ops.save_fl PUSHF 0x01 NOP6 ... 0x07 NOP POP %RAX The problem here is that offset-0x7, which is an instruction boundary in both possible instruction patch streams, has two conflicting stack layouts. [ The above examples were stolen from Peter Zijlstra. ] The new flattened CFI array is used both for the detection of conflicts (like the second example above) and the generation of linear ORC entries. BTW, another benefit of these changes is that, thanks to some related cleanups (new fake nops and alt_group struct) objtool can finally be rid of fake jumps, which were a constant source of headaches. [1] https://lkml.kernel.org/r/20201111170536.arx2zbn4ngvjoov7@treble Cc: Shinichiro Kawasaki Signed-off-by: Josh Poimboeuf Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/Documentation/stack-validation.txt | 14 - tools/objtool/check.c | 196 +++++++++++-------= ----- tools/objtool/check.h | 6=20 tools/objtool/orc_gen.c | 56 +++++- 4 files changed, 160 insertions(+), 112 deletions(-) --- a/tools/objtool/Documentation/stack-validation.txt +++ b/tools/objtool/Documentation/stack-validation.txt @@ -315,13 +315,15 @@ they mean, and suggestions for how to fi function tracing inserts additional calls, which is not obvious from= the sources). =20 -10. file.o: warning: func()+0x5c: alternative modifies stack +10. file.o: warning: func()+0x5c: stack layout conflict in alternatives =20 - This means that an alternative includes instructions that modify the - stack. The problem is that there is only one ORC unwind table, this me= ans - that the ORC unwind entries must be valid for each of the alternatives. - The easiest way to enforce this is to ensure alternatives do not conta= in - any ORC entries, which in turn implies the above constraint. + This means that in the use of the alternative() or ALTERNATIVE() + macro, the code paths have conflicting modifications to the stack. + The problem is that there is only one ORC unwind table, which means + that the ORC unwind entries must be consistent for all possible + instruction boundaries regardless of which code has been patched. + This limitation can be overcome by massaging the alternatives with + NOPs to shift the stack changes around so they no longer conflict. =20 11. file.o: warning: unannotated intra-function call =20 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -19,8 +19,6 @@ #include #include =20 -#define FAKE_JUMP_OFFSET -1 - struct alternative { struct list_head list; struct instruction *insn; @@ -789,9 +787,6 @@ static int add_jump_destinations(struct if (!is_static_jump(insn)) continue; =20 - if (insn->offset =3D=3D FAKE_JUMP_OFFSET) - continue; - reloc =3D find_reloc_by_dest_range(file->elf, insn->sec, insn->offset, insn->len); if (!reloc) { @@ -991,28 +986,15 @@ static int add_call_destinations(struct } =20 /* - * The .alternatives section requires some extra special care, over and ab= ove - * what other special sections require: - * - * 1. Because alternatives are patched in-place, we need to insert a fake = jump - * instruction at the end so that validate_branch() skips all the origi= nal - * replaced instructions when validating the new instruction path. - * - * 2. An added wrinkle is that the new instruction length might be zero. = In - * that case the old instructions are replaced with noops. We simulate= that - * by creating a fake jump as the only new instruction. - * - * 3. In some cases, the alternative section includes an instruction which - * conditionally jumps to the _end_ of the entry. We have to modify th= ese - * jumps' destinations to point back to .text rather than the end of the - * entry in .altinstr_replacement. + * The .alternatives section requires some extra special care over and abo= ve + * other special sections because alternatives are patched in place. */ static int handle_group_alt(struct objtool_file *file, struct special_alt *special_alt, struct instruction *orig_insn, struct instruction **new_insn) { - struct instruction *last_orig_insn, *last_new_insn, *insn, *fake_jump =3D= NULL; + struct instruction *last_orig_insn, *last_new_insn =3D NULL, *insn, *nop = =3D NULL; struct alt_group *orig_alt_group, *new_alt_group; unsigned long dest_off; =20 @@ -1022,6 +1004,13 @@ static int handle_group_alt(struct objto WARN("malloc failed"); return -1; } + orig_alt_group->cfi =3D calloc(special_alt->orig_len, + sizeof(struct cfi_state *)); + if (!orig_alt_group->cfi) { + WARN("calloc failed"); + return -1; + } + last_orig_insn =3D NULL; insn =3D orig_insn; sec_for_each_insn_from(file, insn) { @@ -1035,42 +1024,45 @@ static int handle_group_alt(struct objto orig_alt_group->first_insn =3D orig_insn; orig_alt_group->last_insn =3D last_orig_insn; =20 - if (next_insn_same_sec(file, last_orig_insn)) { - fake_jump =3D malloc(sizeof(*fake_jump)); - if (!fake_jump) { - WARN("malloc failed"); - return -1; - } - memset(fake_jump, 0, sizeof(*fake_jump)); - INIT_LIST_HEAD(&fake_jump->alts); - INIT_LIST_HEAD(&fake_jump->stack_ops); - init_cfi_state(&fake_jump->cfi); - - fake_jump->sec =3D special_alt->new_sec; - fake_jump->offset =3D FAKE_JUMP_OFFSET; - fake_jump->type =3D INSN_JUMP_UNCONDITIONAL; - fake_jump->jump_dest =3D list_next_entry(last_orig_insn, list); - fake_jump->func =3D orig_insn->func; + + new_alt_group =3D malloc(sizeof(*new_alt_group)); + if (!new_alt_group) { + WARN("malloc failed"); + return -1; } =20 - if (!special_alt->new_len) { - if (!fake_jump) { - WARN("%s: empty alternative at end of section", - special_alt->orig_sec->name); + if (special_alt->new_len < special_alt->orig_len) { + /* + * Insert a fake nop at the end to make the replacement + * alt_group the same size as the original. This is needed to + * allow propagate_alt_cfi() to do its magic. When the last + * instruction affects the stack, the instruction after it (the + * nop) will propagate the new state to the shared CFI array. + */ + nop =3D malloc(sizeof(*nop)); + if (!nop) { + WARN("malloc failed"); return -1; } - - *new_insn =3D fake_jump; - return 0; + memset(nop, 0, sizeof(*nop)); + INIT_LIST_HEAD(&nop->alts); + INIT_LIST_HEAD(&nop->stack_ops); + init_cfi_state(&nop->cfi); + + nop->sec =3D special_alt->new_sec; + nop->offset =3D special_alt->new_off + special_alt->new_len; + nop->len =3D special_alt->orig_len - special_alt->new_len; + nop->type =3D INSN_NOP; + nop->func =3D orig_insn->func; + nop->alt_group =3D new_alt_group; + nop->ignore =3D orig_insn->ignore_alts; } =20 - new_alt_group =3D malloc(sizeof(*new_alt_group)); - if (!new_alt_group) { - WARN("malloc failed"); - return -1; + if (!special_alt->new_len) { + *new_insn =3D nop; + goto end; } =20 - last_new_insn =3D NULL; insn =3D *new_insn; sec_for_each_insn_from(file, insn) { struct reloc *alt_reloc; @@ -1109,14 +1101,8 @@ static int handle_group_alt(struct objto continue; =20 dest_off =3D arch_jump_destination(insn); - if (dest_off =3D=3D special_alt->new_off + special_alt->new_len) { - if (!fake_jump) { - WARN("%s: alternative jump to end of section", - special_alt->orig_sec->name); - return -1; - } - insn->jump_dest =3D fake_jump; - } + if (dest_off =3D=3D special_alt->new_off + special_alt->new_len) + insn->jump_dest =3D next_insn_same_sec(file, last_orig_insn); =20 if (!insn->jump_dest) { WARN_FUNC("can't find alternative jump destination", @@ -1131,13 +1117,13 @@ static int handle_group_alt(struct objto return -1; } =20 + if (nop) + list_add(&nop->list, &last_new_insn->list); +end: new_alt_group->orig_group =3D orig_alt_group; new_alt_group->first_insn =3D *new_insn; - new_alt_group->last_insn =3D last_new_insn; - - if (fake_jump) - list_add(&fake_jump->list, &last_new_insn->list); - + new_alt_group->last_insn =3D nop ? : last_new_insn; + new_alt_group->cfi =3D orig_alt_group->cfi; return 0; } =20 @@ -2237,22 +2223,47 @@ static int update_cfi_state(struct instr return 0; } =20 -static int handle_insn_ops(struct instruction *insn, struct insn_state *st= ate) +/* + * The stack layouts of alternatives instructions can sometimes diverge wh= en + * they have stack modifications. That's fine as long as the potential st= ack + * layouts don't conflict at any given potential instruction boundary. + * + * Flatten the CFIs of the different alternative code streams (both origin= al + * and replacement) into a single shared CFI array which can be used to de= tect + * conflicts and nicely feed a linear array of ORC entries to the unwinder. + */ +static int propagate_alt_cfi(struct objtool_file *file, struct instruction= *insn) { - struct stack_op *op; + struct cfi_state **alt_cfi; + int group_off; =20 - list_for_each_entry(op, &insn->stack_ops, list) { - struct cfi_state old_cfi =3D state->cfi; - int res; + if (!insn->alt_group) + return 0; =20 - res =3D update_cfi_state(insn, &state->cfi, op); - if (res) - return res; + alt_cfi =3D insn->alt_group->cfi; + group_off =3D insn->offset - insn->alt_group->first_insn->offset; =20 - if (insn->alt_group && memcmp(&state->cfi, &old_cfi, sizeof(struct cfi_s= tate))) { - WARN_FUNC("alternative modifies stack", insn->sec, insn->offset); + if (!alt_cfi[group_off]) { + alt_cfi[group_off] =3D &insn->cfi; + } else { + if (memcmp(alt_cfi[group_off], &insn->cfi, sizeof(struct cfi_state))) { + WARN_FUNC("stack layout conflict in alternatives", + insn->sec, insn->offset); return -1; } + } + + return 0; +} + +static int handle_insn_ops(struct instruction *insn, struct insn_state *st= ate) +{ + struct stack_op *op; + + list_for_each_entry(op, &insn->stack_ops, list) { + + if (update_cfi_state(insn, &state->cfi, op)) + return 1; =20 if (op->dest.type =3D=3D OP_DEST_PUSHF) { if (!state->uaccess_stack) { @@ -2442,28 +2453,20 @@ static int validate_return(struct symbol return 0; } =20 -/* - * Alternatives should not contain any ORC entries, this in turn means they - * should not contain any CFI ops, which implies all instructions should h= ave - * the same same CFI state. - * - * It is possible to constuct alternatives that have unreachable holes tha= t go - * unreported (because they're NOPs), such holes would result in CFI_UNDEF= INED - * states which then results in ORC entries, which we just said we didn't = want. - * - * Avoid them by copying the CFI entry of the first instruction into the w= hole - * alternative. - */ -static void fill_alternative_cfi(struct objtool_file *file, struct instruc= tion *insn) +static struct instruction *next_insn_to_validate(struct objtool_file *file, + struct instruction *insn) { - struct instruction *first_insn =3D insn; struct alt_group *alt_group =3D insn->alt_group; =20 - sec_for_each_insn_continue(file, insn) { - if (insn->alt_group !=3D alt_group) - break; - insn->cfi =3D first_insn->cfi; - } + /* + * Simulate the fact that alternatives are patched in-place. When the + * end of a replacement alt_group is reached, redirect objtool flow to + * the end of the original alt_group. + */ + if (alt_group && insn =3D=3D alt_group->last_insn && alt_group->orig_grou= p) + return next_insn_same_sec(file, alt_group->orig_group->last_insn); + + return next_insn_same_sec(file, insn); } =20 /* @@ -2484,7 +2487,7 @@ static int validate_branch(struct objtoo sec =3D insn->sec; =20 while (1) { - next_insn =3D next_insn_same_sec(file, insn); + next_insn =3D next_insn_to_validate(file, insn); =20 if (file->c_file && func && insn->func && func !=3D insn->func->pfunc) { WARN("%s() falls through to next function %s()", @@ -2517,6 +2520,9 @@ static int validate_branch(struct objtoo =20 insn->visited |=3D visited; =20 + if (propagate_alt_cfi(file, insn)) + return 1; + if (!insn->ignore_alts && !list_empty(&insn->alts)) { bool skip_orig =3D false; =20 @@ -2532,9 +2538,6 @@ static int validate_branch(struct objtoo } } =20 - if (insn->alt_group) - fill_alternative_cfi(file, insn); - if (skip_orig) return 0; } @@ -2767,9 +2770,6 @@ static bool ignore_unreachable_insn(stru !strcmp(insn->sec->name, ".altinstr_aux")) return true; =20 - if (insn->type =3D=3D INSN_JUMP_UNCONDITIONAL && insn->offset =3D=3D FAKE= _JUMP_OFFSET) - return true; - if (!insn->func) return false; =20 --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -28,6 +28,12 @@ struct alt_group { =20 /* First and last instructions in the group */ struct instruction *first_insn, *last_insn; + + /* + * Byte-offset-addressed len-sized array of pointers to CFI structs. + * This is shared with the other alt_groups in the same alternative. + */ + struct cfi_state **cfi; }; =20 struct instruction { --- a/tools/objtool/orc_gen.c +++ b/tools/objtool/orc_gen.c @@ -141,6 +141,13 @@ static int orc_list_add(struct list_head return 0; } =20 +static unsigned long alt_group_len(struct alt_group *alt_group) +{ + return alt_group->last_insn->offset + + alt_group->last_insn->len - + alt_group->first_insn->offset; +} + int orc_create(struct objtool_file *file) { struct section *sec, *ip_rsec, *orc_sec; @@ -165,15 +172,48 @@ int orc_create(struct objtool_file *file continue; =20 sec_for_each_insn(file, sec, insn) { - if (init_orc_entry(&orc, &insn->cfi)) - return -1; - if (!memcmp(&prev_orc, &orc, sizeof(orc))) + struct alt_group *alt_group =3D insn->alt_group; + int i; + + if (!alt_group) { + if (init_orc_entry(&orc, &insn->cfi)) + return -1; + if (!memcmp(&prev_orc, &orc, sizeof(orc))) + continue; + if (orc_list_add(&orc_list, &orc, sec, + insn->offset)) + return -1; + nr++; + prev_orc =3D orc; + empty =3D false; continue; - if (orc_list_add(&orc_list, &orc, sec, insn->offset)) - return -1; - nr++; - prev_orc =3D orc; - empty =3D false; + } + + /* + * Alternatives can have different stack layout + * possibilities (but they shouldn't conflict). + * Instead of traversing the instructions, use the + * alt_group's flattened byte-offset-addressed CFI + * array. + */ + for (i =3D 0; i < alt_group_len(alt_group); i++) { + struct cfi_state *cfi =3D alt_group->cfi[i]; + if (!cfi) + continue; + if (init_orc_entry(&orc, cfi)) + return -1; + if (!memcmp(&prev_orc, &orc, sizeof(orc))) + continue; + if (orc_list_add(&orc_list, &orc, insn->sec, + insn->offset + i)) + return -1; + nr++; + prev_orc =3D orc; + empty =3D false; + } + + /* Skip to the end of the alt_group */ + insn =3D alt_group->last_insn; } =20 /* Add a section terminator */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5013C43334 for ; Sat, 23 Jul 2022 09:57:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237473AbiGWJ5A (ORCPT ); Sat, 23 Jul 2022 05:57:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237353AbiGWJ4r (ORCPT ); Sat, 23 Jul 2022 05:56:47 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A57BB3C8D7; Sat, 23 Jul 2022 02:56:41 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 28C75611CD; Sat, 23 Jul 2022 09:56:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 32FD0C341C0; Sat, 23 Jul 2022 09:56:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570200; bh=7XZ89n0r+YQreOjHqIUzWGHyoEFUcZJ1T7G3vKlv2/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Tc585F74iUpp1nsFKKHfzuneosqr4RTs2E97jTJAu8++GSYecgqnhKEc56SeWckIn kLeaYzdIfMAsHxhmjPCNjShBLP9SzZvxpUdeJkgVSN77Pjg8Bfsm7yqePHFbK2KxiB JW1rG/xmsRDZf62AZNjZkRoL123WvrLSF4eWhglQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 006/148] objtool: Support retpoline jump detection for vmlinux.o Date: Sat, 23 Jul 2022 11:53:38 +0200 Message-Id: <20220723095226.156267320@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 31a7424bc58063a8e0466c3c10f31a52ec2be4f6 upstream. Objtool converts direct retpoline jumps to type INSN_JUMP_DYNAMIC, since that's what they are semantically. That conversion doesn't work in vmlinux.o validation because the indirect thunk function is present in the object, so the intra-object jump check succeeds before the retpoline jump check gets a chance. Rearrange the checks: check for a retpoline jump before checking for an intra-object jump. Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/4302893513770dde68ddc22a9d6a2a04aca491dd.16= 11263461.git.jpoimboe@redhat.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -795,10 +795,6 @@ static int add_jump_destinations(struct } else if (reloc->sym->type =3D=3D STT_SECTION) { dest_sec =3D reloc->sym->sec; dest_off =3D arch_dest_reloc_offset(reloc->addend); - } else if (reloc->sym->sec->idx) { - dest_sec =3D reloc->sym->sec; - dest_off =3D reloc->sym->sym.st_value + - arch_dest_reloc_offset(reloc->addend); } else if (!strncmp(reloc->sym->name, "__x86_indirect_thunk_", 21) || !strncmp(reloc->sym->name, "__x86_retpoline_", 16)) { /* @@ -812,6 +808,10 @@ static int add_jump_destinations(struct =20 insn->retpoline_safe =3D true; continue; + } else if (reloc->sym->sec->idx) { + dest_sec =3D reloc->sym->sec; + dest_off =3D reloc->sym->sym.st_value + + arch_dest_reloc_offset(reloc->addend); } else { /* external sibling call */ insn->call_dest =3D reloc->sym; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A4AAC43334 for ; Sat, 23 Jul 2022 09:57:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237504AbiGWJ5M (ORCPT ); Sat, 23 Jul 2022 05:57:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237385AbiGWJ4s (ORCPT ); Sat, 23 Jul 2022 05:56:48 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69E8A3A4AA; Sat, 23 Jul 2022 02:56:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F01906116A; Sat, 23 Jul 2022 09:56:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07BE0C341C7; Sat, 23 Jul 2022 09:56:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570203; bh=KOPBfmIxu9SkTNKcXX0nwvznMmnj5WtPQ0BxdN1yLmw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bHpz6UKalOirJcTPZZ48QuK+UTUOz/XG1hHLq8Hix8GcE1MTqX5lj6A1T37FuRDo8 46R1mUJSu6ZqmeQoVRGX5QvNdHSJU6qozanGkhnEthwF8FxEBNYPneETW5FhGRsqpo YzcYuFpn5N31NHxPJv9Ttd+HahXYJi6sPwDDyhJU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 007/148] objtool: Assume only ELF functions do sibling calls Date: Sat, 23 Jul 2022 11:53:39 +0200 Message-Id: <20220723095226.474288186@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit ecf11ba4d066fe527586c6edd6ca68457ca55cf4 upstream. There's an inconsistency in how sibling calls are detected in non-function asm code, depending on the scope of the object. If the target code is external to the object, objtool considers it a sibling call. If the target code is internal but not a function, objtool *doesn't* consider it a sibling call. This can cause some inconsistencies between per-object and vmlinux.o validation. Instead, assume only ELF functions can do sibling calls. This generally matches existing reality, and makes sibling call validation consistent between vmlinux.o and per-object. Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/0e9ab6f3628cc7bf3bde7aa6762d54d7df19ad78.16= 11263461.git.jpoimboe@redhat.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -109,15 +109,20 @@ static struct instruction *prev_insn_sam =20 static bool is_sibling_call(struct instruction *insn) { + /* + * Assume only ELF functions can make sibling calls. This ensures + * sibling call detection consistency between vmlinux.o and individual + * objects. + */ + if (!insn->func) + return false; + /* An indirect jump is either a sibling call or a jump to a table. */ if (insn->type =3D=3D INSN_JUMP_DYNAMIC) return list_empty(&insn->alts); =20 - if (!is_static_jump(insn)) - return false; - /* add_jump_destinations() sets insn->call_dest for sibling calls. */ - return !!insn->call_dest; + return (is_static_jump(insn) && insn->call_dest); } =20 /* @@ -788,7 +793,7 @@ static int add_jump_destinations(struct continue; =20 reloc =3D find_reloc_by_dest_range(file->elf, insn->sec, - insn->offset, insn->len); + insn->offset, insn->len); if (!reloc) { dest_sec =3D insn->sec; dest_off =3D arch_jump_destination(insn); @@ -808,18 +813,21 @@ static int add_jump_destinations(struct =20 insn->retpoline_safe =3D true; continue; - } else if (reloc->sym->sec->idx) { - dest_sec =3D reloc->sym->sec; - dest_off =3D reloc->sym->sym.st_value + - arch_dest_reloc_offset(reloc->addend); - } else { - /* external sibling call */ + } else if (insn->func) { + /* internal or external sibling call (with reloc) */ insn->call_dest =3D reloc->sym; if (insn->call_dest->static_call_tramp) { list_add_tail(&insn->static_call_node, &file->static_call_list); } continue; + } else if (reloc->sym->sec->idx) { + dest_sec =3D reloc->sym->sec; + dest_off =3D reloc->sym->sym.st_value + + arch_dest_reloc_offset(reloc->addend); + } else { + /* non-func asm code jumping to another file */ + continue; } =20 insn->jump_dest =3D find_insn(file, dest_sec, dest_off); @@ -868,7 +876,7 @@ static int add_jump_destinations(struct } else if (insn->jump_dest->func->pfunc !=3D insn->func->pfunc && insn->jump_dest->offset =3D=3D insn->jump_dest->func->offset) { =20 - /* internal sibling call */ + /* internal sibling call (without reloc) */ insn->call_dest =3D insn->jump_dest->func; if (insn->call_dest->static_call_tramp) { list_add_tail(&insn->static_call_node, @@ -2570,7 +2578,7 @@ static int validate_branch(struct objtoo =20 case INSN_JUMP_CONDITIONAL: case INSN_JUMP_UNCONDITIONAL: - if (func && is_sibling_call(insn)) { + if (is_sibling_call(insn)) { ret =3D validate_sibling_call(insn, &state); if (ret) return ret; @@ -2592,7 +2600,7 @@ static int validate_branch(struct objtoo =20 case INSN_JUMP_DYNAMIC: case INSN_JUMP_DYNAMIC_CONDITIONAL: - if (func && is_sibling_call(insn)) { + if (is_sibling_call(insn)) { ret =3D validate_sibling_call(insn, &state); if (ret) return ret; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E19CC43334 for ; Sat, 23 Jul 2022 09:57:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237523AbiGWJ5O (ORCPT ); Sat, 23 Jul 2022 05:57:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237398AbiGWJ4y (ORCPT ); Sat, 23 Jul 2022 05:56:54 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E4E23B948; Sat, 23 Jul 2022 02:56:47 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D42486116A; Sat, 23 Jul 2022 09:56:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E021EC341C0; Sat, 23 Jul 2022 09:56:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570206; bh=+THqdilnftkwgLT3AbiDopT9tOjoWmndlCmioZBsvV0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jHkhlYg0LF9+r1saA9g6jbJCJYzFtqt+DyUx0jCiPvIS9Jkv8ohuUYZpsSU+wNbHg yoYwwFL5dMKpnJubWPMgV0BHdZLa/4gbTTohBVIvoamrj6BSOlWDvqMV8PIE9C6Uk8 E9IxZ9OiXUBw8y0QOv24mxwlU+/AHOXqIrqU/JgI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 008/148] objtool: Combine UNWIND_HINT_RET_OFFSET and UNWIND_HINT_FUNC Date: Sat, 23 Jul 2022 11:53:40 +0200 Message-Id: <20220723095226.748134676@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit b735bd3e68824316655252a931a3353a6ebc036f upstream. The ORC metadata generated for UNWIND_HINT_FUNC isn't actually very func-like. With certain usages it can cause stack state mismatches because it doesn't set the return address (CFI_RA). Also, users of UNWIND_HINT_RET_OFFSET no longer need to set a custom return stack offset. Instead they just need to specify a func-like situation, so the current ret_offset code is hacky for no good reason. Solve both problems by simplifying the RET_OFFSET handling and converting it into a more useful UNWIND_HINT_FUNC. If we end up needing the old 'ret_offset' functionality again in the future, we should be able to support it pretty easily with the addition of a custom 'sp_offset' in UNWIND_HINT_FUNC. Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/db9d1f5d79dddfbb3725ef6d8ec3477ad199948d.16= 11263462.git.jpoimboe@redhat.com [bwh: Backported to 5.10: - Don't use bswap_if_needed() since we don't have any of the other fixes for mixed-endian cross-compilation - Adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/unwind_hints.h | 13 +----------- arch/x86/kernel/ftrace_64.S | 2 - arch/x86/lib/retpoline.S | 2 - include/linux/objtool.h | 5 +++- tools/include/linux/objtool.h | 5 +++- tools/objtool/arch/x86/decode.c | 4 +-- tools/objtool/check.c | 37 ++++++++++++++-----------------= ----- tools/objtool/check.h | 1=20 8 files changed, 29 insertions(+), 40 deletions(-) --- a/arch/x86/include/asm/unwind_hints.h +++ b/arch/x86/include/asm/unwind_hints.h @@ -48,17 +48,8 @@ UNWIND_HINT_REGS base=3D\base offset=3D\offset partial=3D1 .endm =20 -.macro UNWIND_HINT_FUNC sp_offset=3D8 - UNWIND_HINT sp_reg=3DORC_REG_SP sp_offset=3D\sp_offset type=3DUNWIND_HINT= _TYPE_CALL -.endm - -/* - * RET_OFFSET: Used on instructions that terminate a function; mostly RETU= RN - * and sibling calls. On these, sp_offset denotes the expected offset from - * initial_func_cfi. - */ -.macro UNWIND_HINT_RET_OFFSET sp_offset=3D8 - UNWIND_HINT sp_reg=3DORC_REG_SP type=3DUNWIND_HINT_TYPE_RET_OFFSET sp_off= set=3D\sp_offset +.macro UNWIND_HINT_FUNC + UNWIND_HINT sp_reg=3DORC_REG_SP sp_offset=3D8 type=3DUNWIND_HINT_TYPE_FUNC .endm =20 #endif /* __ASSEMBLY__ */ --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -265,7 +265,7 @@ SYM_INNER_LABEL(ftrace_regs_caller_end, restore_mcount_regs 8 /* Restore flags */ popfq - UNWIND_HINT_RET_OFFSET + UNWIND_HINT_FUNC jmp ftrace_epilogue =20 SYM_FUNC_END(ftrace_regs_caller) --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -28,7 +28,7 @@ SYM_FUNC_START_NOALIGN(__x86_retpoline_\ jmp .Lspec_trap_\@ .Ldo_rop_\@: mov %\reg, (%_ASM_SP) - UNWIND_HINT_RET_OFFSET + UNWIND_HINT_FUNC ret SYM_FUNC_END(__x86_retpoline_\reg) =20 --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -29,11 +29,14 @@ struct unwind_hint { * * UNWIND_HINT_TYPE_REGS_PARTIAL: Used in entry code to indicate that * sp_reg+sp_offset points to the iret return frame. + * + * UNWIND_HINT_FUNC: Generate the unwind metadata of a callable function. + * Useful for code which doesn't have an ELF function annotation. */ #define UNWIND_HINT_TYPE_CALL 0 #define UNWIND_HINT_TYPE_REGS 1 #define UNWIND_HINT_TYPE_REGS_PARTIAL 2 -#define UNWIND_HINT_TYPE_RET_OFFSET 3 +#define UNWIND_HINT_TYPE_FUNC 3 =20 #ifdef CONFIG_STACK_VALIDATION =20 --- a/tools/include/linux/objtool.h +++ b/tools/include/linux/objtool.h @@ -29,11 +29,14 @@ struct unwind_hint { * * UNWIND_HINT_TYPE_REGS_PARTIAL: Used in entry code to indicate that * sp_reg+sp_offset points to the iret return frame. + * + * UNWIND_HINT_FUNC: Generate the unwind metadata of a callable function. + * Useful for code which doesn't have an ELF function annotation. */ #define UNWIND_HINT_TYPE_CALL 0 #define UNWIND_HINT_TYPE_REGS 1 #define UNWIND_HINT_TYPE_REGS_PARTIAL 2 -#define UNWIND_HINT_TYPE_RET_OFFSET 3 +#define UNWIND_HINT_TYPE_FUNC 3 =20 #ifdef CONFIG_STACK_VALIDATION =20 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -563,8 +563,8 @@ void arch_initial_func_cfi_state(struct state->cfa.offset =3D 8; =20 /* initial RA (return address) */ - state->regs[16].base =3D CFI_CFA; - state->regs[16].offset =3D -8; + state->regs[CFI_RA].base =3D CFI_CFA; + state->regs[CFI_RA].offset =3D -8; } =20 const char *arch_nop_insn(int len) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1423,13 +1423,20 @@ static int add_jump_table_alts(struct ob return 0; } =20 +static void set_func_state(struct cfi_state *state) +{ + state->cfa =3D initial_func_cfi.cfa; + memcpy(&state->regs, &initial_func_cfi.regs, + CFI_NUM_REGS * sizeof(struct cfi_reg)); + state->stack_size =3D initial_func_cfi.cfa.offset; +} + static int read_unwind_hints(struct objtool_file *file) { struct section *sec, *relocsec; struct reloc *reloc; struct unwind_hint *hint; struct instruction *insn; - struct cfi_reg *cfa; int i; =20 sec =3D find_section_by_name(file->elf, ".discard.unwind_hints"); @@ -1464,22 +1471,20 @@ static int read_unwind_hints(struct objt return -1; } =20 - cfa =3D &insn->cfi.cfa; + insn->hint =3D true; =20 - if (hint->type =3D=3D UNWIND_HINT_TYPE_RET_OFFSET) { - insn->ret_offset =3D hint->sp_offset; + if (hint->type =3D=3D UNWIND_HINT_TYPE_FUNC) { + set_func_state(&insn->cfi); continue; } =20 - insn->hint =3D true; - if (arch_decode_hint_reg(insn, hint->sp_reg)) { WARN_FUNC("unsupported unwind_hint sp base reg %d", insn->sec, insn->offset, hint->sp_reg); return -1; } =20 - cfa->offset =3D hint->sp_offset; + insn->cfi.cfa.offset =3D hint->sp_offset; insn->cfi.type =3D hint->type; insn->cfi.end =3D hint->end; } @@ -1742,27 +1747,18 @@ static bool is_fentry_call(struct instru =20 static bool has_modified_stack_frame(struct instruction *insn, struct insn= _state *state) { - u8 ret_offset =3D insn->ret_offset; struct cfi_state *cfi =3D &state->cfi; int i; =20 if (cfi->cfa.base !=3D initial_func_cfi.cfa.base || cfi->drap) return true; =20 - if (cfi->cfa.offset !=3D initial_func_cfi.cfa.offset + ret_offset) + if (cfi->cfa.offset !=3D initial_func_cfi.cfa.offset) return true; =20 - if (cfi->stack_size !=3D initial_func_cfi.cfa.offset + ret_offset) + if (cfi->stack_size !=3D initial_func_cfi.cfa.offset) return true; =20 - /* - * If there is a ret offset hint then don't check registers - * because a callee-saved register might have been pushed on - * the stack. - */ - if (ret_offset) - return false; - for (i =3D 0; i < CFI_NUM_REGS; i++) { if (cfi->regs[i].base !=3D initial_func_cfi.regs[i].base || cfi->regs[i].offset !=3D initial_func_cfi.regs[i].offset) @@ -2863,10 +2859,7 @@ static int validate_section(struct objto continue; =20 init_insn_state(&state, sec); - state.cfi.cfa =3D initial_func_cfi.cfa; - memcpy(&state.cfi.regs, &initial_func_cfi.regs, - CFI_NUM_REGS * sizeof(struct cfi_reg)); - state.cfi.stack_size =3D initial_func_cfi.cfa.offset; + set_func_state(&state.cfi); =20 warnings +=3D validate_symbol(file, sec, func, &state); } --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -50,7 +50,6 @@ struct instruction { bool retpoline_safe; s8 instr; u8 visited; - u8 ret_offset; struct alt_group *alt_group; struct symbol *call_dest; struct instruction *jump_dest; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90A0BC433EF for ; Sat, 23 Jul 2022 09:57:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237553AbiGWJ5T (ORCPT ); Sat, 23 Jul 2022 05:57:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237407AbiGWJ44 (ORCPT ); Sat, 23 Jul 2022 05:56:56 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AC5C3FA0C; Sat, 23 Jul 2022 02:56:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B0725611CD; Sat, 23 Jul 2022 09:56:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B865EC341C0; Sat, 23 Jul 2022 09:56:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570209; bh=A6uIYjf26dxGw4rLoh2QwKZAbV4FlEfHPIwrpmluWPU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FHAxwIr8d+A0p/XVYgjWfyf5BKuh0kU/UxWeOEDF7jeUGxsUKNzAPbeSoliEVWmbu nVkJ26mMLIuKCYOfJZSZbPLDKHk72u7mSAgys/o3rZm68h5eQ4FUEbzzwQqCeeQeTj xdtgqBFSIokFhB6Vox3UjmqkQi5MY/11JwG9xyhA= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Juergen Gross , Boris Ostrovsky , Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 009/148] x86/xen: Support objtool validation in xen-asm.S Date: Sat, 23 Jul 2022 11:53:41 +0200 Message-Id: <20220723095227.051592198@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit cde07a4e4434ddfb9b1616ac971edf6d66329804 upstream. The OBJECT_FILES_NON_STANDARD annotation is used to tell objtool to ignore a file. File-level ignores won't work when validating vmlinux.o. Tweak the ELF metadata and unwind hints to allow objtool to follow the code. Cc: Juergen Gross Reviewed-by: Boris Ostrovsky Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/8b042a09c69e8645f3b133ef6653ba28f896807d.16= 11263462.git.jpoimboe@redhat.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/xen/Makefile | 1 - arch/x86/xen/xen-asm.S | 29 +++++++++++++++++++---------- 2 files changed, 19 insertions(+), 11 deletions(-) --- a/arch/x86/xen/Makefile +++ b/arch/x86/xen/Makefile @@ -1,5 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 -OBJECT_FILES_NON_STANDARD_xen-asm.o :=3D y =20 ifdef CONFIG_FUNCTION_TRACER # Do not profile debug and lowlevel utilities --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -14,6 +14,7 @@ #include #include #include +#include =20 #include =20 @@ -147,6 +148,7 @@ SYM_FUNC_END(xen_read_cr2_direct); =20 .macro xen_pv_trap name SYM_CODE_START(xen_\name) + UNWIND_HINT_EMPTY pop %rcx pop %r11 jmp \name @@ -186,6 +188,7 @@ xen_pv_trap asm_exc_xen_hypervisor_callb SYM_CODE_START(xen_early_idt_handler_array) i =3D 0 .rept NUM_EXCEPTION_VECTORS + UNWIND_HINT_EMPTY pop %rcx pop %r11 jmp early_idt_handler_array + i*EARLY_IDT_HANDLER_SIZE @@ -212,11 +215,13 @@ hypercall_iret =3D hypercall_page + __HYPE * rsp->rax } */ SYM_CODE_START(xen_iret) + UNWIND_HINT_EMPTY pushq $0 jmp hypercall_iret SYM_CODE_END(xen_iret) =20 SYM_CODE_START(xen_sysret64) + UNWIND_HINT_EMPTY /* * We're already on the usermode stack at this point, but * still with the kernel gs, so we can easily switch back. @@ -271,7 +276,8 @@ SYM_CODE_END(xenpv_restore_regs_and_retu */ =20 /* Normal 64-bit system call target */ -SYM_FUNC_START(xen_syscall_target) +SYM_CODE_START(xen_syscall_target) + UNWIND_HINT_EMPTY popq %rcx popq %r11 =20 @@ -284,12 +290,13 @@ SYM_FUNC_START(xen_syscall_target) movq $__USER_CS, 1*8(%rsp) =20 jmp entry_SYSCALL_64_after_hwframe -SYM_FUNC_END(xen_syscall_target) +SYM_CODE_END(xen_syscall_target) =20 #ifdef CONFIG_IA32_EMULATION =20 /* 32-bit compat syscall target */ -SYM_FUNC_START(xen_syscall32_target) +SYM_CODE_START(xen_syscall32_target) + UNWIND_HINT_EMPTY popq %rcx popq %r11 =20 @@ -302,10 +309,11 @@ SYM_FUNC_START(xen_syscall32_target) movq $__USER32_CS, 1*8(%rsp) =20 jmp entry_SYSCALL_compat_after_hwframe -SYM_FUNC_END(xen_syscall32_target) +SYM_CODE_END(xen_syscall32_target) =20 /* 32-bit compat sysenter target */ -SYM_FUNC_START(xen_sysenter_target) +SYM_CODE_START(xen_sysenter_target) + UNWIND_HINT_EMPTY /* * NB: Xen is polite and clears TF from EFLAGS for us. This means * that we don't need to guard against single step exceptions here. @@ -322,17 +330,18 @@ SYM_FUNC_START(xen_sysenter_target) movq $__USER32_CS, 1*8(%rsp) =20 jmp entry_SYSENTER_compat_after_hwframe -SYM_FUNC_END(xen_sysenter_target) +SYM_CODE_END(xen_sysenter_target) =20 #else /* !CONFIG_IA32_EMULATION */ =20 -SYM_FUNC_START_ALIAS(xen_syscall32_target) -SYM_FUNC_START(xen_sysenter_target) +SYM_CODE_START(xen_syscall32_target) +SYM_CODE_START(xen_sysenter_target) + UNWIND_HINT_EMPTY lea 16(%rsp), %rsp /* strip %rcx, %r11 */ mov $-ENOSYS, %rax pushq $0 jmp hypercall_iret -SYM_FUNC_END(xen_sysenter_target) -SYM_FUNC_END_ALIAS(xen_syscall32_target) +SYM_CODE_END(xen_sysenter_target) +SYM_CODE_END(xen_syscall32_target) =20 #endif /* CONFIG_IA32_EMULATION */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C11C2C433EF for ; Sat, 23 Jul 2022 09:59:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237726AbiGWJ7N (ORCPT ); Sat, 23 Jul 2022 05:59:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237524AbiGWJ6g (ORCPT ); Sat, 23 Jul 2022 05:58:36 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63E321ADBD; Sat, 23 Jul 2022 02:57:32 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 16D4CB82C1F; Sat, 23 Jul 2022 09:57:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6757BC341C0; Sat, 23 Jul 2022 09:57:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570249; bh=iodpCEhKE7vLB7Qn5wLtvrYYWXkmr5UUF3gKEJUzpEg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wSTdx66IfIH4etwzBAcZYN3r69Wf8W4jHy98h+id8uOxZLRXkcsyH4VPihvzjPQ6q as29sb93PCJkoZkqchtIAGistfDRhxJmPJXOXoMSYOYu2/GBrwWDh4nyEx0Rd8QO+/ NclGGd5+UYEF+shwGM06dEfE9tfskXMP+oVtL+BU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Juergen Gross , Boris Ostrovsky , Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 010/148] x86/xen: Support objtool vmlinux.o validation in xen-head.S Date: Sat, 23 Jul 2022 11:53:42 +0200 Message-Id: <20220723095227.283321475@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit f4b4bc10b0b85ec66f1a9bf5dddf475e6695b6d2 upstream. The Xen hypercall page is filled with zeros, causing objtool to fall through all the empty hypercall functions until it reaches a real function, resulting in a stack state mismatch. The build-time contents of the hypercall page don't matter because the page gets rewritten by the hypervisor. Make it more palatable to objtool by making each hypervisor function a true empty function, with nops and a return. Cc: Juergen Gross Reviewed-by: Boris Ostrovsky Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/0883bde1d7a1fb3b6a4c952bc0200e873752f609.16= 11263462.git.jpoimboe@redhat.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/xen/xen-head.S | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -68,8 +68,9 @@ SYM_CODE_END(asm_cpu_bringup_and_idle) .balign PAGE_SIZE SYM_CODE_START(hypercall_page) .rept (PAGE_SIZE / 32) - UNWIND_HINT_EMPTY - .skip 32 + UNWIND_HINT_FUNC + .skip 31, 0x90 + ret .endr =20 #define HYPERCALL(n) \ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A4F5C433EF for ; Sat, 23 Jul 2022 09:57:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237519AbiGWJ5f (ORCPT ); Sat, 23 Jul 2022 05:57:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237429AbiGWJ5M (ORCPT ); Sat, 23 Jul 2022 05:57:12 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF3EA3F32F; Sat, 23 Jul 2022 02:56:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 391BDB82C1F; Sat, 23 Jul 2022 09:56:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8A605C341C0; Sat, 23 Jul 2022 09:56:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570214; bh=Giw7ed+DIPgp8ePeOm7IAwsUx0fnFlZ6gBV2EK6t2Zc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H0tIPUmbM7kr/8q1hIpT0FDMpED/j/DMgoOAglWw/ggWj9rdLC1Y4dE71lqQ+GjJo l9QcqARKK7gR3WH+UTk7tPWUcqv2AIOO37nV9Y7P9TQQt2H0CaWtTlYwrOXAu0Er14 5BEPQwzIANvzdvKSZy8jV0aK8yLIwmQY8NXHt1TU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Juergen Gross , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 011/148] x86/alternative: Merge include files Date: Sat, 23 Jul 2022 11:53:43 +0200 Message-Id: <20220723095227.566370762@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Juergen Gross commit 5e21a3ecad1500e35b46701e7f3f232e15d78e69 upstream. Merge arch/x86/include/asm/alternative-asm.h into arch/x86/include/asm/alternative.h in order to make it easier to use common definitions later. Signed-off-by: Juergen Gross Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210311142319.4723-2-jgross@suse.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/entry_32.S | 2=20 arch/x86/entry/vdso/vdso32/system_call.S | 2=20 arch/x86/include/asm/alternative-asm.h | 114 --------------------------= ----- arch/x86/include/asm/alternative.h | 112 ++++++++++++++++++++++++++= +++- arch/x86/include/asm/nospec-branch.h | 1=20 arch/x86/include/asm/smap.h | 5 - arch/x86/lib/atomic64_386_32.S | 2=20 arch/x86/lib/atomic64_cx8_32.S | 2=20 arch/x86/lib/copy_page_64.S | 2=20 arch/x86/lib/copy_user_64.S | 2=20 arch/x86/lib/memcpy_64.S | 2=20 arch/x86/lib/memmove_64.S | 2=20 arch/x86/lib/memset_64.S | 2=20 arch/x86/lib/retpoline.S | 2=20 14 files changed, 120 insertions(+), 132 deletions(-) delete mode 100644 arch/x86/include/asm/alternative-asm.h --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -40,7 +40,7 @@ #include #include #include -#include +#include #include #include #include --- a/arch/x86/entry/vdso/vdso32/system_call.S +++ b/arch/x86/entry/vdso/vdso32/system_call.S @@ -6,7 +6,7 @@ #include #include #include -#include +#include =20 .text .globl __kernel_vsyscall --- a/arch/x86/include/asm/alternative-asm.h +++ /dev/null @@ -1,114 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_X86_ALTERNATIVE_ASM_H -#define _ASM_X86_ALTERNATIVE_ASM_H - -#ifdef __ASSEMBLY__ - -#include - -#ifdef CONFIG_SMP - .macro LOCK_PREFIX -672: lock - .pushsection .smp_locks,"a" - .balign 4 - .long 672b - . - .popsection - .endm -#else - .macro LOCK_PREFIX - .endm -#endif - -/* - * objtool annotation to ignore the alternatives and only consider the ori= ginal - * instruction(s). - */ -.macro ANNOTATE_IGNORE_ALTERNATIVE - .Lannotate_\@: - .pushsection .discard.ignore_alts - .long .Lannotate_\@ - . - .popsection -.endm - -/* - * Issue one struct alt_instr descriptor entry (need to put it into - * the section .altinstructions, see below). This entry contains - * enough information for the alternatives patching code to patch an - * instruction. See apply_alternatives(). - */ -.macro altinstruction_entry orig alt feature orig_len alt_len pad_len - .long \orig - . - .long \alt - . - .word \feature - .byte \orig_len - .byte \alt_len - .byte \pad_len -.endm - -/* - * Define an alternative between two instructions. If @feature is - * present, early code in apply_alternatives() replaces @oldinstr with - * @newinstr. ".skip" directive takes care of proper instruction padding - * in case @newinstr is longer than @oldinstr. - */ -.macro ALTERNATIVE oldinstr, newinstr, feature -140: - \oldinstr -141: - .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstruction_entry 140b,143f,\feature,142b-140b,144f-143f,142b-141b - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr -144: - .popsection -.endm - -#define old_len 141b-140b -#define new_len1 144f-143f -#define new_len2 145f-144f - -/* - * gas compatible max based on the idea from: - * http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax - * - * The additional "-" is needed because gas uses a "true" value of -1. - */ -#define alt_max_short(a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) - - -/* - * Same as ALTERNATIVE macro above but for two alternatives. If CPU - * has @feature1, it replaces @oldinstr with @newinstr1. If CPU has - * @feature2, it replaces @oldinstr with @feature2. - */ -.macro ALTERNATIVE_2 oldinstr, newinstr1, feature1, newinstr2, feature2 -140: - \oldinstr -141: - .skip -((alt_max_short(new_len1, new_len2) - (old_len)) > 0) * \ - (alt_max_short(new_len1, new_len2) - (old_len)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f,142b-141b - altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f,142b-141b - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr1 -144: - \newinstr2 -145: - .popsection -.endm - -#endif /* __ASSEMBLY__ */ - -#endif /* _ASM_X86_ALTERNATIVE_ASM_H */ --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -2,13 +2,14 @@ #ifndef _ASM_X86_ALTERNATIVE_H #define _ASM_X86_ALTERNATIVE_H =20 -#ifndef __ASSEMBLY__ - #include -#include #include #include =20 +#ifndef __ASSEMBLY__ + +#include + /* * Alternative inline assembly for SMP. * @@ -271,6 +272,111 @@ static inline int alternatives_text_rese */ #define ASM_NO_INPUT_CLOBBER(clbr...) "i" (0) : clbr =20 +#else /* __ASSEMBLY__ */ + +#ifdef CONFIG_SMP + .macro LOCK_PREFIX +672: lock + .pushsection .smp_locks,"a" + .balign 4 + .long 672b - . + .popsection + .endm +#else + .macro LOCK_PREFIX + .endm +#endif + +/* + * objtool annotation to ignore the alternatives and only consider the ori= ginal + * instruction(s). + */ +.macro ANNOTATE_IGNORE_ALTERNATIVE + .Lannotate_\@: + .pushsection .discard.ignore_alts + .long .Lannotate_\@ - . + .popsection +.endm + +/* + * Issue one struct alt_instr descriptor entry (need to put it into + * the section .altinstructions, see below). This entry contains + * enough information for the alternatives patching code to patch an + * instruction. See apply_alternatives(). + */ +.macro altinstruction_entry orig alt feature orig_len alt_len pad_len + .long \orig - . + .long \alt - . + .word \feature + .byte \orig_len + .byte \alt_len + .byte \pad_len +.endm + +/* + * Define an alternative between two instructions. If @feature is + * present, early code in apply_alternatives() replaces @oldinstr with + * @newinstr. ".skip" directive takes care of proper instruction padding + * in case @newinstr is longer than @oldinstr. + */ +.macro ALTERNATIVE oldinstr, newinstr, feature +140: + \oldinstr +141: + .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 +142: + + .pushsection .altinstructions,"a" + altinstruction_entry 140b,143f,\feature,142b-140b,144f-143f,142b-141b + .popsection + + .pushsection .altinstr_replacement,"ax" +143: + \newinstr +144: + .popsection +.endm + +#define old_len 141b-140b +#define new_len1 144f-143f +#define new_len2 145f-144f + +/* + * gas compatible max based on the idea from: + * http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax + * + * The additional "-" is needed because gas uses a "true" value of -1. + */ +#define alt_max_short(a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) + + +/* + * Same as ALTERNATIVE macro above but for two alternatives. If CPU + * has @feature1, it replaces @oldinstr with @newinstr1. If CPU has + * @feature2, it replaces @oldinstr with @feature2. + */ +.macro ALTERNATIVE_2 oldinstr, newinstr1, feature1, newinstr2, feature2 +140: + \oldinstr +141: + .skip -((alt_max_short(new_len1, new_len2) - (old_len)) > 0) * \ + (alt_max_short(new_len1, new_len2) - (old_len)),0x90 +142: + + .pushsection .altinstructions,"a" + altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f,142b-141b + altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f,142b-141b + .popsection + + .pushsection .altinstr_replacement,"ax" +143: + \newinstr1 +144: + \newinstr2 +145: + .popsection +.endm + #endif /* __ASSEMBLY__ */ =20 #endif /* _ASM_X86_ALTERNATIVE_H */ --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -7,7 +7,6 @@ #include =20 #include -#include #include #include #include --- a/arch/x86/include/asm/smap.h +++ b/arch/x86/include/asm/smap.h @@ -11,6 +11,7 @@ =20 #include #include +#include =20 /* "Raw" instruction opcodes */ #define __ASM_CLAC ".byte 0x0f,0x01,0xca" @@ -18,8 +19,6 @@ =20 #ifdef __ASSEMBLY__ =20 -#include - #ifdef CONFIG_X86_SMAP =20 #define ASM_CLAC \ @@ -37,8 +36,6 @@ =20 #else /* __ASSEMBLY__ */ =20 -#include - #ifdef CONFIG_X86_SMAP =20 static __always_inline void clac(void) --- a/arch/x86/lib/atomic64_386_32.S +++ b/arch/x86/lib/atomic64_386_32.S @@ -6,7 +6,7 @@ */ =20 #include -#include +#include =20 /* if you want SMP support, implement these with real spinlocks */ .macro LOCK reg --- a/arch/x86/lib/atomic64_cx8_32.S +++ b/arch/x86/lib/atomic64_cx8_32.S @@ -6,7 +6,7 @@ */ =20 #include -#include +#include =20 .macro read64 reg movl %ebx, %eax --- a/arch/x86/lib/copy_page_64.S +++ b/arch/x86/lib/copy_page_64.S @@ -3,7 +3,7 @@ =20 #include #include -#include +#include #include =20 /* --- a/arch/x86/lib/copy_user_64.S +++ b/arch/x86/lib/copy_user_64.S @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include --- a/arch/x86/lib/memcpy_64.S +++ b/arch/x86/lib/memcpy_64.S @@ -4,7 +4,7 @@ #include #include #include -#include +#include #include =20 .pushsection .noinstr.text, "ax" --- a/arch/x86/lib/memmove_64.S +++ b/arch/x86/lib/memmove_64.S @@ -8,7 +8,7 @@ */ #include #include -#include +#include #include =20 #undef memmove --- a/arch/x86/lib/memset_64.S +++ b/arch/x86/lib/memset_64.S @@ -3,7 +3,7 @@ =20 #include #include -#include +#include #include =20 /* --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -4,7 +4,7 @@ #include #include #include -#include +#include #include #include #include From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F88AC43334 for ; Sat, 23 Jul 2022 09:58:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237379AbiGWJ6Q (ORCPT ); Sat, 23 Jul 2022 05:58:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237421AbiGWJ52 (ORCPT ); Sat, 23 Jul 2022 05:57:28 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 422C74B49A; Sat, 23 Jul 2022 02:57:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id AADF6B82C1B; Sat, 23 Jul 2022 09:57:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17B8CC341C0; Sat, 23 Jul 2022 09:57:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570226; bh=edXK0hiq1Nl4cx9AKafrgyyFNZRfcBMbs6qOf68SPA8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YJWz7MbzcVIe9MGiBbrekmDt7qJ5TC7CZU8lUQaGEgy/zw5v3qNy8SZhP1HFplWn8 ta7xQWQQLN3K9SCuw2EoMoiuZQ7g9XSaRdEPcZCcgxEP7M0YQOUVywRf5yIc9fhAxE zqufgsafPrfb7YZfYD3aSfZCzoL/UflY+LRVN/4A= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Juergen Gross , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 012/148] x86/alternative: Support not-feature Date: Sat, 23 Jul 2022 11:53:44 +0200 Message-Id: <20220723095227.812296055@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Juergen Gross commit dda7bb76484978316bb412a353789ebc5901de36 upstream. Add support for alternative patching for the case a feature is not present on the current CPU. For users of ALTERNATIVE() and friends, an inverted feature is specified by applying the ALT_NOT() macro to it, e.g.: ALTERNATIVE(old, new, ALT_NOT(feature)); Committer note: The decision to encode the NOT-bit in the feature bit itself is because a future change which would make objtool generate such alternative calls, would keep the code in objtool itself fairly simple. Also, this allows for the alternative macros to support the NOT feature without having to change them. Finally, the u16 cpuid member encoding the X86_FEATURE_ flags is not an ABI so if more bits are needed, cpuid itself can be enlarged or a flags field can be added to struct alt_instr after having considered the size growth in either cases. Signed-off-by: Juergen Gross Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210311142319.4723-6-jgross@suse.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/alternative.h | 3 +++ arch/x86/kernel/alternative.c | 20 +++++++++++++++----- 2 files changed, 18 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -6,6 +6,9 @@ #include #include =20 +#define ALTINSTR_FLAG_INV (1 << 15) +#define ALT_NOT(feat) ((feat) | ALTINSTR_FLAG_INV) + #ifndef __ASSEMBLY__ =20 #include --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -388,21 +388,31 @@ void __init_or_module noinline apply_alt */ for (a =3D start; a < end; a++) { int insn_buff_sz =3D 0; + /* Mask away "NOT" flag bit for feature to test. */ + u16 feature =3D a->cpuid & ~ALTINSTR_FLAG_INV; =20 instr =3D (u8 *)&a->instr_offset + a->instr_offset; replacement =3D (u8 *)&a->repl_offset + a->repl_offset; BUG_ON(a->instrlen > sizeof(insn_buff)); - BUG_ON(a->cpuid >=3D (NCAPINTS + NBUGINTS) * 32); - if (!boot_cpu_has(a->cpuid)) { + BUG_ON(feature >=3D (NCAPINTS + NBUGINTS) * 32); + + /* + * Patch if either: + * - feature is present + * - feature not present but ALTINSTR_FLAG_INV is set to mean, + * patch if feature is *NOT* present. + */ + if (!boot_cpu_has(feature) =3D=3D !(a->cpuid & ALTINSTR_FLAG_INV)) { if (a->padlen > 1) optimize_nops(a, instr); =20 continue; } =20 - DPRINTK("feat: %d*32+%d, old: (%pS (%px) len: %d), repl: (%px, len: %d),= pad: %d", - a->cpuid >> 5, - a->cpuid & 0x1f, + DPRINTK("feat: %s%d*32+%d, old: (%pS (%px) len: %d), repl: (%px, len: %d= ), pad: %d", + (a->cpuid & ALTINSTR_FLAG_INV) ? "!" : "", + feature >> 5, + feature & 0x1f, instr, instr, a->instrlen, replacement, a->replacementlen, a->padlen); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8930DC43334 for ; Sat, 23 Jul 2022 09:58:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237636AbiGWJ6V (ORCPT ); Sat, 23 Jul 2022 05:58:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237484AbiGWJ53 (ORCPT ); Sat, 23 Jul 2022 05:57:29 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A95AB474FE; Sat, 23 Jul 2022 02:57:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 45C2F611CD; Sat, 23 Jul 2022 09:57:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 54F13C341C0; Sat, 23 Jul 2022 09:57:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570229; bh=HAUe1xX18R+K6PokYOpqGZLlTgYNlzycxBTgy03jkzE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kDIjm3FZ/UJWR32918tVyC86Cpsmq4vEud0zga1FPc28SI+TboLqQnoflhFzLT+uP Z4EriJ06UoXkFWdbgHwdZi2NkqnST+cP7kKTnhTqeq99zsVAGyVuW3JtQgikrcJEpv zsIBgclv7HqOamJTZ3nABfpOcnrtlOSo9s9V7W4M= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Juergen Gross , Borislav Petkov , "Peter Zijlstra (Intel)" , Ben Hutchings Subject: [PATCH 5.10 013/148] x86/alternative: Support ALTERNATIVE_TERNARY Date: Sat, 23 Jul 2022 11:53:45 +0200 Message-Id: <20220723095228.112228206@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Juergen Gross commit e208b3c4a9748b2c17aa09ba663b5096ccf82dce upstream. Add ALTERNATIVE_TERNARY support for replacing an initial instruction with either of two instructions depending on a feature: ALTERNATIVE_TERNARY "default_instr", FEATURE_NR, "feature_on_instr", "feature_off_instr" which will start with "default_instr" and at patch time will, depending on FEATURE_NR being set or not, patch that with either "feature_on_instr" or "feature_off_instr". [ bp: Add comment ontop. ] Signed-off-by: Juergen Gross Signed-off-by: Borislav Petkov Acked-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20210311142319.4723-7-jgross@suse.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/alternative.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -179,6 +179,11 @@ static inline int alternatives_text_rese ALTINSTR_REPLACEMENT(newinstr2, feature2, 2) \ ".popsection\n" =20 +/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ +#define ALTERNATIVE_TERNARY(oldinstr, feature, newinstr_yes, newinstr_no) \ + ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ + newinstr_yes, feature) + #define ALTERNATIVE_3(oldinsn, newinsn1, feat1, newinsn2, feat2, newinsn3,= feat3) \ OLDINSTR_3(oldinsn, 1, 2, 3) \ ".pushsection .altinstructions,\"a\"\n" \ @@ -210,6 +215,9 @@ static inline int alternatives_text_rese #define alternative_2(oldinstr, newinstr1, feature1, newinstr2, feature2) \ asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, feature1, newinstr= 2, feature2) ::: "memory") =20 +#define alternative_ternary(oldinstr, feature, newinstr_yes, newinstr_no) \ + asm_inline volatile(ALTERNATIVE_TERNARY(oldinstr, feature, newinstr_yes, = newinstr_no) ::: "memory") + /* * Alternative inline assembly with input. * @@ -380,6 +388,11 @@ static inline int alternatives_text_rese .popsection .endm =20 +/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ +#define ALTERNATIVE_TERNARY(oldinstr, feature, newinstr_yes, newinstr_no) \ + ALTERNATIVE_2 oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ + newinstr_yes, feature + #endif /* __ASSEMBLY__ */ =20 #endif /* _ASM_X86_ALTERNATIVE_H */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20804CCA489 for ; Sat, 23 Jul 2022 09:58:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237650AbiGWJ62 (ORCPT ); Sat, 23 Jul 2022 05:58:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237492AbiGWJ5c (ORCPT ); Sat, 23 Jul 2022 05:57:32 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A0011CB18; Sat, 23 Jul 2022 02:57:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id CCFAAB82C1F; Sat, 23 Jul 2022 09:57:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2A363C341C0; Sat, 23 Jul 2022 09:57:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570232; bh=2RZAygbDaVQjxYKpwYLxrZgZ1eBMkhJLQtxtddsgiTM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=w2NQHdpiKs2fKOEg1Q54fGDO8p6KHCRVakkqpFuDD/OubBT8MQvqeXqNVUkmTO3do tkThRk4jB1BDbhU5NRwG/GE5FXAzJxkoAgyNmUptfsrl5Yc7Sak7JADDg7ufNx4nIR 2p7K/Q4M6u8ZSFyh3OGn6g8bU+UBQnVJv2keYfMY= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Juergen Gross , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 014/148] x86/alternative: Use ALTERNATIVE_TERNARY() in _static_cpu_has() Date: Sat, 23 Jul 2022 11:53:46 +0200 Message-Id: <20220723095228.363904048@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Juergen Gross commit 2fe2a2c7a97c9bc32acc79154b75e754280f7867 upstream. _static_cpu_has() contains a completely open coded version of ALTERNATIVE_TERNARY(). Replace that with the macro instead. Signed-off-by: Juergen Gross Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210311142319.4723-8-jgross@suse.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/cpufeature.h | 41 ++++++++-------------------------= ----- 1 file changed, 9 insertions(+), 32 deletions(-) --- a/arch/x86/include/asm/cpufeature.h +++ b/arch/x86/include/asm/cpufeature.h @@ -8,6 +8,7 @@ =20 #include #include +#include =20 enum cpuid_leafs { @@ -172,39 +173,15 @@ extern void clear_cpu_cap(struct cpuinfo */ static __always_inline bool _static_cpu_has(u16 bit) { - asm_volatile_goto("1: jmp 6f\n" - "2:\n" - ".skip -(((5f-4f) - (2b-1b)) > 0) * " - "((5f-4f) - (2b-1b)),0x90\n" - "3:\n" - ".section .altinstructions,\"a\"\n" - " .long 1b - .\n" /* src offset */ - " .long 4f - .\n" /* repl offset */ - " .word %P[always]\n" /* always replace */ - " .byte 3b - 1b\n" /* src len */ - " .byte 5f - 4f\n" /* repl len */ - " .byte 3b - 2b\n" /* pad len */ - ".previous\n" - ".section .altinstr_replacement,\"ax\"\n" - "4: jmp %l[t_no]\n" - "5:\n" - ".previous\n" - ".section .altinstructions,\"a\"\n" - " .long 1b - .\n" /* src offset */ - " .long 0\n" /* no replacement */ - " .word %P[feature]\n" /* feature bit */ - " .byte 3b - 1b\n" /* src len */ - " .byte 0\n" /* repl len */ - " .byte 0\n" /* pad len */ - ".previous\n" - ".section .altinstr_aux,\"ax\"\n" - "6:\n" - " testb %[bitnum],%[cap_byte]\n" - " jnz %l[t_yes]\n" - " jmp %l[t_no]\n" - ".previous\n" + asm_volatile_goto( + ALTERNATIVE_TERNARY("jmp 6f", %P[feature], "", "jmp %l[t_no]") + ".section .altinstr_aux,\"ax\"\n" + "6:\n" + " testb %[bitnum],%[cap_byte]\n" + " jnz %l[t_yes]\n" + " jmp %l[t_no]\n" + ".previous\n" : : [feature] "i" (bit), - [always] "i" (X86_FEATURE_ALWAYS), [bitnum] "i" (1 << (bit & 7)), [cap_byte] "m" (((const char *)boot_cpu_data.x86_capability)[bit >>= 3]) : : t_yes, t_no); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E322C433EF for ; Sat, 23 Jul 2022 09:58:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237657AbiGWJ6a (ORCPT ); Sat, 23 Jul 2022 05:58:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237426AbiGWJ5c (ORCPT ); Sat, 23 Jul 2022 05:57:32 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5AC023AE79; Sat, 23 Jul 2022 02:57:16 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EC690611BD; Sat, 23 Jul 2022 09:57:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06C18C341C0; Sat, 23 Jul 2022 09:57:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570235; bh=9b44PqloIFpmblyMUGGLSuLs+WWjcV6QDufCaUVInTE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wWtDoMNJznnV/PMO/6SeAuVUKqnMWENpkPLgShTV/6pYtQ0HcXVcnSh3nqOWK4kN0 xt98epCzfr7oE1+gosKqlcHFjSBEKBq933Fzq985b4xZHMrcywX5cV/XHKlKmrOXfy szTGqnl9VZNRuimpd982U8ns91/Wko4mVYKr4+yc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 015/148] x86/insn: Rename insn_decode() to insn_decode_from_regs() Date: Sat, 23 Jul 2022 11:53:47 +0200 Message-Id: <20220723095228.677502278@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit 9e761296c52dcdb1aaa151b65bd39accb05740d9 upstream. Rename insn_decode() to insn_decode_from_regs() to denote that it receives regs as param and uses registers from there during decoding. Free the former name for a more generic version of the function. No functional changes. Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210304174237.31945-2-bp@alien8.de Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/insn-eval.h | 4 ++-- arch/x86/kernel/sev-es.c | 2 +- arch/x86/kernel/umip.c | 2 +- arch/x86/lib/insn-eval.c | 6 +++--- 4 files changed, 7 insertions(+), 7 deletions(-) --- a/arch/x86/include/asm/insn-eval.h +++ b/arch/x86/include/asm/insn-eval.h @@ -26,7 +26,7 @@ int insn_fetch_from_user(struct pt_regs unsigned char buf[MAX_INSN_SIZE]); int insn_fetch_from_user_inatomic(struct pt_regs *regs, unsigned char buf[MAX_INSN_SIZE]); -bool insn_decode(struct insn *insn, struct pt_regs *regs, - unsigned char buf[MAX_INSN_SIZE], int buf_size); +bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs, + unsigned char buf[MAX_INSN_SIZE], int buf_size); =20 #endif /* _ASM_X86_INSN_EVAL_H */ --- a/arch/x86/kernel/sev-es.c +++ b/arch/x86/kernel/sev-es.c @@ -236,7 +236,7 @@ static enum es_result vc_decode_insn(str return ES_EXCEPTION; } =20 - if (!insn_decode(&ctxt->insn, ctxt->regs, buffer, res)) + if (!insn_decode_from_regs(&ctxt->insn, ctxt->regs, buffer, res)) return ES_DECODE_FAILED; } else { res =3D vc_fetch_insn_kernel(ctxt, buffer); --- a/arch/x86/kernel/umip.c +++ b/arch/x86/kernel/umip.c @@ -356,7 +356,7 @@ bool fixup_umip_exception(struct pt_regs if (!nr_copied) return false; =20 - if (!insn_decode(&insn, regs, buf, nr_copied)) + if (!insn_decode_from_regs(&insn, regs, buf, nr_copied)) return false; =20 umip_inst =3D identify_insn(&insn); --- a/arch/x86/lib/insn-eval.c +++ b/arch/x86/lib/insn-eval.c @@ -1492,7 +1492,7 @@ int insn_fetch_from_user_inatomic(struct } =20 /** - * insn_decode() - Decode an instruction + * insn_decode_from_regs() - Decode an instruction * @insn: Structure to store decoded instruction * @regs: Structure with register values as seen when entering kernel mode * @buf: Buffer containing the instruction bytes @@ -1505,8 +1505,8 @@ int insn_fetch_from_user_inatomic(struct * * True if instruction was decoded, False otherwise. */ -bool insn_decode(struct insn *insn, struct pt_regs *regs, - unsigned char buf[MAX_INSN_SIZE], int buf_size) +bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs, + unsigned char buf[MAX_INSN_SIZE], int buf_size) { int seg_defs; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 714C9C43334 for ; Sat, 23 Jul 2022 09:58:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237437AbiGWJ6h (ORCPT ); Sat, 23 Jul 2022 05:58:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237514AbiGWJ5e (ORCPT ); Sat, 23 Jul 2022 05:57:34 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 342123F336; Sat, 23 Jul 2022 02:57:19 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C5E24611D4; Sat, 23 Jul 2022 09:57:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D0570C341C7; Sat, 23 Jul 2022 09:57:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570238; bh=bMri/2ybeRizR5q4f7KPtsMM4WqlBC0dFA8XjkQWNXU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iW8fpyIafSNs4i8Iy+eSovwavWNLBFCtOnimOpgEwJE+tby3X63duZIULx3sD+J1+ eV9pX0ExUNE2YDHexX9IoQb456jVBAYDStEZoV/dcBJvrO7nwXW2ZFUrW9YmaY97Tr ukpzdAPEjtbLtFhoJFCCIrPglk8VxfUk6Eu5qJOA= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Masami Hiramatsu , Ben Hutchings Subject: [PATCH 5.10 016/148] x86/insn: Add a __ignore_sync_check__ marker Date: Sat, 23 Jul 2022 11:53:48 +0200 Message-Id: <20220723095228.958106688@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit d30c7b820be5c4777fe6c3b0c21f9d0064251e51 upstream. Add an explicit __ignore_sync_check__ marker which will be used to mark lines which are supposed to be ignored by file synchronization check scripts, its advantage being that it explicitly denotes such lines in the code. Signed-off-by: Borislav Petkov Reviewed-by: Masami Hiramatsu Link: https://lkml.kernel.org/r/20210304174237.31945-4-bp@alien8.de Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/inat.h | 2 +- arch/x86/include/asm/insn.h | 2 +- arch/x86/lib/inat.c | 2 +- arch/x86/lib/insn.c | 6 +++--- tools/arch/x86/include/asm/inat.h | 2 +- tools/arch/x86/include/asm/insn.h | 2 +- tools/arch/x86/lib/inat.c | 2 +- tools/arch/x86/lib/insn.c | 6 +++--- tools/objtool/sync-check.sh | 17 +++++++++++++---- tools/perf/check-headers.sh | 15 +++++++++++---- 10 files changed, 36 insertions(+), 20 deletions(-) --- a/arch/x86/include/asm/inat.h +++ b/arch/x86/include/asm/inat.h @@ -6,7 +6,7 @@ * * Written by Masami Hiramatsu */ -#include +#include /* __ignore_sync_check__ */ =20 /* * Internal bits. Don't use bitmasks directly, because these bits are --- a/arch/x86/include/asm/insn.h +++ b/arch/x86/include/asm/insn.h @@ -8,7 +8,7 @@ */ =20 /* insn_attr_t is defined in inat.h */ -#include +#include /* __ignore_sync_check__ */ =20 struct insn_field { union { --- a/arch/x86/lib/inat.c +++ b/arch/x86/lib/inat.c @@ -4,7 +4,7 @@ * * Written by Masami Hiramatsu */ -#include +#include /* __ignore_sync_check__ */ =20 /* Attribute tables are generated from opcode map */ #include "inat-tables.c" --- a/arch/x86/lib/insn.c +++ b/arch/x86/lib/insn.c @@ -10,10 +10,10 @@ #else #include #endif -#include -#include +#include /*__ignore_sync_check__ */ +#include /* __ignore_sync_check__ */ =20 -#include +#include /* __ignore_sync_check__ */ =20 /* Verify next sizeof(t) bytes can be on the same instruction */ #define validate_next(t, insn, n) \ --- a/tools/arch/x86/include/asm/inat.h +++ b/tools/arch/x86/include/asm/inat.h @@ -6,7 +6,7 @@ * * Written by Masami Hiramatsu */ -#include "inat_types.h" +#include "inat_types.h" /* __ignore_sync_check__ */ =20 /* * Internal bits. Don't use bitmasks directly, because these bits are --- a/tools/arch/x86/include/asm/insn.h +++ b/tools/arch/x86/include/asm/insn.h @@ -8,7 +8,7 @@ */ =20 /* insn_attr_t is defined in inat.h */ -#include "inat.h" +#include "inat.h" /* __ignore_sync_check__ */ =20 struct insn_field { union { --- a/tools/arch/x86/lib/inat.c +++ b/tools/arch/x86/lib/inat.c @@ -4,7 +4,7 @@ * * Written by Masami Hiramatsu */ -#include "../include/asm/insn.h" +#include "../include/asm/insn.h" /* __ignore_sync_check__ */ =20 /* Attribute tables are generated from opcode map */ #include "inat-tables.c" --- a/tools/arch/x86/lib/insn.c +++ b/tools/arch/x86/lib/insn.c @@ -10,10 +10,10 @@ #else #include #endif -#include "../include/asm/inat.h" -#include "../include/asm/insn.h" +#include "../include/asm/inat.h" /* __ignore_sync_check__ */ +#include "../include/asm/insn.h" /* __ignore_sync_check__ */ =20 -#include "../include/asm/emulate_prefix.h" +#include "../include/asm/emulate_prefix.h" /* __ignore_sync_check__ */ =20 /* Verify next sizeof(t) bytes can be on the same instruction */ #define validate_next(t, insn, n) \ --- a/tools/objtool/sync-check.sh +++ b/tools/objtool/sync-check.sh @@ -16,11 +16,14 @@ arch/x86/include/asm/emulate_prefix.h arch/x86/lib/x86-opcode-map.txt arch/x86/tools/gen-insn-attr-x86.awk include/linux/static_call_types.h -arch/x86/include/asm/inat.h -I '^#include [\"<]\(asm/\)*inat_types.h[\= ">]' -arch/x86/include/asm/insn.h -I '^#include [\"<]\(asm/\)*inat.h[\">]' -arch/x86/lib/inat.c -I '^#include [\"<]\(../include/\)*asm/ins= n.h[\">]' -arch/x86/lib/insn.c -I '^#include [\"<]\(../include/\)*asm/in\= (at\|sn\).h[\">]' -I '^#include [\"<]\(../include/\)*asm/emulate_prefix.h[\= ">]' " + +SYNC_CHECK_FILES=3D' +arch/x86/include/asm/inat.h +arch/x86/include/asm/insn.h +arch/x86/lib/inat.c +arch/x86/lib/insn.c +' fi =20 check_2 () { @@ -63,3 +66,9 @@ while read -r file_entry; do done <string sy= scall @@ -129,6 +136,10 @@ for i in $FILES; do check $i -B done =20 +for i in $SYNC_CHECK_FILES; do + check $i '-I "^.*\/\*.*__ignore_sync_check__.*\*\/.*$"' +done + # diff with extra ignore lines check arch/x86/lib/memcpy_64.S '-I "^EXPORT_SYMBOL" -I "^#include <= asm/export.h>" -I"^SYM_FUNC_START\(_LOCAL\)*(memcpy_\(erms\|orig\))"' check arch/x86/lib/memset_64.S '-I "^EXPORT_SYMBOL" -I "^#include <= asm/export.h>" -I"^SYM_FUNC_START\(_LOCAL\)*(memset_\(erms\|orig\))"' @@ -137,10 +148,6 @@ check include/uapi/linux/mman.h '- check include/linux/build_bug.h '-I "^#\(ifndef\|endif\)\( \/\/\)* s= tatic_assert$"' check include/linux/ctype.h '-I "isdigit("' check lib/ctype.c '-I "^EXPORT_SYMBOL" -I "^#include " -B' -check arch/x86/include/asm/inat.h '-I "^#include [\"<]\(asm/\)*inat_ty= pes.h[\">]"' -check arch/x86/include/asm/insn.h '-I "^#include [\"<]\(asm/\)*inat.h[= \">]"' -check arch/x86/lib/inat.c '-I "^#include [\"<]\(../include/\)*asm/in= sn.h[\">]"' -check arch/x86/lib/insn.c '-I "^#include [\"<]\(../include/\)*= asm/in\(at\|sn\).h[\">]" -I "^#include [\"<]\(../include/\)*asm/emulate_pre= fix.h[\">]"' =20 # diff non-symmetric files check_2 tools/perf/arch/x86/entry/syscalls/syscall_64.tbl arch/x86/entry/s= yscalls/syscall_64.tbl From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 750A1C43334 for ; Sat, 23 Jul 2022 09:58:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237680AbiGWJ6z (ORCPT ); Sat, 23 Jul 2022 05:58:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237559AbiGWJ6P (ORCPT ); Sat, 23 Jul 2022 05:58:15 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1B586555B; Sat, 23 Jul 2022 02:57:24 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7E8B9B827CC; Sat, 23 Jul 2022 09:57:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A4C8EC341CA; Sat, 23 Jul 2022 09:57:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570241; bh=d/Fe/oadpda4DuQ7wRcCWPnk9oSwr0PlrqLD2dd/tC0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PrbOIMlTj7a2aJugJFPkQMYEFzvT3pmO3AgRf5IhvRyp6tqKfMNxjjKZTu703eVWV wuYFcq1UjagOPHJRO97V9hwt8GPe15+x4HG/pWjYMMpZ9dZjiKsYQkrA+MK+URgk1d jAUu0mn+wBHefN/J7EBjpkxRwni+a1LkPdZ2ZQZs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Masami Hiramatsu , Ben Hutchings Subject: [PATCH 5.10 017/148] x86/insn: Add an insn_decode() API Date: Sat, 23 Jul 2022 11:53:49 +0200 Message-Id: <20220723095229.257414629@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit 93281c4a96572a34504244969b938e035204778d upstream. Users of the instruction decoder should use this to decode instruction bytes. For that, have insn*() helpers return an int value to denote success/failure. When there's an error fetching the next insn byte and the insn falls short, return -ENODATA to denote that. While at it, make insn_get_opcode() more stricter as to whether what has seen so far is a valid insn and if not. Copy linux/kconfig.h for the tools-version of the decoder so that it can use IS_ENABLED(). Also, cast the INSN_MODE_KERN dummy define value to (enum insn_mode) for tools use of the decoder because perf tool builds with -Werror and errors out with -Werror=3Dsign-compare otherwise. Signed-off-by: Borislav Petkov Acked-by: Masami Hiramatsu Link: https://lkml.kernel.org/r/20210304174237.31945-5-bp@alien8.de Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/insn.h | 24 ++-- arch/x86/lib/insn.c | 216 +++++++++++++++++++++++++++++----= --- tools/arch/x86/include/asm/insn.h | 24 ++-- tools/arch/x86/lib/insn.c | 222 +++++++++++++++++++++++++++++----= ----- tools/include/linux/kconfig.h | 73 ++++++++++++ 5 files changed, 452 insertions(+), 107 deletions(-) create mode 100644 tools/include/linux/kconfig.h --- a/arch/x86/include/asm/insn.h +++ b/arch/x86/include/asm/insn.h @@ -87,13 +87,23 @@ struct insn { #define X86_VEX_M_MAX 0x1f /* VEX3.M Maximum value */ =20 extern void insn_init(struct insn *insn, const void *kaddr, int buf_len, i= nt x86_64); -extern void insn_get_prefixes(struct insn *insn); -extern void insn_get_opcode(struct insn *insn); -extern void insn_get_modrm(struct insn *insn); -extern void insn_get_sib(struct insn *insn); -extern void insn_get_displacement(struct insn *insn); -extern void insn_get_immediate(struct insn *insn); -extern void insn_get_length(struct insn *insn); +extern int insn_get_prefixes(struct insn *insn); +extern int insn_get_opcode(struct insn *insn); +extern int insn_get_modrm(struct insn *insn); +extern int insn_get_sib(struct insn *insn); +extern int insn_get_displacement(struct insn *insn); +extern int insn_get_immediate(struct insn *insn); +extern int insn_get_length(struct insn *insn); + +enum insn_mode { + INSN_MODE_32, + INSN_MODE_64, + /* Mode is determined by the current kernel build. */ + INSN_MODE_KERN, + INSN_NUM_MODES, +}; + +extern int insn_decode(struct insn *insn, const void *kaddr, int buf_len, = enum insn_mode m); =20 /* Attribute will be determined after getting ModRM (for opcode groups) */ static inline void insn_get_attribute(struct insn *insn) --- a/arch/x86/lib/insn.c +++ b/arch/x86/lib/insn.c @@ -13,6 +13,9 @@ #include /*__ignore_sync_check__ */ #include /* __ignore_sync_check__ */ =20 +#include +#include + #include /* __ignore_sync_check__ */ =20 /* Verify next sizeof(t) bytes can be on the same instruction */ @@ -97,8 +100,12 @@ static void insn_get_emulate_prefix(stru * Populates the @insn->prefixes bitmap, and updates @insn->next_byte * to point to the (first) opcode. No effect if @insn->prefixes.got * is already set. + * + * * Returns: + * 0: on success + * < 0: on error */ -void insn_get_prefixes(struct insn *insn) +int insn_get_prefixes(struct insn *insn) { struct insn_field *prefixes =3D &insn->prefixes; insn_attr_t attr; @@ -106,7 +113,7 @@ void insn_get_prefixes(struct insn *insn int i, nb; =20 if (prefixes->got) - return; + return 0; =20 insn_get_emulate_prefix(insn); =20 @@ -217,8 +224,10 @@ vex_end: =20 prefixes->got =3D 1; =20 + return 0; + err_out: - return; + return -ENODATA; } =20 /** @@ -230,16 +239,25 @@ err_out: * If necessary, first collects any preceding (prefix) bytes. * Sets @insn->opcode.value =3D opcode1. No effect if @insn->opcode.got * is already 1. + * + * Returns: + * 0: on success + * < 0: on error */ -void insn_get_opcode(struct insn *insn) +int insn_get_opcode(struct insn *insn) { struct insn_field *opcode =3D &insn->opcode; + int pfx_id, ret; insn_byte_t op; - int pfx_id; + if (opcode->got) - return; - if (!insn->prefixes.got) - insn_get_prefixes(insn); + return 0; + + if (!insn->prefixes.got) { + ret =3D insn_get_prefixes(insn); + if (ret) + return ret; + } =20 /* Get first opcode */ op =3D get_next(insn_byte_t, insn); @@ -254,9 +272,13 @@ void insn_get_opcode(struct insn *insn) insn->attr =3D inat_get_avx_attribute(op, m, p); if ((inat_must_evex(insn->attr) && !insn_is_evex(insn)) || (!inat_accept_vex(insn->attr) && - !inat_is_group(insn->attr))) - insn->attr =3D 0; /* This instruction is bad */ - goto end; /* VEX has only 1 byte for opcode */ + !inat_is_group(insn->attr))) { + /* This instruction is bad */ + insn->attr =3D 0; + return -EINVAL; + } + /* VEX has only 1 byte for opcode */ + goto end; } =20 insn->attr =3D inat_get_opcode_attribute(op); @@ -267,13 +289,18 @@ void insn_get_opcode(struct insn *insn) pfx_id =3D insn_last_prefix_id(insn); insn->attr =3D inat_get_escape_attribute(op, pfx_id, insn->attr); } - if (inat_must_vex(insn->attr)) - insn->attr =3D 0; /* This instruction is bad */ + + if (inat_must_vex(insn->attr)) { + /* This instruction is bad */ + insn->attr =3D 0; + return -EINVAL; + } end: opcode->got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 /** @@ -283,15 +310,25 @@ err_out: * Populates @insn->modrm and updates @insn->next_byte to point past the * ModRM byte, if any. If necessary, first collects the preceding bytes * (prefixes and opcode(s)). No effect if @insn->modrm.got is already 1. + * + * Returns: + * 0: on success + * < 0: on error */ -void insn_get_modrm(struct insn *insn) +int insn_get_modrm(struct insn *insn) { struct insn_field *modrm =3D &insn->modrm; insn_byte_t pfx_id, mod; + int ret; + if (modrm->got) - return; - if (!insn->opcode.got) - insn_get_opcode(insn); + return 0; + + if (!insn->opcode.got) { + ret =3D insn_get_opcode(insn); + if (ret) + return ret; + } =20 if (inat_has_modrm(insn->attr)) { mod =3D get_next(insn_byte_t, insn); @@ -301,17 +338,22 @@ void insn_get_modrm(struct insn *insn) pfx_id =3D insn_last_prefix_id(insn); insn->attr =3D inat_get_group_attribute(mod, pfx_id, insn->attr); - if (insn_is_avx(insn) && !inat_accept_vex(insn->attr)) - insn->attr =3D 0; /* This is bad */ + if (insn_is_avx(insn) && !inat_accept_vex(insn->attr)) { + /* Bad insn */ + insn->attr =3D 0; + return -EINVAL; + } } } =20 if (insn->x86_64 && inat_is_force64(insn->attr)) insn->opnd_bytes =3D 8; + modrm->got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 =20 @@ -325,11 +367,16 @@ err_out: int insn_rip_relative(struct insn *insn) { struct insn_field *modrm =3D &insn->modrm; + int ret; =20 if (!insn->x86_64) return 0; - if (!modrm->got) - insn_get_modrm(insn); + + if (!modrm->got) { + ret =3D insn_get_modrm(insn); + if (ret) + return 0; + } /* * For rip-relative instructions, the mod field (top 2 bits) * is zero and the r/m field (bottom 3 bits) is 0x5. @@ -343,15 +390,25 @@ int insn_rip_relative(struct insn *insn) * * If necessary, first collects the instruction up to and including the * ModRM byte. + * + * Returns: + * 0: if decoding succeeded + * < 0: otherwise. */ -void insn_get_sib(struct insn *insn) +int insn_get_sib(struct insn *insn) { insn_byte_t modrm; + int ret; =20 if (insn->sib.got) - return; - if (!insn->modrm.got) - insn_get_modrm(insn); + return 0; + + if (!insn->modrm.got) { + ret =3D insn_get_modrm(insn); + if (ret) + return ret; + } + if (insn->modrm.nbytes) { modrm =3D (insn_byte_t)insn->modrm.value; if (insn->addr_bytes !=3D 2 && @@ -362,8 +419,10 @@ void insn_get_sib(struct insn *insn) } insn->sib.got =3D 1; =20 + return 0; + err_out: - return; + return -ENODATA; } =20 =20 @@ -374,15 +433,25 @@ err_out: * If necessary, first collects the instruction up to and including the * SIB byte. * Displacement value is sign-expanded. + * + * * Returns: + * 0: if decoding succeeded + * < 0: otherwise. */ -void insn_get_displacement(struct insn *insn) +int insn_get_displacement(struct insn *insn) { insn_byte_t mod, rm, base; + int ret; =20 if (insn->displacement.got) - return; - if (!insn->sib.got) - insn_get_sib(insn); + return 0; + + if (!insn->sib.got) { + ret =3D insn_get_sib(insn); + if (ret) + return ret; + } + if (insn->modrm.nbytes) { /* * Interpreting the modrm byte: @@ -425,9 +494,10 @@ void insn_get_displacement(struct insn * } out: insn->displacement.got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 /* Decode moffset16/32/64. Return 0 if failed */ @@ -538,20 +608,30 @@ err_out: } =20 /** - * insn_get_immediate() - Get the immediates of instruction + * insn_get_immediate() - Get the immediate in an instruction * @insn: &struct insn containing instruction * * If necessary, first collects the instruction up to and including the * displacement bytes. * Basically, most of immediates are sign-expanded. Unsigned-value can be - * get by bit masking with ((1 << (nbytes * 8)) - 1) + * computed by bit masking with ((1 << (nbytes * 8)) - 1) + * + * Returns: + * 0: on success + * < 0: on error */ -void insn_get_immediate(struct insn *insn) +int insn_get_immediate(struct insn *insn) { + int ret; + if (insn->immediate.got) - return; - if (!insn->displacement.got) - insn_get_displacement(insn); + return 0; + + if (!insn->displacement.got) { + ret =3D insn_get_displacement(insn); + if (ret) + return ret; + } =20 if (inat_has_moffset(insn->attr)) { if (!__get_moffset(insn)) @@ -604,9 +684,10 @@ void insn_get_immediate(struct insn *ins } done: insn->immediate.got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 /** @@ -615,13 +696,58 @@ err_out: * * If necessary, first collects the instruction up to and including the * immediates bytes. - */ -void insn_get_length(struct insn *insn) + * + * Returns: + * - 0 on success + * - < 0 on error +*/ +int insn_get_length(struct insn *insn) { + int ret; + if (insn->length) - return; - if (!insn->immediate.got) - insn_get_immediate(insn); + return 0; + + if (!insn->immediate.got) { + ret =3D insn_get_immediate(insn); + if (ret) + return ret; + } + insn->length =3D (unsigned char)((unsigned long)insn->next_byte - (unsigned long)insn->kaddr); + + return 0; +} + +/** + * insn_decode() - Decode an x86 instruction + * @insn: &struct insn to be initialized + * @kaddr: address (in kernel memory) of instruction (or copy thereof) + * @buf_len: length of the insn buffer at @kaddr + * @m: insn mode, see enum insn_mode + * + * Returns: + * 0: if decoding succeeded + * < 0: otherwise. + */ +int insn_decode(struct insn *insn, const void *kaddr, int buf_len, enum in= sn_mode m) +{ + int ret; + +/* #define INSN_MODE_KERN -1 __ignore_sync_check__ mode is only valid in t= he kernel */ + + if (m =3D=3D INSN_MODE_KERN) + insn_init(insn, kaddr, buf_len, IS_ENABLED(CONFIG_X86_64)); + else + insn_init(insn, kaddr, buf_len, m =3D=3D INSN_MODE_64); + + ret =3D insn_get_length(insn); + if (ret) + return ret; + + if (insn_complete(insn)) + return 0; + + return -EINVAL; } --- a/tools/arch/x86/include/asm/insn.h +++ b/tools/arch/x86/include/asm/insn.h @@ -87,13 +87,23 @@ struct insn { #define X86_VEX_M_MAX 0x1f /* VEX3.M Maximum value */ =20 extern void insn_init(struct insn *insn, const void *kaddr, int buf_len, i= nt x86_64); -extern void insn_get_prefixes(struct insn *insn); -extern void insn_get_opcode(struct insn *insn); -extern void insn_get_modrm(struct insn *insn); -extern void insn_get_sib(struct insn *insn); -extern void insn_get_displacement(struct insn *insn); -extern void insn_get_immediate(struct insn *insn); -extern void insn_get_length(struct insn *insn); +extern int insn_get_prefixes(struct insn *insn); +extern int insn_get_opcode(struct insn *insn); +extern int insn_get_modrm(struct insn *insn); +extern int insn_get_sib(struct insn *insn); +extern int insn_get_displacement(struct insn *insn); +extern int insn_get_immediate(struct insn *insn); +extern int insn_get_length(struct insn *insn); + +enum insn_mode { + INSN_MODE_32, + INSN_MODE_64, + /* Mode is determined by the current kernel build. */ + INSN_MODE_KERN, + INSN_NUM_MODES, +}; + +extern int insn_decode(struct insn *insn, const void *kaddr, int buf_len, = enum insn_mode m); =20 /* Attribute will be determined after getting ModRM (for opcode groups) */ static inline void insn_get_attribute(struct insn *insn) --- a/tools/arch/x86/lib/insn.c +++ b/tools/arch/x86/lib/insn.c @@ -10,10 +10,13 @@ #else #include #endif -#include "../include/asm/inat.h" /* __ignore_sync_check__ */ -#include "../include/asm/insn.h" /* __ignore_sync_check__ */ +#include /* __ignore_sync_check__ */ +#include /* __ignore_sync_check__ */ =20 -#include "../include/asm/emulate_prefix.h" /* __ignore_sync_check__ */ +#include +#include + +#include /* __ignore_sync_check__ */ =20 /* Verify next sizeof(t) bytes can be on the same instruction */ #define validate_next(t, insn, n) \ @@ -97,8 +100,12 @@ static void insn_get_emulate_prefix(stru * Populates the @insn->prefixes bitmap, and updates @insn->next_byte * to point to the (first) opcode. No effect if @insn->prefixes.got * is already set. + * + * * Returns: + * 0: on success + * < 0: on error */ -void insn_get_prefixes(struct insn *insn) +int insn_get_prefixes(struct insn *insn) { struct insn_field *prefixes =3D &insn->prefixes; insn_attr_t attr; @@ -106,7 +113,7 @@ void insn_get_prefixes(struct insn *insn int i, nb; =20 if (prefixes->got) - return; + return 0; =20 insn_get_emulate_prefix(insn); =20 @@ -217,8 +224,10 @@ vex_end: =20 prefixes->got =3D 1; =20 + return 0; + err_out: - return; + return -ENODATA; } =20 /** @@ -230,16 +239,25 @@ err_out: * If necessary, first collects any preceding (prefix) bytes. * Sets @insn->opcode.value =3D opcode1. No effect if @insn->opcode.got * is already 1. + * + * Returns: + * 0: on success + * < 0: on error */ -void insn_get_opcode(struct insn *insn) +int insn_get_opcode(struct insn *insn) { struct insn_field *opcode =3D &insn->opcode; + int pfx_id, ret; insn_byte_t op; - int pfx_id; + if (opcode->got) - return; - if (!insn->prefixes.got) - insn_get_prefixes(insn); + return 0; + + if (!insn->prefixes.got) { + ret =3D insn_get_prefixes(insn); + if (ret) + return ret; + } =20 /* Get first opcode */ op =3D get_next(insn_byte_t, insn); @@ -254,9 +272,13 @@ void insn_get_opcode(struct insn *insn) insn->attr =3D inat_get_avx_attribute(op, m, p); if ((inat_must_evex(insn->attr) && !insn_is_evex(insn)) || (!inat_accept_vex(insn->attr) && - !inat_is_group(insn->attr))) - insn->attr =3D 0; /* This instruction is bad */ - goto end; /* VEX has only 1 byte for opcode */ + !inat_is_group(insn->attr))) { + /* This instruction is bad */ + insn->attr =3D 0; + return -EINVAL; + } + /* VEX has only 1 byte for opcode */ + goto end; } =20 insn->attr =3D inat_get_opcode_attribute(op); @@ -267,13 +289,18 @@ void insn_get_opcode(struct insn *insn) pfx_id =3D insn_last_prefix_id(insn); insn->attr =3D inat_get_escape_attribute(op, pfx_id, insn->attr); } - if (inat_must_vex(insn->attr)) - insn->attr =3D 0; /* This instruction is bad */ + + if (inat_must_vex(insn->attr)) { + /* This instruction is bad */ + insn->attr =3D 0; + return -EINVAL; + } end: opcode->got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 /** @@ -283,15 +310,25 @@ err_out: * Populates @insn->modrm and updates @insn->next_byte to point past the * ModRM byte, if any. If necessary, first collects the preceding bytes * (prefixes and opcode(s)). No effect if @insn->modrm.got is already 1. + * + * Returns: + * 0: on success + * < 0: on error */ -void insn_get_modrm(struct insn *insn) +int insn_get_modrm(struct insn *insn) { struct insn_field *modrm =3D &insn->modrm; insn_byte_t pfx_id, mod; + int ret; + if (modrm->got) - return; - if (!insn->opcode.got) - insn_get_opcode(insn); + return 0; + + if (!insn->opcode.got) { + ret =3D insn_get_opcode(insn); + if (ret) + return ret; + } =20 if (inat_has_modrm(insn->attr)) { mod =3D get_next(insn_byte_t, insn); @@ -301,17 +338,22 @@ void insn_get_modrm(struct insn *insn) pfx_id =3D insn_last_prefix_id(insn); insn->attr =3D inat_get_group_attribute(mod, pfx_id, insn->attr); - if (insn_is_avx(insn) && !inat_accept_vex(insn->attr)) - insn->attr =3D 0; /* This is bad */ + if (insn_is_avx(insn) && !inat_accept_vex(insn->attr)) { + /* Bad insn */ + insn->attr =3D 0; + return -EINVAL; + } } } =20 if (insn->x86_64 && inat_is_force64(insn->attr)) insn->opnd_bytes =3D 8; + modrm->got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 =20 @@ -325,11 +367,16 @@ err_out: int insn_rip_relative(struct insn *insn) { struct insn_field *modrm =3D &insn->modrm; + int ret; =20 if (!insn->x86_64) return 0; - if (!modrm->got) - insn_get_modrm(insn); + + if (!modrm->got) { + ret =3D insn_get_modrm(insn); + if (ret) + return 0; + } /* * For rip-relative instructions, the mod field (top 2 bits) * is zero and the r/m field (bottom 3 bits) is 0x5. @@ -343,15 +390,25 @@ int insn_rip_relative(struct insn *insn) * * If necessary, first collects the instruction up to and including the * ModRM byte. + * + * Returns: + * 0: if decoding succeeded + * < 0: otherwise. */ -void insn_get_sib(struct insn *insn) +int insn_get_sib(struct insn *insn) { insn_byte_t modrm; + int ret; =20 if (insn->sib.got) - return; - if (!insn->modrm.got) - insn_get_modrm(insn); + return 0; + + if (!insn->modrm.got) { + ret =3D insn_get_modrm(insn); + if (ret) + return ret; + } + if (insn->modrm.nbytes) { modrm =3D (insn_byte_t)insn->modrm.value; if (insn->addr_bytes !=3D 2 && @@ -362,8 +419,10 @@ void insn_get_sib(struct insn *insn) } insn->sib.got =3D 1; =20 + return 0; + err_out: - return; + return -ENODATA; } =20 =20 @@ -374,15 +433,25 @@ err_out: * If necessary, first collects the instruction up to and including the * SIB byte. * Displacement value is sign-expanded. + * + * * Returns: + * 0: if decoding succeeded + * < 0: otherwise. */ -void insn_get_displacement(struct insn *insn) +int insn_get_displacement(struct insn *insn) { insn_byte_t mod, rm, base; + int ret; =20 if (insn->displacement.got) - return; - if (!insn->sib.got) - insn_get_sib(insn); + return 0; + + if (!insn->sib.got) { + ret =3D insn_get_sib(insn); + if (ret) + return ret; + } + if (insn->modrm.nbytes) { /* * Interpreting the modrm byte: @@ -425,9 +494,10 @@ void insn_get_displacement(struct insn * } out: insn->displacement.got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 /* Decode moffset16/32/64. Return 0 if failed */ @@ -538,20 +608,30 @@ err_out: } =20 /** - * insn_get_immediate() - Get the immediates of instruction + * insn_get_immediate() - Get the immediate in an instruction * @insn: &struct insn containing instruction * * If necessary, first collects the instruction up to and including the * displacement bytes. * Basically, most of immediates are sign-expanded. Unsigned-value can be - * get by bit masking with ((1 << (nbytes * 8)) - 1) + * computed by bit masking with ((1 << (nbytes * 8)) - 1) + * + * Returns: + * 0: on success + * < 0: on error */ -void insn_get_immediate(struct insn *insn) +int insn_get_immediate(struct insn *insn) { + int ret; + if (insn->immediate.got) - return; - if (!insn->displacement.got) - insn_get_displacement(insn); + return 0; + + if (!insn->displacement.got) { + ret =3D insn_get_displacement(insn); + if (ret) + return ret; + } =20 if (inat_has_moffset(insn->attr)) { if (!__get_moffset(insn)) @@ -604,9 +684,10 @@ void insn_get_immediate(struct insn *ins } done: insn->immediate.got =3D 1; + return 0; =20 err_out: - return; + return -ENODATA; } =20 /** @@ -615,13 +696,58 @@ err_out: * * If necessary, first collects the instruction up to and including the * immediates bytes. - */ -void insn_get_length(struct insn *insn) + * + * Returns: + * - 0 on success + * - < 0 on error +*/ +int insn_get_length(struct insn *insn) { + int ret; + if (insn->length) - return; - if (!insn->immediate.got) - insn_get_immediate(insn); + return 0; + + if (!insn->immediate.got) { + ret =3D insn_get_immediate(insn); + if (ret) + return ret; + } + insn->length =3D (unsigned char)((unsigned long)insn->next_byte - (unsigned long)insn->kaddr); + + return 0; +} + +/** + * insn_decode() - Decode an x86 instruction + * @insn: &struct insn to be initialized + * @kaddr: address (in kernel memory) of instruction (or copy thereof) + * @buf_len: length of the insn buffer at @kaddr + * @m: insn mode, see enum insn_mode + * + * Returns: + * 0: if decoding succeeded + * < 0: otherwise. + */ +int insn_decode(struct insn *insn, const void *kaddr, int buf_len, enum in= sn_mode m) +{ + int ret; + +#define INSN_MODE_KERN (enum insn_mode)-1 /* __ignore_sync_check__ mode is= only valid in the kernel */ + + if (m =3D=3D INSN_MODE_KERN) + insn_init(insn, kaddr, buf_len, IS_ENABLED(CONFIG_X86_64)); + else + insn_init(insn, kaddr, buf_len, m =3D=3D INSN_MODE_64); + + ret =3D insn_get_length(insn); + if (ret) + return ret; + + if (insn_complete(insn)) + return 0; + + return -EINVAL; } --- /dev/null +++ b/tools/include/linux/kconfig.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _TOOLS_LINUX_KCONFIG_H +#define _TOOLS_LINUX_KCONFIG_H + +/* CONFIG_CC_VERSION_TEXT (Do not delete this comment. See help in Kconfig= ) */ + +#ifdef CONFIG_CPU_BIG_ENDIAN +#define __BIG_ENDIAN 4321 +#else +#define __LITTLE_ENDIAN 1234 +#endif + +#define __ARG_PLACEHOLDER_1 0, +#define __take_second_arg(__ignored, val, ...) val + +/* + * The use of "&&" / "||" is limited in certain expressions. + * The following enable to calculate "and" / "or" with macro expansion onl= y. + */ +#define __and(x, y) ___and(x, y) +#define ___and(x, y) ____and(__ARG_PLACEHOLDER_##x, y) +#define ____and(arg1_or_junk, y) __take_second_arg(arg1_or_junk y, 0) + +#define __or(x, y) ___or(x, y) +#define ___or(x, y) ____or(__ARG_PLACEHOLDER_##x, y) +#define ____or(arg1_or_junk, y) __take_second_arg(arg1_or_junk 1, y) + +/* + * Helper macros to use CONFIG_ options in C/CPP expressions. Note that + * these only work with boolean and tristate options. + */ + +/* + * Getting something that works in C and CPP for an arg that may or may + * not be defined is tricky. Here, if we have "#define CONFIG_BOOGER 1" + * we match on the placeholder define, insert the "0," for arg1 and genera= te + * the triplet (0, 1, 0). Then the last step cherry picks the 2nd arg (a = one). + * When CONFIG_BOOGER is not defined, we generate a (... 1, 0) pair, and w= hen + * the last step cherry picks the 2nd arg, we get a zero. + */ +#define __is_defined(x) ___is_defined(x) +#define ___is_defined(val) ____is_defined(__ARG_PLACEHOLDER_##val) +#define ____is_defined(arg1_or_junk) __take_second_arg(arg1_or_junk 1, 0) + +/* + * IS_BUILTIN(CONFIG_FOO) evaluates to 1 if CONFIG_FOO is set to 'y', 0 + * otherwise. For boolean options, this is equivalent to + * IS_ENABLED(CONFIG_FOO). + */ +#define IS_BUILTIN(option) __is_defined(option) + +/* + * IS_MODULE(CONFIG_FOO) evaluates to 1 if CONFIG_FOO is set to 'm', 0 + * otherwise. + */ +#define IS_MODULE(option) __is_defined(option##_MODULE) + +/* + * IS_REACHABLE(CONFIG_FOO) evaluates to 1 if the currently compiled + * code can call a function defined in code compiled based on CONFIG_FOO. + * This is similar to IS_ENABLED(), but returns false when invoked from + * built-in code when CONFIG_FOO is set to 'm'. + */ +#define IS_REACHABLE(option) __or(IS_BUILTIN(option), \ + __and(IS_MODULE(option), __is_defined(MODULE))) + +/* + * IS_ENABLED(CONFIG_FOO) evaluates to 1 if CONFIG_FOO is set to 'y' or 'm= ', + * 0 otherwise. + */ +#define IS_ENABLED(option) __or(IS_BUILTIN(option), IS_MODULE(option)) + +#endif /* _TOOLS_LINUX_KCONFIG_H */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FA4EC43334 for ; Sat, 23 Jul 2022 09:58:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237691AbiGWJ65 (ORCPT ); Sat, 23 Jul 2022 05:58:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237328AbiGWJ6Z (ORCPT ); Sat, 23 Jul 2022 05:58:25 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B46F465D48; Sat, 23 Jul 2022 02:57:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4CA4DB82C1D; Sat, 23 Jul 2022 09:57:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B1890C341C0; Sat, 23 Jul 2022 09:57:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570244; bh=BG/lLgX4NwTF3EnPyMMuoXxPZhMhojq+mHhCZtZ5XxQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0FumJefwwlu8LbJXH9oAB5DiD+8oH/K/SG1NLapz/wmafOK9BA4W5zPHlv14O6VJe sEUQOwHqZwklp7fH8fSMUWEMGy2ZTnekxhlLkL+sHwtGPmt4Wq3BXiAMeqYqfAGihe 1XQE0hW253SZd9m8BfOuMJASArccBB9fQ9ESUtME= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 018/148] x86/insn-eval: Handle return values from the decoder Date: Sat, 23 Jul 2022 11:53:50 +0200 Message-Id: <20220723095229.574894347@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit 6e8c83d2a3afbfd5ee019ec720b75a42df515caa upstream. Now that the different instruction-inspecting functions return a value, test that and return early from callers if error has been encountered. While at it, do not call insn_get_modrm() when calling insn_get_displacement() because latter will make sure to call insn_get_modrm() if ModRM hasn't been parsed yet. Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210304174237.31945-6-bp@alien8.de Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/lib/insn-eval.c | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) --- a/arch/x86/lib/insn-eval.c +++ b/arch/x86/lib/insn-eval.c @@ -928,10 +928,11 @@ static int get_seg_base_limit(struct ins static int get_eff_addr_reg(struct insn *insn, struct pt_regs *regs, int *regoff, long *eff_addr) { - insn_get_modrm(insn); + int ret; =20 - if (!insn->modrm.nbytes) - return -EINVAL; + ret =3D insn_get_modrm(insn); + if (ret) + return ret; =20 if (X86_MODRM_MOD(insn->modrm.value) !=3D 3) return -EINVAL; @@ -977,14 +978,14 @@ static int get_eff_addr_modrm(struct ins int *regoff, long *eff_addr) { long tmp; + int ret; =20 if (insn->addr_bytes !=3D 8 && insn->addr_bytes !=3D 4) return -EINVAL; =20 - insn_get_modrm(insn); - - if (!insn->modrm.nbytes) - return -EINVAL; + ret =3D insn_get_modrm(insn); + if (ret) + return ret; =20 if (X86_MODRM_MOD(insn->modrm.value) > 2) return -EINVAL; @@ -1106,18 +1107,21 @@ static int get_eff_addr_modrm_16(struct * @base_offset will have a register, as an offset from the base of pt_reg= s, * that can be used to resolve the associated segment. * - * -EINVAL on error. + * Negative value on error. */ static int get_eff_addr_sib(struct insn *insn, struct pt_regs *regs, int *base_offset, long *eff_addr) { long base, indx; int indx_offset; + int ret; =20 if (insn->addr_bytes !=3D 8 && insn->addr_bytes !=3D 4) return -EINVAL; =20 - insn_get_modrm(insn); + ret =3D insn_get_modrm(insn); + if (ret) + return ret; =20 if (!insn->modrm.nbytes) return -EINVAL; @@ -1125,7 +1129,9 @@ static int get_eff_addr_sib(struct insn if (X86_MODRM_MOD(insn->modrm.value) > 2) return -EINVAL; =20 - insn_get_sib(insn); + ret =3D insn_get_sib(insn); + if (ret) + return ret; =20 if (!insn->sib.nbytes) return -EINVAL; @@ -1194,8 +1200,8 @@ static void __user *get_addr_ref_16(stru short eff_addr; long tmp; =20 - insn_get_modrm(insn); - insn_get_displacement(insn); + if (insn_get_displacement(insn)) + goto out; =20 if (insn->addr_bytes !=3D 2) goto out; @@ -1529,7 +1535,9 @@ bool insn_decode_from_regs(struct insn * insn->addr_bytes =3D INSN_CODE_SEG_ADDR_SZ(seg_defs); insn->opnd_bytes =3D INSN_CODE_SEG_OPND_SZ(seg_defs); =20 - insn_get_length(insn); + if (insn_get_length(insn)) + return false; + if (buf_size < insn->length) return false; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 918F4C433EF for ; Sat, 23 Jul 2022 09:59:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237701AbiGWJ7B (ORCPT ); Sat, 23 Jul 2022 05:59:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237640AbiGWJ60 (ORCPT ); Sat, 23 Jul 2022 05:58:26 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B97D66AE5; Sat, 23 Jul 2022 02:57:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 837EF611BD; Sat, 23 Jul 2022 09:57:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B84AC341C0; Sat, 23 Jul 2022 09:57:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570246; bh=5boINS1JazQ7nJt1mdx/fzP/a/m8Nko02sPbWZvm4I4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BSJsm8rR5INQVlioTM8WdjZyPV8m3gX7RD2fuWAPwQwGnrdaDdY8T235tB7wLyDct yfeT5A0EPDfK6eD9Zan+j/qySdkPpwppadw4EAhMNn8Nvd+iMsat3JvLskDIZYndWl lrXocwh+nT6NeWfv9a26YRPsfrw9a1Q/OjT5B6r4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 019/148] x86/alternative: Use insn_decode() Date: Sat, 23 Jul 2022 11:53:51 +0200 Message-Id: <20220723095229.850848060@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit 63c66cde7bbcc79aac14b25861c5b2495eede57b upstream. No functional changes, just simplification. Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/20210304174237.31945-10-bp@alien8.de Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/alternative.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1284,15 +1284,15 @@ static void text_poke_loc_init(struct te const void *opcode, size_t len, const void *emulate) { struct insn insn; + int ret; =20 memcpy((void *)tp->text, opcode, len); if (!emulate) emulate =3D opcode; =20 - kernel_insn_init(&insn, emulate, MAX_INSN_SIZE); - insn_get_length(&insn); + ret =3D insn_decode(&insn, emulate, MAX_INSN_SIZE, INSN_MODE_KERN); =20 - BUG_ON(!insn_complete(&insn)); + BUG_ON(ret < 0); BUG_ON(len !=3D insn.length); =20 tp->rel_addr =3D addr - (void *)_stext; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2B07C433EF for ; Sat, 23 Jul 2022 09:57:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237590AbiGWJ5k (ORCPT ); Sat, 23 Jul 2022 05:57:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237430AbiGWJ5M (ORCPT ); Sat, 23 Jul 2022 05:57:12 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF7313D58B; Sat, 23 Jul 2022 02:56:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4CC0F6117F; Sat, 23 Jul 2022 09:56:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B53BC341C0; Sat, 23 Jul 2022 09:56:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570217; bh=aXeY+HDy7y5ZCMPBdTis/j+MlUbP49bXjsG5jcTgf60=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LYfYDShHpoxkeJr26r9iGae+ilT0KKixbkijnsFhx3VGPrGOL1+UM3STHOlonFdlC UgI8ie35BA4udq0vnvZkj0s9A19RuE1esLP8hkb78YyiflbIQwkj17QCQE9bqXl+d+ q2MRXsztNv+o1e+/8xmU1lsqco3S3xXDGqqsl1L0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Ben Hutchings Subject: [PATCH 5.10 020/148] x86: Add insn_decode_kernel() Date: Sat, 23 Jul 2022 11:53:52 +0200 Message-Id: <20220723095230.124860988@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Ben Hutchings This was done by commit 52fa82c21f64e900a72437269a5cc9e0034b424e upstream, but this backport avoids changing all callers of the old decoder API. Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/insn.h | 2 ++ arch/x86/kernel/alternative.c | 2 +- tools/arch/x86/include/asm/insn.h | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/insn.h +++ b/arch/x86/include/asm/insn.h @@ -105,6 +105,8 @@ enum insn_mode { =20 extern int insn_decode(struct insn *insn, const void *kaddr, int buf_len, = enum insn_mode m); =20 +#define insn_decode_kernel(_insn, _ptr) insn_decode((_insn), (_ptr), MAX_I= NSN_SIZE, INSN_MODE_KERN) + /* Attribute will be determined after getting ModRM (for opcode groups) */ static inline void insn_get_attribute(struct insn *insn) { --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1290,7 +1290,7 @@ static void text_poke_loc_init(struct te if (!emulate) emulate =3D opcode; =20 - ret =3D insn_decode(&insn, emulate, MAX_INSN_SIZE, INSN_MODE_KERN); + ret =3D insn_decode_kernel(&insn, emulate); =20 BUG_ON(ret < 0); BUG_ON(len !=3D insn.length); --- a/tools/arch/x86/include/asm/insn.h +++ b/tools/arch/x86/include/asm/insn.h @@ -105,6 +105,8 @@ enum insn_mode { =20 extern int insn_decode(struct insn *insn, const void *kaddr, int buf_len, = enum insn_mode m); =20 +#define insn_decode_kernel(_insn, _ptr) insn_decode((_insn), (_ptr), MAX_I= NSN_SIZE, INSN_MODE_KERN) + /* Attribute will be determined after getting ModRM (for opcode groups) */ static inline void insn_get_attribute(struct insn *insn) { From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 018DAC43334 for ; Sat, 23 Jul 2022 09:57:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237604AbiGWJ5w (ORCPT ); Sat, 23 Jul 2022 05:57:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237531AbiGWJ5Q (ORCPT ); Sat, 23 Jul 2022 05:57:16 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49577459BE; Sat, 23 Jul 2022 02:57:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F3488B82C1B; Sat, 23 Jul 2022 09:57:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 24944C341C0; Sat, 23 Jul 2022 09:56:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570220; bh=k54y2zJbA/r6+ZajALij55gOxoHlB2/hOqkPpqVghB8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K5P9WFsi8eX+Nt+lEScIUzDPTzhOL6A0NwUNpS/gnED8jwy9DGX6t1ESBW2kCJLBr dDdHcXJgN4/Eq+2Cz5GhUWgiA9OVc9CmCbDN9AZ8ijbjez1D0lX/RNEYfGyGc8qw/3 kIm/do2FBPazwLue6eGlDxdYheyxWqTQ3HzaSZOo= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Ben Hutchings Subject: [PATCH 5.10 021/148] x86/alternatives: Optimize optimize_nops() Date: Sat, 23 Jul 2022 11:53:53 +0200 Message-Id: <20220723095230.410843401@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 23c1ad538f4f371bdb67d8a112314842d5db7e5a upstream. Currently, optimize_nops() scans to see if the alternative starts with NOPs. However, the emit pattern is: 141: \oldinstr 142: .skip (len-(142b-141b)), 0x90 That is, when 'oldinstr' is short, the tail is padded with NOPs. This case never gets optimized. Rewrite optimize_nops() to replace any trailing string of NOPs inside the alternative to larger NOPs. Also run it irrespective of patching, replacing NOPs in both the original and replaced code. A direct consequence is that 'padlen' becomes superfluous, so remove it. [ bp: - Adjust commit message - remove a stale comment about needing to pad - add a comment in optimize_nops() - exit early if the NOP verif. loop catches a mismatch - function should not not add NOPs in that case - fix the "optimized NOPs" offsets output ] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Link: https://lkml.kernel.org/r/20210326151259.442992235@infradead.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/alternative.h | 17 ++------- arch/x86/kernel/alternative.c | 49 ++++++++++++++++-----= ----- tools/objtool/arch/x86/include/arch_special.h | 2 - 3 files changed, 37 insertions(+), 31 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -65,7 +65,6 @@ struct alt_instr { u16 cpuid; /* cpuid bit set for replacement */ u8 instrlen; /* length of original instruction */ u8 replacementlen; /* length of new instruction */ - u8 padlen; /* length of build-time padding */ } __packed; =20 /* @@ -104,7 +103,6 @@ static inline int alternatives_text_rese =20 #define alt_end_marker "663" #define alt_slen "662b-661b" -#define alt_pad_len alt_end_marker"b-662b" #define alt_total_slen alt_end_marker"b-661b" #define alt_rlen(num) e_replacement(num)"f-"b_replacement(num)"f" =20 @@ -151,8 +149,7 @@ static inline int alternatives_text_rese " .long " b_replacement(num)"f - .\n" /* new instruction */ \ " .word " __stringify(feature) "\n" /* feature bit */ \ " .byte " alt_total_slen "\n" /* source len */ \ - " .byte " alt_rlen(num) "\n" /* replacement len */ \ - " .byte " alt_pad_len "\n" /* pad len */ + " .byte " alt_rlen(num) "\n" /* replacement len */ =20 #define ALTINSTR_REPLACEMENT(newinstr, feature, num) /* replacement */ \ "# ALT: replacement " #num "\n" \ @@ -224,9 +221,6 @@ static inline int alternatives_text_rese * Peculiarities: * No memory clobber here. * Argument numbers start with 1. - * Best is to use constraints that are fixed size (like (%1) ... "r") - * If you use variable sized constraints like "m" or "g" in the - * replacement make sure to pad to the worst case length. * Leaving an unused argument 0 to keep API compatibility. */ #define alternative_input(oldinstr, newinstr, feature, input...) \ @@ -315,13 +309,12 @@ static inline int alternatives_text_rese * enough information for the alternatives patching code to patch an * instruction. See apply_alternatives(). */ -.macro altinstruction_entry orig alt feature orig_len alt_len pad_len +.macro altinstruction_entry orig alt feature orig_len alt_len .long \orig - . .long \alt - . .word \feature .byte \orig_len .byte \alt_len - .byte \pad_len .endm =20 /* @@ -338,7 +331,7 @@ static inline int alternatives_text_rese 142: =20 .pushsection .altinstructions,"a" - altinstruction_entry 140b,143f,\feature,142b-140b,144f-143f,142b-141b + altinstruction_entry 140b,143f,\feature,142b-140b,144f-143f .popsection =20 .pushsection .altinstr_replacement,"ax" @@ -375,8 +368,8 @@ static inline int alternatives_text_rese 142: =20 .pushsection .altinstructions,"a" - altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f,142b-141b - altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f,142b-141b + altinstruction_entry 140b,143f,\feature1,142b-140b,144f-143f + altinstruction_entry 140b,144f,\feature2,142b-140b,145f-144f .popsection =20 .pushsection .altinstr_replacement,"ax" --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -344,19 +344,35 @@ done: static void __init_or_module noinline optimize_nops(struct alt_instr *a, u= 8 *instr) { unsigned long flags; - int i; + struct insn insn; + int nop, i =3D 0; =20 - for (i =3D 0; i < a->padlen; i++) { - if (instr[i] !=3D 0x90) + /* + * Jump over the non-NOP insns, the remaining bytes must be single-byte + * NOPs, optimize them. + */ + for (;;) { + if (insn_decode_kernel(&insn, &instr[i])) + return; + + if (insn.length =3D=3D 1 && insn.opcode.bytes[0] =3D=3D 0x90) + break; + + if ((i +=3D insn.length) >=3D a->instrlen) + return; + } + + for (nop =3D i; i < a->instrlen; i++) { + if (WARN_ONCE(instr[i] !=3D 0x90, "Not a NOP at 0x%px\n", &instr[i])) return; } =20 local_irq_save(flags); - add_nops(instr + (a->instrlen - a->padlen), a->padlen); + add_nops(instr + nop, i - nop); local_irq_restore(flags); =20 DUMP_BYTES(instr, a->instrlen, "%px: [%d:%d) optimized NOPs: ", - instr, a->instrlen - a->padlen, a->padlen); + instr, nop, a->instrlen); } =20 /* @@ -402,19 +418,15 @@ void __init_or_module noinline apply_alt * - feature not present but ALTINSTR_FLAG_INV is set to mean, * patch if feature is *NOT* present. */ - if (!boot_cpu_has(feature) =3D=3D !(a->cpuid & ALTINSTR_FLAG_INV)) { - if (a->padlen > 1) - optimize_nops(a, instr); - - continue; - } + if (!boot_cpu_has(feature) =3D=3D !(a->cpuid & ALTINSTR_FLAG_INV)) + goto next; =20 - DPRINTK("feat: %s%d*32+%d, old: (%pS (%px) len: %d), repl: (%px, len: %d= ), pad: %d", + DPRINTK("feat: %s%d*32+%d, old: (%pS (%px) len: %d), repl: (%px, len: %d= )", (a->cpuid & ALTINSTR_FLAG_INV) ? "!" : "", feature >> 5, feature & 0x1f, instr, instr, a->instrlen, - replacement, a->replacementlen, a->padlen); + replacement, a->replacementlen); =20 DUMP_BYTES(instr, a->instrlen, "%px: old_insn: ", instr); DUMP_BYTES(replacement, a->replacementlen, "%px: rpl_insn: ", replacemen= t); @@ -438,14 +450,15 @@ void __init_or_module noinline apply_alt if (a->replacementlen && is_jmp(replacement[0])) recompute_jump(a, instr, replacement, insn_buff); =20 - if (a->instrlen > a->replacementlen) { - add_nops(insn_buff + a->replacementlen, - a->instrlen - a->replacementlen); - insn_buff_sz +=3D a->instrlen - a->replacementlen; - } + for (; insn_buff_sz < a->instrlen; insn_buff_sz++) + insn_buff[insn_buff_sz] =3D 0x90; + DUMP_BYTES(insn_buff, insn_buff_sz, "%px: final_insn: ", instr); =20 text_poke_early(instr, insn_buff, insn_buff_sz); + +next: + optimize_nops(a, instr); } } =20 --- a/tools/objtool/arch/x86/include/arch_special.h +++ b/tools/objtool/arch/x86/include/arch_special.h @@ -10,7 +10,7 @@ #define JUMP_ORIG_OFFSET 0 #define JUMP_NEW_OFFSET 4 =20 -#define ALT_ENTRY_SIZE 13 +#define ALT_ENTRY_SIZE 12 #define ALT_ORIG_OFFSET 0 #define ALT_NEW_OFFSET 4 #define ALT_FEATURE_OFFSET 8 From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D260C433EF for ; Sat, 23 Jul 2022 09:58:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237407AbiGWJ6J (ORCPT ); Sat, 23 Jul 2022 05:58:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237532AbiGWJ5Q (ORCPT ); Sat, 23 Jul 2022 05:57:16 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6ACC474D8; Sat, 23 Jul 2022 02:57:04 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 35D606117F; Sat, 23 Jul 2022 09:57:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 16104C341C0; Sat, 23 Jul 2022 09:57:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570223; bh=Sn8JMrIl2IDnUyeKrj7T8ivawqQC2JCHM7kxIMlJeUI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Yy6i3M85xxbhlI2OaUqU2+dieXnJD27nVVrn7gs+erkkskPKaWJIro1hxC6QHTLqL A49imZfH1o4abpWOHfKUIXM5csnAFuSlH0XWfuI5T/E5G90vOZ9bCIcpt24/riw1fy qf1gRkLGuDnRQWWZMi41fGHRr0iWdqgj6EVgw08I= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Ben Hutchings Subject: [PATCH 5.10 022/148] x86/retpoline: Simplify retpolines Date: Sat, 23 Jul 2022 11:53:54 +0200 Message-Id: <20220723095230.677297796@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 119251855f9adf9421cb5eb409933092141ab2c7 upstream. Due to: c9c324dc22aa ("objtool: Support stack layout changes in alternatives") it is now possible to simplify the retpolines. Currently our retpolines consist of 2 symbols: - __x86_indirect_thunk_\reg: the compiler target - __x86_retpoline_\reg: the actual retpoline. Both are consecutive in code and aligned such that for any one register they both live in the same cacheline: 0000000000000000 <__x86_indirect_thunk_rax>: 0: ff e0 jmpq *%rax 2: 90 nop 3: 90 nop 4: 90 nop 0000000000000005 <__x86_retpoline_rax>: 5: e8 07 00 00 00 callq 11 <__x86_retpoline_rax+0xc> a: f3 90 pause c: 0f ae e8 lfence f: eb f9 jmp a <__x86_retpoline_rax+0x5> 11: 48 89 04 24 mov %rax,(%rsp) 15: c3 retq 16: 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:0x0(%rax,%rax,1) The thunk is an alternative_2, where one option is a JMP to the retpoline. This was done so that objtool didn't need to deal with alternatives with stack ops. But that problem has been solved, so now it is possible to fold the entire retpoline into the alternative to simplify and consolidate unused bytes: 0000000000000000 <__x86_indirect_thunk_rax>: 0: ff e0 jmpq *%rax 2: 90 nop 3: 90 nop 4: 90 nop 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 66 66 2e 0f 1f 84 00 00 00 00 00 data16 nopw %cs:0x0(%rax,%r= ax,1) 1c: 0f 1f 40 00 nopl 0x0(%rax) Notice that since the longest alternative sequence is now: 0: e8 07 00 00 00 callq c <.altinstr_replacement+0xc> 5: f3 90 pause 7: 0f ae e8 lfence a: eb f9 jmp 5 <.altinstr_replacement+0x5> c: 48 89 04 24 mov %rax,(%rsp) 10: c3 retq 17 bytes, we have 15 bytes NOP at the end of our 32 byte slot. (IOW, if we can shrink the retpoline by 1 byte we can pack it more densely). [ bp: Massage commit message. ] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Link: https://lkml.kernel.org/r/20210326151259.506071949@infradead.org [bwh: Backported to 5.10: - Use X86_FEATRURE_RETPOLINE_LFENCE flag instead of X86_FEATURE_RETPOLINE_AMD, since the later renaming of this flag has already been applied - Adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/asm-prototypes.h | 7 ------- arch/x86/include/asm/nospec-branch.h | 6 +++--- arch/x86/lib/retpoline.S | 34 +++++++++++++++++------------= ----- tools/objtool/check.c | 3 +-- 4 files changed, 21 insertions(+), 29 deletions(-) --- a/arch/x86/include/asm/asm-prototypes.h +++ b/arch/x86/include/asm/asm-prototypes.h @@ -22,15 +22,8 @@ extern void cmpxchg8b_emu(void); #define DECL_INDIRECT_THUNK(reg) \ extern asmlinkage void __x86_indirect_thunk_ ## reg (void); =20 -#define DECL_RETPOLINE(reg) \ - extern asmlinkage void __x86_retpoline_ ## reg (void); - #undef GEN #define GEN(reg) DECL_INDIRECT_THUNK(reg) #include =20 -#undef GEN -#define GEN(reg) DECL_RETPOLINE(reg) -#include - #endif /* CONFIG_RETPOLINE */ --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -80,7 +80,7 @@ .macro JMP_NOSPEC reg:req #ifdef CONFIG_RETPOLINE ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ - __stringify(jmp __x86_retpoline_\reg), X86_FEATURE_RETPOLINE, \ + __stringify(jmp __x86_indirect_thunk_\reg), X86_FEATURE_RETPOLINE,= \ __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), X86_FEAT= URE_RETPOLINE_LFENCE #else jmp *%\reg @@ -90,7 +90,7 @@ .macro CALL_NOSPEC reg:req #ifdef CONFIG_RETPOLINE ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; call *%\reg), \ - __stringify(call __x86_retpoline_\reg), X86_FEATURE_RETPOLINE, \ + __stringify(call __x86_indirect_thunk_\reg), X86_FEATURE_RETPOLINE= , \ __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; call *%\reg), X86_FEA= TURE_RETPOLINE_LFENCE #else call *%\reg @@ -128,7 +128,7 @@ ALTERNATIVE_2( \ ANNOTATE_RETPOLINE_SAFE \ "call *%[thunk_target]\n", \ - "call __x86_retpoline_%V[thunk_target]\n", \ + "call __x86_indirect_thunk_%V[thunk_target]\n", \ X86_FEATURE_RETPOLINE, \ "lfence;\n" \ ANNOTATE_RETPOLINE_SAFE \ --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -10,27 +10,31 @@ #include #include =20 -.macro THUNK reg - .section .text.__x86.indirect_thunk - - .align 32 -SYM_FUNC_START(__x86_indirect_thunk_\reg) - JMP_NOSPEC \reg -SYM_FUNC_END(__x86_indirect_thunk_\reg) - -SYM_FUNC_START_NOALIGN(__x86_retpoline_\reg) +.macro RETPOLINE reg ANNOTATE_INTRA_FUNCTION_CALL - call .Ldo_rop_\@ + call .Ldo_rop_\@ .Lspec_trap_\@: UNWIND_HINT_EMPTY pause lfence - jmp .Lspec_trap_\@ + jmp .Lspec_trap_\@ .Ldo_rop_\@: - mov %\reg, (%_ASM_SP) + mov %\reg, (%_ASM_SP) UNWIND_HINT_FUNC ret -SYM_FUNC_END(__x86_retpoline_\reg) +.endm + +.macro THUNK reg + .section .text.__x86.indirect_thunk + + .align 32 +SYM_FUNC_START(__x86_indirect_thunk_\reg) + + ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ + __stringify(RETPOLINE \reg), X86_FEATURE_RETPOLINE, \ + __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), X86_FEAT= URE_RETPOLINE_LFENCE + +SYM_FUNC_END(__x86_indirect_thunk_\reg) =20 .endm =20 @@ -48,7 +52,6 @@ SYM_FUNC_END(__x86_retpoline_\reg) =20 #define __EXPORT_THUNK(sym) _ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym) #define EXPORT_THUNK(reg) __EXPORT_THUNK(__x86_indirect_thunk_ ## reg) -#define EXPORT_RETPOLINE(reg) __EXPORT_THUNK(__x86_retpoline_ ## reg) =20 #undef GEN #define GEN(reg) THUNK reg @@ -58,6 +61,3 @@ SYM_FUNC_END(__x86_retpoline_\reg) #define GEN(reg) EXPORT_THUNK(reg) #include =20 -#undef GEN -#define GEN(reg) EXPORT_RETPOLINE(reg) -#include --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -800,8 +800,7 @@ static int add_jump_destinations(struct } else if (reloc->sym->type =3D=3D STT_SECTION) { dest_sec =3D reloc->sym->sec; dest_off =3D arch_dest_reloc_offset(reloc->addend); - } else if (!strncmp(reloc->sym->name, "__x86_indirect_thunk_", 21) || - !strncmp(reloc->sym->name, "__x86_retpoline_", 16)) { + } else if (!strncmp(reloc->sym->name, "__x86_indirect_thunk_", 21)) { /* * Retpoline jumps are really dynamic jumps in * disguise, so convert them accordingly. From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3179C433EF for ; Sat, 23 Jul 2022 10:00:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237881AbiGWKAV (ORCPT ); Sat, 23 Jul 2022 06:00:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237777AbiGWJ7g (ORCPT ); Sat, 23 Jul 2022 05:59:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0747F70E54; Sat, 23 Jul 2022 02:58:05 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8AEDB611BD; Sat, 23 Jul 2022 09:58:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B677C341C0; Sat, 23 Jul 2022 09:58:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570284; bh=lsyuTmXLAidtYPc/lopZFL7NAHk8kmzndFF44qwQcAs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UcaF+rmWykXZEH6L6JVTuoUWExanRUCO//olGC9XPWulS4DiBWhvvt9uqid0c5Cq2 f+NNSGaGqJxhpa41aNCFg5o81Ham4Y0WBA1qB7s23LCbBRAXUSjDiVLETAN2F1ITqx bE3Z81IM/cLoNpZfy2cmb8gqZzSFFwCGJ2oivJB4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 023/148] objtool: Correctly handle retpoline thunk calls Date: Sat, 23 Jul 2022 11:53:55 +0200 Message-Id: <20220723095230.917495578@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit bcb1b6ff39da7e8a6a986eb08126fba2b5e13c32 upstream. Just like JMP handling, convert a direct CALL to a retpoline thunk into a retpoline safe indirect CALL. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151259.567568238@infradead.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -953,6 +953,18 @@ static int add_call_destinations(struct dest_off); return -1; } + + } else if (!strncmp(reloc->sym->name, "__x86_indirect_thunk_", 21)) { + /* + * Retpoline calls are really dynamic calls in + * disguise, so convert them accordingly. + */ + insn->type =3D INSN_CALL_DYNAMIC; + insn->retpoline_safe =3D true; + + remove_insn_ops(insn); + continue; + } else insn->call_dest =3D reloc->sym; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D212C433EF for ; Sat, 23 Jul 2022 09:59:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237528AbiGWJ7U (ORCPT ); Sat, 23 Jul 2022 05:59:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237530AbiGWJ6l (ORCPT ); Sat, 23 Jul 2022 05:58:41 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD5043F32F; Sat, 23 Jul 2022 02:57:33 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3CB07611BF; Sat, 23 Jul 2022 09:57:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4BD52C341C0; Sat, 23 Jul 2022 09:57:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570252; bh=U0FSku5Q4Vl2EtM25WEcra7RlQWFvp8fSemwdMB1/5M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YTOLFjfIhE/ggkQI0QKqeRgXR1mB2l38W2FKT3S7RLziKxO4d9zD6Js9adNGdYr6G Dxd8GqEiIEG7kWpCsOrcUNBEbNqlBMBR8OqGbnrlvPAA55IxX1ZHvy4YwyfwYOtHWs OFsRTZpgqEQuGi0gKttaW19RxWLVZF8Wt9igS+AI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 024/148] objtool: Handle per arch retpoline naming Date: Sat, 23 Jul 2022 11:53:56 +0200 Message-Id: <20220723095231.169139222@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 530b4ddd9dd92b263081f5c7786d39a8129c8b2d upstream. The __x86_indirect_ naming is obviously not generic. Shorten to allow matching some additional magic names later. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151259.630296706@infradead.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch.h | 2 ++ tools/objtool/arch/x86/decode.c | 5 +++++ tools/objtool/check.c | 9 +++++++-- 3 files changed, 14 insertions(+), 2 deletions(-) --- a/tools/objtool/arch.h +++ b/tools/objtool/arch.h @@ -86,4 +86,6 @@ const char *arch_nop_insn(int len); =20 int arch_decode_hint_reg(struct instruction *insn, u8 sp_reg); =20 +bool arch_is_retpoline(struct symbol *sym); + #endif /* _ARCH_H */ --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -620,3 +620,8 @@ int arch_decode_hint_reg(struct instruct =20 return 0; } + +bool arch_is_retpoline(struct symbol *sym) +{ + return !strncmp(sym->name, "__x86_indirect_", 15); +} --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -778,6 +778,11 @@ static int add_ignore_alternatives(struc return 0; } =20 +__weak bool arch_is_retpoline(struct symbol *sym) +{ + return false; +} + /* * Find the destination instructions for all jumps. */ @@ -800,7 +805,7 @@ static int add_jump_destinations(struct } else if (reloc->sym->type =3D=3D STT_SECTION) { dest_sec =3D reloc->sym->sec; dest_off =3D arch_dest_reloc_offset(reloc->addend); - } else if (!strncmp(reloc->sym->name, "__x86_indirect_thunk_", 21)) { + } else if (arch_is_retpoline(reloc->sym)) { /* * Retpoline jumps are really dynamic jumps in * disguise, so convert them accordingly. @@ -954,7 +959,7 @@ static int add_call_destinations(struct return -1; } =20 - } else if (!strncmp(reloc->sym->name, "__x86_indirect_thunk_", 21)) { + } else if (arch_is_retpoline(reloc->sym)) { /* * Retpoline calls are really dynamic calls in * disguise, so convert them accordingly. From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 483ACCCA483 for ; Sat, 23 Jul 2022 09:59:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237788AbiGWJ7j (ORCPT ); Sat, 23 Jul 2022 05:59:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237488AbiGWJ64 (ORCPT ); Sat, 23 Jul 2022 05:58:56 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE94C6D2CF; Sat, 23 Jul 2022 02:57:43 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7873BB827CC; Sat, 23 Jul 2022 09:57:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CCC8EC341C0; Sat, 23 Jul 2022 09:57:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570261; bh=fUn8IkFO0m8VZnxYLrY3/qaO1ly3x4v0tCiWNKY25ks=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qfghO25L21MW3RlV1WF/RsFn+Hlwdt9E85eMWfIcQvu/mgFak0Hg9Tmhp1hxcxHnm 3OxRznv1Cfhe7w+aD9nFUI4/9zOa3/86ZO3tTewJnH+4X0pSQ2c1GNcKnwqk6ZvUZe 1KQTbr8UdvT5B0wYxDVGUVnzB9AOJ9eTBb+ChZoA= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 025/148] objtool: Rework the elf_rebuild_reloc_section() logic Date: Sat, 23 Jul 2022 11:53:57 +0200 Message-Id: <20220723095231.451413936@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 3a647607b57ad8346e659ddd3b951ac292c83690 upstream. Instead of manually calling elf_rebuild_reloc_section() on sections we've called elf_add_reloc() on, have elf_write() DTRT. This makes it easier to add random relocations in places without carefully tracking when we're done and need to flush what section. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151259.754213408@infradead.org [bwh: Backported to 5.10: drop changes in create_mcount_loc_sections()] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 3 --- tools/objtool/elf.c | 20 ++++++++++++++------ tools/objtool/elf.h | 1 - tools/objtool/orc_gen.c | 3 --- 4 files changed, 14 insertions(+), 13 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -542,9 +542,6 @@ static int create_static_call_sections(s idx++; } =20 - if (elf_rebuild_reloc_section(file->elf, reloc_sec)) - return -1; - return 0; } =20 --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -530,6 +530,8 @@ void elf_add_reloc(struct elf *elf, stru =20 list_add_tail(&reloc->list, &sec->reloc_list); elf_hash_add(elf->reloc_hash, &reloc->hash, reloc_hash(reloc)); + + sec->changed =3D true; } =20 static int read_rel_reloc(struct section *sec, int i, struct reloc *reloc,= unsigned int *symndx) @@ -609,7 +611,9 @@ static int read_relocs(struct elf *elf) return -1; } =20 - elf_add_reloc(elf, reloc); + list_add_tail(&reloc->list, &sec->reloc_list); + elf_hash_add(elf->reloc_hash, &reloc->hash, reloc_hash(reloc)); + nr_reloc++; } max_reloc =3D max(max_reloc, nr_reloc); @@ -920,14 +924,11 @@ static int elf_rebuild_rela_reloc_sectio return 0; } =20 -int elf_rebuild_reloc_section(struct elf *elf, struct section *sec) +static int elf_rebuild_reloc_section(struct elf *elf, struct section *sec) { struct reloc *reloc; int nr; =20 - sec->changed =3D true; - elf->changed =3D true; - nr =3D 0; list_for_each_entry(reloc, &sec->reloc_list, list) nr++; @@ -991,9 +992,15 @@ int elf_write(struct elf *elf) struct section *sec; Elf_Scn *s; =20 - /* Update section headers for changed sections: */ + /* Update changed relocation sections and section headers: */ list_for_each_entry(sec, &elf->sections, list) { if (sec->changed) { + if (sec->base && + elf_rebuild_reloc_section(elf, sec)) { + WARN("elf_rebuild_reloc_section"); + return -1; + } + s =3D elf_getscn(elf->elf, sec->idx); if (!s) { WARN_ELF("elf_getscn"); @@ -1005,6 +1012,7 @@ int elf_write(struct elf *elf) } =20 sec->changed =3D false; + elf->changed =3D true; } } =20 --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -142,7 +142,6 @@ struct reloc *find_reloc_by_dest_range(c struct symbol *find_func_containing(struct section *sec, unsigned long off= set); void insn_to_reloc_sym_addend(struct section *sec, unsigned long offset, struct reloc *reloc); -int elf_rebuild_reloc_section(struct elf *elf, struct section *sec); =20 #define for_each_sec(file, sec) \ list_for_each_entry(sec, &file->elf->sections, list) --- a/tools/objtool/orc_gen.c +++ b/tools/objtool/orc_gen.c @@ -251,8 +251,5 @@ int orc_create(struct objtool_file *file return -1; } =20 - if (elf_rebuild_reloc_section(file->elf, ip_rsec)) - return -1; - return 0; } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A47EDC43334 for ; Sat, 23 Jul 2022 09:59:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237430AbiGWJ7m (ORCPT ); Sat, 23 Jul 2022 05:59:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237491AbiGWJ64 (ORCPT ); Sat, 23 Jul 2022 05:58:56 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 239416D54E; Sat, 23 Jul 2022 02:57:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9D921611CD; Sat, 23 Jul 2022 09:57:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A6F17C341C0; Sat, 23 Jul 2022 09:57:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570264; bh=7ChS9UM2HIV8mnaiUuB29BJ3uVL/4KUef/DVQmisIbg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=yonSjfIl80ygtUpv8A6M7geDETFphxmA5tsdT9r4ZyxUeHmKcbkrjTaNGygDpgmRB quf3Ttwn/6DIC38UMtAqojNDVFdZt7PpD/tiL7aguUWMAMI+JaQkKWqYuLzOB/dqVa UDGCftjH1T8uhCtWKtC+45lvCyhFEcT9yhg+smEo= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 026/148] objtool: Add elf_create_reloc() helper Date: Sat, 23 Jul 2022 11:53:58 +0200 Message-Id: <20220723095231.747365666@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit ef47cc01cb4abcd760d8ac66b9361d6ade4d0846 upstream. We have 4 instances of adding a relocation. Create a common helper to avoid growing even more. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151259.817438847@infradead.org [bwh: Backported to 5.10: drop changes in create_mcount_loc_sections()] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 43 +++++------------------- tools/objtool/elf.c | 86 +++++++++++++++++++++++++++++++------------= ----- tools/objtool/elf.h | 10 +++-- tools/objtool/orc_gen.c | 30 +++------------- 4 files changed, 79 insertions(+), 90 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -433,8 +433,7 @@ reachable: =20 static int create_static_call_sections(struct objtool_file *file) { - struct section *sec, *reloc_sec; - struct reloc *reloc; + struct section *sec; struct static_call_site *site; struct instruction *insn; struct symbol *key_sym; @@ -460,8 +459,7 @@ static int create_static_call_sections(s if (!sec) return -1; =20 - reloc_sec =3D elf_create_reloc_section(file->elf, sec, SHT_RELA); - if (!reloc_sec) + if (!elf_create_reloc_section(file->elf, sec, SHT_RELA)) return -1; =20 idx =3D 0; @@ -471,25 +469,11 @@ static int create_static_call_sections(s memset(site, 0, sizeof(struct static_call_site)); =20 /* populate reloc for 'addr' */ - reloc =3D malloc(sizeof(*reloc)); - - if (!reloc) { - perror("malloc"); + if (elf_add_reloc_to_insn(file->elf, sec, + idx * sizeof(struct static_call_site), + R_X86_64_PC32, + insn->sec, insn->offset)) return -1; - } - memset(reloc, 0, sizeof(*reloc)); - - insn_to_reloc_sym_addend(insn->sec, insn->offset, reloc); - if (!reloc->sym) { - WARN_FUNC("static call tramp: missing containing symbol", - insn->sec, insn->offset); - return -1; - } - - reloc->type =3D R_X86_64_PC32; - reloc->offset =3D idx * sizeof(struct static_call_site); - reloc->sec =3D reloc_sec; - elf_add_reloc(file->elf, reloc); =20 /* find key symbol */ key_name =3D strdup(insn->call_dest->name); @@ -526,18 +510,11 @@ static int create_static_call_sections(s free(key_name); =20 /* populate reloc for 'key' */ - reloc =3D malloc(sizeof(*reloc)); - if (!reloc) { - perror("malloc"); + if (elf_add_reloc(file->elf, sec, + idx * sizeof(struct static_call_site) + 4, + R_X86_64_PC32, key_sym, + is_sibling_call(insn) * STATIC_CALL_SITE_TAIL)) return -1; - } - memset(reloc, 0, sizeof(*reloc)); - reloc->sym =3D key_sym; - reloc->addend =3D is_sibling_call(insn) ? STATIC_CALL_SITE_TAIL : 0; - reloc->type =3D R_X86_64_PC32; - reloc->offset =3D idx * sizeof(struct static_call_site) + 4; - reloc->sec =3D reloc_sec; - elf_add_reloc(file->elf, reloc); =20 idx++; } --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -262,32 +262,6 @@ struct reloc *find_reloc_by_dest(const s return find_reloc_by_dest_range(elf, sec, offset, 1); } =20 -void insn_to_reloc_sym_addend(struct section *sec, unsigned long offset, - struct reloc *reloc) -{ - if (sec->sym) { - reloc->sym =3D sec->sym; - reloc->addend =3D offset; - return; - } - - /* - * The Clang assembler strips section symbols, so we have to reference - * the function symbol instead: - */ - reloc->sym =3D find_symbol_containing(sec, offset); - if (!reloc->sym) { - /* - * Hack alert. This happens when we need to reference the NOP - * pad insn immediately after the function. - */ - reloc->sym =3D find_symbol_containing(sec, offset - 1); - } - - if (reloc->sym) - reloc->addend =3D offset - reloc->sym->offset; -} - static int read_sections(struct elf *elf) { Elf_Scn *s =3D NULL; @@ -524,14 +498,66 @@ err: return -1; } =20 -void elf_add_reloc(struct elf *elf, struct reloc *reloc) +int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, + unsigned int type, struct symbol *sym, int addend) { - struct section *sec =3D reloc->sec; + struct reloc *reloc; =20 - list_add_tail(&reloc->list, &sec->reloc_list); + reloc =3D malloc(sizeof(*reloc)); + if (!reloc) { + perror("malloc"); + return -1; + } + memset(reloc, 0, sizeof(*reloc)); + + reloc->sec =3D sec->reloc; + reloc->offset =3D offset; + reloc->type =3D type; + reloc->sym =3D sym; + reloc->addend =3D addend; + + list_add_tail(&reloc->list, &sec->reloc->reloc_list); elf_hash_add(elf->reloc_hash, &reloc->hash, reloc_hash(reloc)); =20 - sec->changed =3D true; + sec->reloc->changed =3D true; + + return 0; +} + +int elf_add_reloc_to_insn(struct elf *elf, struct section *sec, + unsigned long offset, unsigned int type, + struct section *insn_sec, unsigned long insn_off) +{ + struct symbol *sym; + int addend; + + if (insn_sec->sym) { + sym =3D insn_sec->sym; + addend =3D insn_off; + + } else { + /* + * The Clang assembler strips section symbols, so we have to + * reference the function symbol instead: + */ + sym =3D find_symbol_containing(insn_sec, insn_off); + if (!sym) { + /* + * Hack alert. This happens when we need to reference + * the NOP pad insn immediately after the function. + */ + sym =3D find_symbol_containing(insn_sec, insn_off - 1); + } + + if (!sym) { + WARN("can't find symbol containing %s+0x%lx", insn_sec->name, insn_off); + return -1; + } + + addend =3D insn_off - sym->offset; + } + + return elf_add_reloc(elf, sec, offset, type, sym, addend); } =20 static int read_rel_reloc(struct section *sec, int i, struct reloc *reloc,= unsigned int *symndx) --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -123,7 +123,13 @@ static inline u32 reloc_hash(struct relo struct elf *elf_open_read(const char *name, int flags); struct section *elf_create_section(struct elf *elf, const char *name, unsi= gned int sh_flags, size_t entsize, int nr); struct section *elf_create_reloc_section(struct elf *elf, struct section *= base, int reltype); -void elf_add_reloc(struct elf *elf, struct reloc *reloc); + +int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, + unsigned int type, struct symbol *sym, int addend); +int elf_add_reloc_to_insn(struct elf *elf, struct section *sec, + unsigned long offset, unsigned int type, + struct section *insn_sec, unsigned long insn_off); + int elf_write_insn(struct elf *elf, struct section *sec, unsigned long offset, unsigned int len, const char *insn); @@ -140,8 +146,6 @@ struct reloc *find_reloc_by_dest(const s struct reloc *find_reloc_by_dest_range(const struct elf *elf, struct secti= on *sec, unsigned long offset, unsigned int len); struct symbol *find_func_containing(struct section *sec, unsigned long off= set); -void insn_to_reloc_sym_addend(struct section *sec, unsigned long offset, - struct reloc *reloc); =20 #define for_each_sec(file, sec) \ list_for_each_entry(sec, &file->elf->sections, list) --- a/tools/objtool/orc_gen.c +++ b/tools/objtool/orc_gen.c @@ -81,37 +81,20 @@ static int init_orc_entry(struct orc_ent } =20 static int write_orc_entry(struct elf *elf, struct section *orc_sec, - struct section *ip_rsec, unsigned int idx, + struct section *ip_sec, unsigned int idx, struct section *insn_sec, unsigned long insn_off, struct orc_entry *o) { struct orc_entry *orc; - struct reloc *reloc; =20 /* populate ORC data */ orc =3D (struct orc_entry *)orc_sec->data->d_buf + idx; memcpy(orc, o, sizeof(*orc)); =20 /* populate reloc for ip */ - reloc =3D malloc(sizeof(*reloc)); - if (!reloc) { - perror("malloc"); + if (elf_add_reloc_to_insn(elf, ip_sec, idx * sizeof(int), R_X86_64_PC32, + insn_sec, insn_off)) return -1; - } - memset(reloc, 0, sizeof(*reloc)); - - insn_to_reloc_sym_addend(insn_sec, insn_off, reloc); - if (!reloc->sym) { - WARN("missing symbol for insn at offset 0x%lx", - insn_off); - return -1; - } - - reloc->type =3D R_X86_64_PC32; - reloc->offset =3D idx * sizeof(int); - reloc->sec =3D ip_rsec; - - elf_add_reloc(elf, reloc); =20 return 0; } @@ -150,7 +133,7 @@ static unsigned long alt_group_len(struc =20 int orc_create(struct objtool_file *file) { - struct section *sec, *ip_rsec, *orc_sec; + struct section *sec, *orc_sec; unsigned int nr =3D 0, idx =3D 0; struct orc_list_entry *entry; struct list_head orc_list; @@ -239,13 +222,12 @@ int orc_create(struct objtool_file *file sec =3D elf_create_section(file->elf, ".orc_unwind_ip", 0, sizeof(int), n= r); if (!sec) return -1; - ip_rsec =3D elf_create_reloc_section(file->elf, sec, SHT_RELA); - if (!ip_rsec) + if (!elf_create_reloc_section(file->elf, sec, SHT_RELA)) return -1; =20 /* Write ORC entries to sections: */ list_for_each_entry(entry, &orc_list, list) { - if (write_orc_entry(file->elf, orc_sec, ip_rsec, idx++, + if (write_orc_entry(file->elf, orc_sec, sec, idx++, entry->insn_sec, entry->insn_off, &entry->orc)) return -1; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B979C43334 for ; Sat, 23 Jul 2022 09:59:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237668AbiGWJ7o (ORCPT ); Sat, 23 Jul 2022 05:59:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237573AbiGWJ7H (ORCPT ); Sat, 23 Jul 2022 05:59:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E001D45078; Sat, 23 Jul 2022 02:57:47 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 64D51611BD; Sat, 23 Jul 2022 09:57:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70672C341C0; Sat, 23 Jul 2022 09:57:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570266; bh=P8MOob0Am9dbWMK9nvqs5Mo/UzNxM2C3QOR3rxgRsAo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=D2Ku1SZjmyrUpYjJWBfCdKtR95OydiUszxN5jOiJD+Vtm9O4Ni4KJexcYEcqqOAZ6 LOaEGMGxwh7QhkSI9O8gIx6pDgCRvN+tbmjpj/mRt4QeMFMVV7CxRWTFU52PV8J3wg MQgibhp1uedM27xYf8ta1sBfb9N3+FL6fiqOZhEo= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 027/148] objtool: Create reloc sections implicitly Date: Sat, 23 Jul 2022 11:53:59 +0200 Message-Id: <20220723095232.016728630@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit d0c5c4cc73da0b05b0d9e5f833f2d859e1b45f8e upstream. Have elf_add_reloc() create the relocation section implicitly. Suggested-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151259.880174448@infradead.org [bwh: Backported to 5.10: drop changes in create_mcount_loc_sections()] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 3 --- tools/objtool/elf.c | 9 ++++++++- tools/objtool/elf.h | 1 - tools/objtool/orc_gen.c | 2 -- 4 files changed, 8 insertions(+), 7 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -459,9 +459,6 @@ static int create_static_call_sections(s if (!sec) return -1; =20 - if (!elf_create_reloc_section(file->elf, sec, SHT_RELA)) - return -1; - idx =3D 0; list_for_each_entry(insn, &file->static_call_list, static_call_node) { =20 --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -498,11 +498,18 @@ err: return -1; } =20 +static struct section *elf_create_reloc_section(struct elf *elf, + struct section *base, + int reltype); + int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, unsigned int type, struct symbol *sym, int addend) { struct reloc *reloc; =20 + if (!sec->reloc && !elf_create_reloc_section(elf, sec, SHT_RELA)) + return -1; + reloc =3D malloc(sizeof(*reloc)); if (!reloc) { perror("malloc"); @@ -880,7 +887,7 @@ static struct section *elf_create_rela_r return sec; } =20 -struct section *elf_create_reloc_section(struct elf *elf, +static struct section *elf_create_reloc_section(struct elf *elf, struct section *base, int reltype) { --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -122,7 +122,6 @@ static inline u32 reloc_hash(struct relo =20 struct elf *elf_open_read(const char *name, int flags); struct section *elf_create_section(struct elf *elf, const char *name, unsi= gned int sh_flags, size_t entsize, int nr); -struct section *elf_create_reloc_section(struct elf *elf, struct section *= base, int reltype); =20 int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, unsigned int type, struct symbol *sym, int addend); --- a/tools/objtool/orc_gen.c +++ b/tools/objtool/orc_gen.c @@ -222,8 +222,6 @@ int orc_create(struct objtool_file *file sec =3D elf_create_section(file->elf, ".orc_unwind_ip", 0, sizeof(int), n= r); if (!sec) return -1; - if (!elf_create_reloc_section(file->elf, sec, SHT_RELA)) - return -1; =20 /* Write ORC entries to sections: */ list_for_each_entry(entry, &orc_list, list) { From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A203CCA483 for ; Sat, 23 Jul 2022 09:59:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237751AbiGWJ7y (ORCPT ); Sat, 23 Jul 2022 05:59:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237711AbiGWJ7K (ORCPT ); Sat, 23 Jul 2022 05:59:10 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55460474D8; Sat, 23 Jul 2022 02:57:52 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E3FC4B82C1B; Sat, 23 Jul 2022 09:57:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44E7BC341C0; Sat, 23 Jul 2022 09:57:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570269; bh=LpGgRFhrdCtmBrJ1+C5WSb3c5uJQUargm8UG6qofKak=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rNd2ovEjuqFfKSei/9WpL/YprXfvZx+61oWyq5dzeNKpPKTQYBFuSEnLJM8+jMtp7 761uvtKN5itxTvol2BkAj4D5FMSCOPcSCpwiJ0lj028Jf2Ok88EmVC4KpytOrZtvQX aiC77QpD9KkfJpavH+MdJBS7iL/cwEb8oFy13p1E= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 028/148] objtool: Extract elf_strtab_concat() Date: Sat, 23 Jul 2022 11:54:00 +0200 Message-Id: <20220723095232.278686983@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 417a4dc91e559f92404c2544f785b02ce75784c3 upstream. Create a common helper to append strings to a strtab. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151259.941474004@infradead.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/elf.c | 60 ++++++++++++++++++++++++++++++++---------------= ----- 1 file changed, 38 insertions(+), 22 deletions(-) --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -724,13 +724,48 @@ err: return NULL; } =20 +static int elf_add_string(struct elf *elf, struct section *strtab, char *s= tr) +{ + Elf_Data *data; + Elf_Scn *s; + int len; + + if (!strtab) + strtab =3D find_section_by_name(elf, ".strtab"); + if (!strtab) { + WARN("can't find .strtab section"); + return -1; + } + + s =3D elf_getscn(elf->elf, strtab->idx); + if (!s) { + WARN_ELF("elf_getscn"); + return -1; + } + + data =3D elf_newdata(s); + if (!data) { + WARN_ELF("elf_newdata"); + return -1; + } + + data->d_buf =3D str; + data->d_size =3D strlen(str) + 1; + data->d_align =3D 1; + + len =3D strtab->len; + strtab->len +=3D data->d_size; + strtab->changed =3D true; + + return len; +} + struct section *elf_create_section(struct elf *elf, const char *name, unsigned int sh_flags, size_t entsize, int nr) { struct section *sec, *shstrtab; size_t size =3D entsize * nr; Elf_Scn *s; - Elf_Data *data; =20 sec =3D malloc(sizeof(*sec)); if (!sec) { @@ -787,7 +822,6 @@ struct section *elf_create_section(struc sec->sh.sh_addralign =3D 1; sec->sh.sh_flags =3D SHF_ALLOC | sh_flags; =20 - /* Add section name to .shstrtab (or .strtab for Clang) */ shstrtab =3D find_section_by_name(elf, ".shstrtab"); if (!shstrtab) @@ -796,27 +830,9 @@ struct section *elf_create_section(struc WARN("can't find .shstrtab or .strtab section"); return NULL; } - - s =3D elf_getscn(elf->elf, shstrtab->idx); - if (!s) { - WARN_ELF("elf_getscn"); - return NULL; - } - - data =3D elf_newdata(s); - if (!data) { - WARN_ELF("elf_newdata"); + sec->sh.sh_name =3D elf_add_string(elf, shstrtab, sec->name); + if (sec->sh.sh_name =3D=3D -1) return NULL; - } - - data->d_buf =3D sec->name; - data->d_size =3D strlen(name) + 1; - data->d_align =3D 1; - - sec->sh.sh_name =3D shstrtab->len; - - shstrtab->len +=3D strlen(name) + 1; - shstrtab->changed =3D true; =20 list_add_tail(&sec->list, &elf->sections); elf_hash_add(elf->section_hash, &sec->hash, sec->idx); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77CA9C43334 for ; Sat, 23 Jul 2022 09:59:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237818AbiGWJ75 (ORCPT ); Sat, 23 Jul 2022 05:59:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237745AbiGWJ7O (ORCPT ); Sat, 23 Jul 2022 05:59:14 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C70B6E2E6; Sat, 23 Jul 2022 02:57:55 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EDE05B82C21; Sat, 23 Jul 2022 09:57:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1F87AC341C0; Sat, 23 Jul 2022 09:57:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570272; bh=6MHWoP7j5DCxR+KQoeA5JjHbHKkbd3JWJTvdKzygbME=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TLt91ALEGaAMfRQQ0J7SS+Uzruu/1wUdrPNDeVDpjiEdJ9G4bhltdXNntr6L6tP4Y O1IX5DPpkUZqdIWUd9zgLBzh1grKKud7GzmQbO69Lr66qfnOwSXOLVFDXAqfsme6Aa +WHHERERJZGXGNrj0wHk1Qg4zz2ANeTLG+Qg46L0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 029/148] objtool: Extract elf_symbol_add() Date: Sat, 23 Jul 2022 11:54:01 +0200 Message-Id: <20220723095232.558086259@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 9a7827b7789c630c1efdb121daa42c6e77dce97f upstream. Create a common helper to add symbols. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151300.003468981@infradead.org [bwh: Backported to 5.10: rb_add() parameter order is different] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/elf.c | 56 ++++++++++++++++++++++++++++-------------------= ----- 1 file changed, 31 insertions(+), 25 deletions(-) --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -341,12 +341,39 @@ static int read_sections(struct elf *elf return 0; } =20 +static void elf_add_symbol(struct elf *elf, struct symbol *sym) +{ + struct list_head *entry; + struct rb_node *pnode; + + sym->type =3D GELF_ST_TYPE(sym->sym.st_info); + sym->bind =3D GELF_ST_BIND(sym->sym.st_info); + + sym->offset =3D sym->sym.st_value; + sym->len =3D sym->sym.st_size; + + rb_add(&sym->sec->symbol_tree, &sym->node, symbol_to_offset); + pnode =3D rb_prev(&sym->node); + if (pnode) + entry =3D &rb_entry(pnode, struct symbol, node)->list; + else + entry =3D &sym->sec->symbol_list; + list_add(&sym->list, entry); + elf_hash_add(elf->symbol_hash, &sym->hash, sym->idx); + elf_hash_add(elf->symbol_name_hash, &sym->name_hash, str_hash(sym->name)); + + /* + * Don't store empty STT_NOTYPE symbols in the rbtree. They + * can exist within a function, confusing the sorting. + */ + if (!sym->len) + rb_erase(&sym->node, &sym->sec->symbol_tree); +} + static int read_symbols(struct elf *elf) { struct section *symtab, *symtab_shndx, *sec; struct symbol *sym, *pfunc; - struct list_head *entry; - struct rb_node *pnode; int symbols_nr, i; char *coldstr; Elf_Data *shndx_data =3D NULL; @@ -391,9 +418,6 @@ static int read_symbols(struct elf *elf) goto err; } =20 - sym->type =3D GELF_ST_TYPE(sym->sym.st_info); - sym->bind =3D GELF_ST_BIND(sym->sym.st_info); - if ((sym->sym.st_shndx > SHN_UNDEF && sym->sym.st_shndx < SHN_LORESERVE) || (shndx_data && sym->sym.st_shndx =3D=3D SHN_XINDEX)) { @@ -406,32 +430,14 @@ static int read_symbols(struct elf *elf) sym->name); goto err; } - if (sym->type =3D=3D STT_SECTION) { + if (GELF_ST_TYPE(sym->sym.st_info) =3D=3D STT_SECTION) { sym->name =3D sym->sec->name; sym->sec->sym =3D sym; } } else sym->sec =3D find_section_by_index(elf, 0); =20 - sym->offset =3D sym->sym.st_value; - sym->len =3D sym->sym.st_size; - - rb_add(&sym->sec->symbol_tree, &sym->node, symbol_to_offset); - pnode =3D rb_prev(&sym->node); - if (pnode) - entry =3D &rb_entry(pnode, struct symbol, node)->list; - else - entry =3D &sym->sec->symbol_list; - list_add(&sym->list, entry); - elf_hash_add(elf->symbol_hash, &sym->hash, sym->idx); - elf_hash_add(elf->symbol_name_hash, &sym->name_hash, str_hash(sym->name)= ); - - /* - * Don't store empty STT_NOTYPE symbols in the rbtree. They - * can exist within a function, confusing the sorting. - */ - if (!sym->len) - rb_erase(&sym->node, &sym->sec->symbol_tree); + elf_add_symbol(elf, sym); } =20 if (stats) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7797AC43334 for ; Sat, 23 Jul 2022 10:00:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237841AbiGWKAJ (ORCPT ); Sat, 23 Jul 2022 06:00:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237596AbiGWJ7V (ORCPT ); Sat, 23 Jul 2022 05:59:21 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18F8F48E85; Sat, 23 Jul 2022 02:57:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B7298B82C20; Sat, 23 Jul 2022 09:57:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2273FC341C0; Sat, 23 Jul 2022 09:57:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570275; bh=VvU/1RP84JrPAl0/5+ZpTqj7LWQ1cv8/7kvVC/IdFmQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L7X+Fs+0BYLdwudha8JdcsD78HC1ZQOWGFLFHgryQL0WSGhMzP23LIf2JEFRVOZAG 7/jzbSFJU4qBFxC1JO7AhNuh9TAmWgFjU0f3LIZXNNjdiuXC/P8chMr954OUk9PSWJ SBm1u1iY4Su+oFzE5FCyirW/8LLMVtXpoq3egM0k= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 030/148] objtool: Add elf_create_undef_symbol() Date: Sat, 23 Jul 2022 11:54:02 +0200 Message-Id: <20220723095232.840946456@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 2f2f7e47f0525cbaad5dd9675fd9d8aa8da12046 upstream. Allow objtool to create undefined symbols; this allows creating relocations to symbols not currently in the symbol table. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151300.064743095@infradead.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/elf.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++= +++++ tools/objtool/elf.h | 1=20 2 files changed, 61 insertions(+) --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -766,6 +766,66 @@ static int elf_add_string(struct elf *el return len; } =20 +struct symbol *elf_create_undef_symbol(struct elf *elf, const char *name) +{ + struct section *symtab; + struct symbol *sym; + Elf_Data *data; + Elf_Scn *s; + + sym =3D malloc(sizeof(*sym)); + if (!sym) { + perror("malloc"); + return NULL; + } + memset(sym, 0, sizeof(*sym)); + + sym->name =3D strdup(name); + + sym->sym.st_name =3D elf_add_string(elf, NULL, sym->name); + if (sym->sym.st_name =3D=3D -1) + return NULL; + + sym->sym.st_info =3D GELF_ST_INFO(STB_GLOBAL, STT_NOTYPE); + // st_other 0 + // st_shndx 0 + // st_value 0 + // st_size 0 + + symtab =3D find_section_by_name(elf, ".symtab"); + if (!symtab) { + WARN("can't find .symtab"); + return NULL; + } + + s =3D elf_getscn(elf->elf, symtab->idx); + if (!s) { + WARN_ELF("elf_getscn"); + return NULL; + } + + data =3D elf_newdata(s); + if (!data) { + WARN_ELF("elf_newdata"); + return NULL; + } + + data->d_buf =3D &sym->sym; + data->d_size =3D sizeof(sym->sym); + data->d_align =3D 1; + + sym->idx =3D symtab->len / sizeof(sym->sym); + + symtab->len +=3D data->d_size; + symtab->changed =3D true; + + sym->sec =3D find_section_by_index(elf, 0); + + elf_add_symbol(elf, sym); + + return sym; +} + struct section *elf_create_section(struct elf *elf, const char *name, unsigned int sh_flags, size_t entsize, int nr) { --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -133,6 +133,7 @@ int elf_write_insn(struct elf *elf, stru unsigned long offset, unsigned int len, const char *insn); int elf_write_reloc(struct elf *elf, struct reloc *reloc); +struct symbol *elf_create_undef_symbol(struct elf *elf, const char *name); int elf_write(struct elf *elf); void elf_close(struct elf *elf); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 853C7C433EF for ; Sat, 23 Jul 2022 10:00:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237486AbiGWKAN (ORCPT ); Sat, 23 Jul 2022 06:00:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237543AbiGWJ7X (ORCPT ); Sat, 23 Jul 2022 05:59:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A20D46DBE; Sat, 23 Jul 2022 02:57:59 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DBBE5611D2; Sat, 23 Jul 2022 09:57:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EA70BC341C0; Sat, 23 Jul 2022 09:57:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570278; bh=e0PJ8wVXvSLi9INMnCS5SUkzghqk01lL5LYSFqqqh1w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0pgyvPmZ9fL62VolOlvfBStIPf1vCUnSFX6O2zNYuo8o0Q6Un8o36/IUBXhHyzsU1 lA8rm3HP8pSyEJ2w8Y1tsYq3r03mp0q43Y7VvrZ3RJtUg3ULUbk8ekDFP1Et1kch7w GBsOhSiShmaATOgwjs5HZoZcb/fhH3x/A26iCLLI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 031/148] objtool: Keep track of retpoline call sites Date: Sat, 23 Jul 2022 11:54:03 +0200 Message-Id: <20220723095233.088016764@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 43d5430ad74ef5156353af7aec352426ec7a8e57 upstream. Provide infrastructure for architectures to rewrite/augment compiler generated retpoline calls. Similar to what we do for static_call()s, keep track of the instructions that are retpoline calls. Use the same list_head, since a retpoline call cannot also be a static_call. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151300.130805730@infradead.org [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch.h | 2 ++ tools/objtool/check.c | 34 +++++++++++++++++++++++++++++----- tools/objtool/check.h | 2 +- tools/objtool/objtool.c | 1 + tools/objtool/objtool.h | 1 + 5 files changed, 34 insertions(+), 6 deletions(-) --- a/tools/objtool/arch.h +++ b/tools/objtool/arch.h @@ -88,4 +88,6 @@ int arch_decode_hint_reg(struct instruct =20 bool arch_is_retpoline(struct symbol *sym); =20 +int arch_rewrite_retpolines(struct objtool_file *file); + #endif /* _ARCH_H */ --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -451,7 +451,7 @@ static int create_static_call_sections(s return 0; =20 idx =3D 0; - list_for_each_entry(insn, &file->static_call_list, static_call_node) + list_for_each_entry(insn, &file->static_call_list, call_node) idx++; =20 sec =3D elf_create_section(file->elf, ".static_call_sites", SHF_WRITE, @@ -460,7 +460,7 @@ static int create_static_call_sections(s return -1; =20 idx =3D 0; - list_for_each_entry(insn, &file->static_call_list, static_call_node) { + list_for_each_entry(insn, &file->static_call_list, call_node) { =20 site =3D (struct static_call_site *)sec->data->d_buf + idx; memset(site, 0, sizeof(struct static_call_site)); @@ -786,13 +786,16 @@ static int add_jump_destinations(struct else insn->type =3D INSN_JUMP_DYNAMIC_CONDITIONAL; =20 + list_add_tail(&insn->call_node, + &file->retpoline_call_list); + insn->retpoline_safe =3D true; continue; } else if (insn->func) { /* internal or external sibling call (with reloc) */ insn->call_dest =3D reloc->sym; if (insn->call_dest->static_call_tramp) { - list_add_tail(&insn->static_call_node, + list_add_tail(&insn->call_node, &file->static_call_list); } continue; @@ -854,7 +857,7 @@ static int add_jump_destinations(struct /* internal sibling call (without reloc) */ insn->call_dest =3D insn->jump_dest->func; if (insn->call_dest->static_call_tramp) { - list_add_tail(&insn->static_call_node, + list_add_tail(&insn->call_node, &file->static_call_list); } } @@ -938,6 +941,9 @@ static int add_call_destinations(struct insn->type =3D INSN_CALL_DYNAMIC; insn->retpoline_safe =3D true; =20 + list_add_tail(&insn->call_node, + &file->retpoline_call_list); + remove_insn_ops(insn); continue; =20 @@ -945,7 +951,7 @@ static int add_call_destinations(struct insn->call_dest =3D reloc->sym; =20 if (insn->call_dest && insn->call_dest->static_call_tramp) { - list_add_tail(&insn->static_call_node, + list_add_tail(&insn->call_node, &file->static_call_list); } =20 @@ -1655,6 +1661,11 @@ static void mark_rodata(struct objtool_f file->rodata =3D found; } =20 +__weak int arch_rewrite_retpolines(struct objtool_file *file) +{ + return 0; +} + static int decode_sections(struct objtool_file *file) { int ret; @@ -1683,6 +1694,10 @@ static int decode_sections(struct objtoo if (ret) return ret; =20 + /* + * Must be before add_special_section_alts() as that depends on + * jump_dest being set. + */ ret =3D add_jump_destinations(file); if (ret) return ret; @@ -1719,6 +1734,15 @@ static int decode_sections(struct objtoo if (ret) return ret; =20 + /* + * Must be after add_special_section_alts(), since this will emit + * alternatives. Must be after add_{jump,call}_destination(), since + * those create the call insn lists. + */ + ret =3D arch_rewrite_retpolines(file); + if (ret) + return ret; + return 0; } =20 --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -39,7 +39,7 @@ struct alt_group { struct instruction { struct list_head list; struct hlist_node hash; - struct list_head static_call_node; + struct list_head call_node; struct section *sec; unsigned long offset; unsigned int len; --- a/tools/objtool/objtool.c +++ b/tools/objtool/objtool.c @@ -61,6 +61,7 @@ struct objtool_file *objtool_open_read(c =20 INIT_LIST_HEAD(&file.insn_list); hash_init(file.insn_hash); + INIT_LIST_HEAD(&file.retpoline_call_list); INIT_LIST_HEAD(&file.static_call_list); file.c_file =3D !vmlinux && find_section_by_name(file.elf, ".comment"); file.ignore_unreachables =3D no_unreachable; --- a/tools/objtool/objtool.h +++ b/tools/objtool/objtool.h @@ -18,6 +18,7 @@ struct objtool_file { struct elf *elf; struct list_head insn_list; DECLARE_HASHTABLE(insn_hash, 20); + struct list_head retpoline_call_list; struct list_head static_call_list; bool ignore_unreachables, c_file, hints, rodata; }; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E1C6CCA483 for ; Sat, 23 Jul 2022 10:00:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237873AbiGWKAP (ORCPT ); Sat, 23 Jul 2022 06:00:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237484AbiGWJ7g (ORCPT ); Sat, 23 Jul 2022 05:59:36 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4586459AF; Sat, 23 Jul 2022 02:58:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 86F4BB82C1B; Sat, 23 Jul 2022 09:58:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C82CFC341C0; Sat, 23 Jul 2022 09:58:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570281; bh=6lA7kujKxFk6fxYJsW1W21z8ns4CSf2ZVJFi6YxNeus=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OSIhR5XnlzNgZue0Qg0haEPBvBCZEfS+R6ciY6HUGLaglm7iRYYVXj4sbSlklkSp2 hUZWGcWgD+qs0wzxVOzkxLUiEATqQrmQ5a5p9wPoYuwFyf6+Z3oYGkOtVsq99ZlG6E /xVMx7vfWn+YB8q+BEbco4WnCTZ8qhUGXbK7FgCI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 032/148] objtool: Cache instruction relocs Date: Sat, 23 Jul 2022 11:54:04 +0200 Message-Id: <20220723095233.354758252@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 7bd2a600f3e9d27286bbf23c83d599e9cc7cf245 upstream. Track the reloc of instructions in the new instruction->reloc field to avoid having to look them up again later. ( Technically x86 instructions can have two relocations, but not jumps and calls, for which we're using this. ) Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151300.195441549@infradead.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 28 ++++++++++++++++++++++------ tools/objtool/check.h | 1 + 2 files changed, 23 insertions(+), 6 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -754,6 +754,25 @@ __weak bool arch_is_retpoline(struct sym return false; } =20 +#define NEGATIVE_RELOC ((void *)-1L) + +static struct reloc *insn_reloc(struct objtool_file *file, struct instruct= ion *insn) +{ + if (insn->reloc =3D=3D NEGATIVE_RELOC) + return NULL; + + if (!insn->reloc) { + insn->reloc =3D find_reloc_by_dest_range(file->elf, insn->sec, + insn->offset, insn->len); + if (!insn->reloc) { + insn->reloc =3D NEGATIVE_RELOC; + return NULL; + } + } + + return insn->reloc; +} + /* * Find the destination instructions for all jumps. */ @@ -768,8 +787,7 @@ static int add_jump_destinations(struct if (!is_static_jump(insn)) continue; =20 - reloc =3D find_reloc_by_dest_range(file->elf, insn->sec, - insn->offset, insn->len); + reloc =3D insn_reloc(file, insn); if (!reloc) { dest_sec =3D insn->sec; dest_off =3D arch_jump_destination(insn); @@ -901,8 +919,7 @@ static int add_call_destinations(struct if (insn->type !=3D INSN_CALL) continue; =20 - reloc =3D find_reloc_by_dest_range(file->elf, insn->sec, - insn->offset, insn->len); + reloc =3D insn_reloc(file, insn); if (!reloc) { dest_off =3D arch_jump_destination(insn); insn->call_dest =3D find_call_destination(insn->sec, dest_off); @@ -1085,8 +1102,7 @@ static int handle_group_alt(struct objto * alternatives code can adjust the relative offsets * accordingly. */ - alt_reloc =3D find_reloc_by_dest_range(file->elf, insn->sec, - insn->offset, insn->len); + alt_reloc =3D insn_reloc(file, insn); if (alt_reloc && !arch_support_alt_relocation(special_alt, insn, alt_reloc)) { =20 --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -55,6 +55,7 @@ struct instruction { struct instruction *jump_dest; struct instruction *first_jump_src; struct reloc *jump_table; + struct reloc *reloc; struct list_head alts; struct symbol *func; struct list_head stack_ops; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55A80C433EF for ; Sat, 23 Jul 2022 09:59:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237478AbiGWJ7Y (ORCPT ); Sat, 23 Jul 2022 05:59:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237610AbiGWJ6p (ORCPT ); Sat, 23 Jul 2022 05:58:45 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A6F369F0D; Sat, 23 Jul 2022 02:57:38 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C5243B82C21; Sat, 23 Jul 2022 09:57:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 27074C341C0; Sat, 23 Jul 2022 09:57:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570255; bh=wGMNTjp6VPYf7orV8ztNuBdmDKJ1DhLEaFQidTOFZH0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J9CUGOijz/iAcsS2qR9fd5VB8yYw5N0xDRFJxoUAdcOEHTN9r0mEcuU0fIQbMKiWR AYbJ6wLHZgxvXhH6tGdAtznNUAmBl50k67+Jq43mrcRMzvu3xAu9T9qG8p8/bBL15+ ItokoVG8HMAbsOD0hCsm/0ovkN2J9HGhC5U7u85o= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 033/148] objtool: Skip magical retpoline .altinstr_replacement Date: Sat, 23 Jul 2022 11:54:05 +0200 Message-Id: <20220723095233.591415325@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 50e7b4a1a1b264fc7df0698f2defb93cadf19a7b upstream. When the .altinstr_replacement is a retpoline, skip the alternative. We already special case retpolines anyway. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151300.259429287@infradead.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/special.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -104,6 +104,14 @@ static int get_alt_entry(struct elf *elf return -1; } =20 + /* + * Skip retpoline .altinstr_replacement... we already rewrite the + * instructions for retpolines anyway, see arch_is_retpoline() + * usage in add_{call,jump}_destinations(). + */ + if (arch_is_retpoline(new_reloc->sym)) + return 1; + alt->new_sec =3D new_reloc->sym->sec; alt->new_off =3D (unsigned int)new_reloc->addend; =20 @@ -152,7 +160,9 @@ int special_get_alts(struct elf *elf, st memset(alt, 0, sizeof(*alt)); =20 ret =3D get_alt_entry(elf, entry, sec, idx, alt); - if (ret) + if (ret > 0) + continue; + if (ret < 0) return ret; =20 list_add_tail(&alt->list, alts); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B8C2CCA483 for ; Sat, 23 Jul 2022 09:59:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237765AbiGWJ73 (ORCPT ); Sat, 23 Jul 2022 05:59:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237541AbiGWJ6r (ORCPT ); Sat, 23 Jul 2022 05:58:47 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D9B16A9CF; Sat, 23 Jul 2022 02:57:39 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E8D61611CD; Sat, 23 Jul 2022 09:57:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F063FC341C7; Sat, 23 Jul 2022 09:57:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570258; bh=E7rbmcrV7XwUzajBPQnJcFe7GQ/LEzE1EoPxgXcQPSg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=grD3uyMCvns6A5FCiW/z3Zqkaqp+IF2FkF+0IM6ZFwvrbVwUGMVatDG4zELR6OC9K A99bFgNMT+UP+gtAAGJkutbQB9YWaCcT1/Kp7rk54VTbbS1ZmeuRPchccsN/8rCTqv S4HXxN5m0UtDhzO43QZ2Y3hABe600XonmMaDuQF0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ingo Molnar , Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 034/148] objtool/x86: Rewrite retpoline thunk calls Date: Sat, 23 Jul 2022 11:54:06 +0200 Message-Id: <20220723095233.874778711@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 9bc0bb50727c8ac69fbb33fb937431cf3518ff37 upstream. When the compiler emits: "CALL __x86_indirect_thunk_\reg" for an indirect call, have objtool rewrite it to: ALTERNATIVE "call __x86_indirect_thunk_\reg", "call *%reg", ALT_NOT(X86_FEATURE_RETPOLINE) Additionally, in order to not emit endless identical .altinst_replacement chunks, use a global symbol for them, see __x86_indirect_alt_*. This also avoids objtool from having to do code generation. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Ingo Molnar Reviewed-by: Miroslav Benes Link: https://lkml.kernel.org/r/20210326151300.320177914@infradead.org [bwh: Backported to 5.10: include "arch_elf.h" instead of "arch/elf.h"] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/asm-prototypes.h | 12 ++- arch/x86/lib/retpoline.S | 41 +++++++++++ tools/objtool/arch/x86/decode.c | 117 +++++++++++++++++++++++++++++= +++++ 3 files changed, 167 insertions(+), 3 deletions(-) --- a/arch/x86/include/asm/asm-prototypes.h +++ b/arch/x86/include/asm/asm-prototypes.h @@ -19,11 +19,19 @@ extern void cmpxchg8b_emu(void); =20 #ifdef CONFIG_RETPOLINE =20 -#define DECL_INDIRECT_THUNK(reg) \ +#undef GEN +#define GEN(reg) \ extern asmlinkage void __x86_indirect_thunk_ ## reg (void); +#include + +#undef GEN +#define GEN(reg) \ + extern asmlinkage void __x86_indirect_alt_call_ ## reg (void); +#include =20 #undef GEN -#define GEN(reg) DECL_INDIRECT_THUNK(reg) +#define GEN(reg) \ + extern asmlinkage void __x86_indirect_alt_jmp_ ## reg (void); #include =20 #endif /* CONFIG_RETPOLINE */ --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -10,6 +10,8 @@ #include #include =20 + .section .text.__x86.indirect_thunk + .macro RETPOLINE reg ANNOTATE_INTRA_FUNCTION_CALL call .Ldo_rop_\@ @@ -25,9 +27,9 @@ .endm =20 .macro THUNK reg - .section .text.__x86.indirect_thunk =20 .align 32 + SYM_FUNC_START(__x86_indirect_thunk_\reg) =20 ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ @@ -39,6 +41,32 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg) .endm =20 /* + * This generates .altinstr_replacement symbols for use by objtool. They, + * however, must not actually live in .altinstr_replacement since that wil= l be + * discarded after init, but module alternatives will also reference these + * symbols. + * + * Their names matches the "__x86_indirect_" prefix to mark them as retpol= ines. + */ +.macro ALT_THUNK reg + + .align 1 + +SYM_FUNC_START_NOALIGN(__x86_indirect_alt_call_\reg) + ANNOTATE_RETPOLINE_SAFE +1: call *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_call_\reg) + +SYM_FUNC_START_NOALIGN(__x86_indirect_alt_jmp_\reg) + ANNOTATE_RETPOLINE_SAFE +1: jmp *%\reg +2: .skip 5-(2b-1b), 0x90 +SYM_FUNC_END(__x86_indirect_alt_jmp_\reg) + +.endm + +/* * Despite being an assembler file we can't just use .irp here * because __KSYM_DEPS__ only uses the C preprocessor and would * only see one instance of "__x86_indirect_thunk_\reg" rather @@ -61,3 +89,14 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg) #define GEN(reg) EXPORT_THUNK(reg) #include =20 +#undef GEN +#define GEN(reg) ALT_THUNK reg +#include + +#undef GEN +#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_call_ ## reg) +#include + +#undef GEN +#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_jmp_ ## reg) +#include --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -16,6 +16,7 @@ #include "../../arch.h" #include "../../warn.h" #include +#include "arch_elf.h" =20 static unsigned char op_to_cfi_reg[][2] =3D { {CFI_AX, CFI_R8}, @@ -585,6 +586,122 @@ const char *arch_nop_insn(int len) return nops[len-1]; } =20 +/* asm/alternative.h ? */ + +#define ALTINSTR_FLAG_INV (1 << 15) +#define ALT_NOT(feat) ((feat) | ALTINSTR_FLAG_INV) + +struct alt_instr { + s32 instr_offset; /* original instruction */ + s32 repl_offset; /* offset to replacement instruction */ + u16 cpuid; /* cpuid bit set for replacement */ + u8 instrlen; /* length of original instruction */ + u8 replacementlen; /* length of new instruction */ +} __packed; + +static int elf_add_alternative(struct elf *elf, + struct instruction *orig, struct symbol *sym, + int cpuid, u8 orig_len, u8 repl_len) +{ + const int size =3D sizeof(struct alt_instr); + struct alt_instr *alt; + struct section *sec; + Elf_Scn *s; + + sec =3D find_section_by_name(elf, ".altinstructions"); + if (!sec) { + sec =3D elf_create_section(elf, ".altinstructions", + SHF_WRITE, size, 0); + + if (!sec) { + WARN_ELF("elf_create_section"); + return -1; + } + } + + s =3D elf_getscn(elf->elf, sec->idx); + if (!s) { + WARN_ELF("elf_getscn"); + return -1; + } + + sec->data =3D elf_newdata(s); + if (!sec->data) { + WARN_ELF("elf_newdata"); + return -1; + } + + sec->data->d_size =3D size; + sec->data->d_align =3D 1; + + alt =3D sec->data->d_buf =3D malloc(size); + if (!sec->data->d_buf) { + perror("malloc"); + return -1; + } + memset(sec->data->d_buf, 0, size); + + if (elf_add_reloc_to_insn(elf, sec, sec->sh.sh_size, + R_X86_64_PC32, orig->sec, orig->offset)) { + WARN("elf_create_reloc: alt_instr::instr_offset"); + return -1; + } + + if (elf_add_reloc(elf, sec, sec->sh.sh_size + 4, + R_X86_64_PC32, sym, 0)) { + WARN("elf_create_reloc: alt_instr::repl_offset"); + return -1; + } + + alt->cpuid =3D cpuid; + alt->instrlen =3D orig_len; + alt->replacementlen =3D repl_len; + + sec->sh.sh_size +=3D size; + sec->changed =3D true; + + return 0; +} + +#define X86_FEATURE_RETPOLINE ( 7*32+12) + +int arch_rewrite_retpolines(struct objtool_file *file) +{ + struct instruction *insn; + struct reloc *reloc; + struct symbol *sym; + char name[32] =3D ""; + + list_for_each_entry(insn, &file->retpoline_call_list, call_node) { + + if (!strcmp(insn->sec->name, ".text.__x86.indirect_thunk")) + continue; + + reloc =3D insn->reloc; + + sprintf(name, "__x86_indirect_alt_%s_%s", + insn->type =3D=3D INSN_JUMP_DYNAMIC ? "jmp" : "call", + reloc->sym->name + 21); + + sym =3D find_symbol_by_name(file->elf, name); + if (!sym) { + sym =3D elf_create_undef_symbol(file->elf, name); + if (!sym) { + WARN("elf_create_undef_symbol"); + return -1; + } + } + + if (elf_add_alternative(file->elf, insn, sym, + ALT_NOT(X86_FEATURE_RETPOLINE), 5, 5)) { + WARN("elf_add_alternative"); + return -1; + } + } + + return 0; +} + int arch_decode_hint_reg(struct instruction *insn, u8 sp_reg) { struct cfi_reg *cfa =3D &insn->cfi.cfa; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86503C433EF for ; Sat, 23 Jul 2022 10:02:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237529AbiGWKCV (ORCPT ); Sat, 23 Jul 2022 06:02:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238016AbiGWKBy (ORCPT ); Sat, 23 Jul 2022 06:01:54 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 781E661101; Sat, 23 Jul 2022 02:58:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id B95D2CE0DC2; Sat, 23 Jul 2022 09:58:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9FD22C341C0; Sat, 23 Jul 2022 09:58:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570321; bh=yEjzjL/duoq1Md+YLkmgI3lo63MssRrpZ/IVKlR4NWg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R0qp9kZV7RsdyzWlN2drKT7LT3LJ3KpbLIcJyPSvoo5bdKydsWGqddBavLKSBG8A5 6oClmzZqISaADhaUfoKeMkFC5KyuvqXqlR6rJCfG6ttQpuMZ4ETL/1m9yOC2pkEct3 KthF0dJgAVcLMYOnO2NBUfgCILZWwJRmeK9OrCag= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Ard Biesheuvel , Sami Tolvanen , "Peter Zijlstra (Intel)" , Herbert Xu , Ben Hutchings Subject: [PATCH 5.10 035/148] objtool: Support asm jump tables Date: Sat, 23 Jul 2022 11:54:07 +0200 Message-Id: <20220723095234.140361076@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 99033461e685b48549ec77608b4bda75ddf772ce upstream. Objtool detection of asm jump tables would normally just work, except for the fact that asm retpolines use alternatives. Objtool thinks the alternative code path (a jump to the retpoline) is a sibling call. Don't treat alternative indirect branches as sibling calls when the original instruction has a jump table. Signed-off-by: Josh Poimboeuf Tested-by: Ard Biesheuvel Acked-by: Ard Biesheuvel Tested-by: Sami Tolvanen Acked-by: Peter Zijlstra (Intel) Acked-by: Herbert Xu Link: https://lore.kernel.org/r/460cf4dc675d64e1124146562cabd2c05aa322e8.16= 14182415.git.jpoimboe@redhat.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -107,6 +107,18 @@ static struct instruction *prev_insn_sam for (insn =3D next_insn_same_sec(file, insn); insn; \ insn =3D next_insn_same_sec(file, insn)) =20 +static bool is_jump_table_jump(struct instruction *insn) +{ + struct alt_group *alt_group =3D insn->alt_group; + + if (insn->jump_table) + return true; + + /* Retpoline alternative for a jump table? */ + return alt_group && alt_group->orig_group && + alt_group->orig_group->first_insn->jump_table; +} + static bool is_sibling_call(struct instruction *insn) { /* @@ -119,7 +131,7 @@ static bool is_sibling_call(struct instr =20 /* An indirect jump is either a sibling call or a jump to a table. */ if (insn->type =3D=3D INSN_JUMP_DYNAMIC) - return list_empty(&insn->alts); + return !is_jump_table_jump(insn); =20 /* add_jump_destinations() sets insn->call_dest for sibling calls. */ return (is_static_jump(insn) && insn->call_dest); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05A12C43334 for ; Sat, 23 Jul 2022 10:00:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237901AbiGWKA1 (ORCPT ); Sat, 23 Jul 2022 06:00:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237787AbiGWJ7i (ORCPT ); Sat, 23 Jul 2022 05:59:38 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E339D7170D; Sat, 23 Jul 2022 02:58:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 63E8E611CD; Sat, 23 Jul 2022 09:58:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 713A6C341C0; Sat, 23 Jul 2022 09:58:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570286; bh=J1SCdMjT+p5aK9/SabHy6rOQb3kdXulj/Hyw//Dih+4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TpVftAacqwlJ0VHEtAJtWHz2KwHFTaCVRpOtkimU9qf+94szQABfZ3zM49/2WIt0l lTrrS7Bw8ePDDHvDPH/xFIiYLm4FceuBOcFjzy97pkHa1SyLWKskyZ1rRUUNghqzSO vfF+0GmRIP7cTj/l5WzotKZEF5y+KP+mfZMWcQQk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Richard Narron , Borislav Petkov , "Peter Zijlstra (Intel)" , Ben Hutchings Subject: [PATCH 5.10 036/148] x86/alternative: Optimize single-byte NOPs at an arbitrary position Date: Sat, 23 Jul 2022 11:54:08 +0200 Message-Id: <20220723095234.428043991@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit 2b31e8ed96b260ce2c22bd62ecbb9458399e3b62 upstream. Up until now the assumption was that an alternative patching site would have some instructions at the beginning and trailing single-byte NOPs (0x90) padding. Therefore, the patching machinery would go and optimize those single-byte NOPs into longer ones. However, this assumption is broken on 32-bit when code like hv_do_hypercall() in hyperv_init() would use the ratpoline speculation killer CALL_NOSPEC. The 32-bit version of that macro would align certain insns to 16 bytes, leading to the compiler issuing a one or more single-byte NOPs, depending on the holes it needs to fill for alignment. That would lead to the warning in optimize_nops() to fire: ------------[ cut here ]------------ Not a NOP at 0xc27fb598 WARNING: CPU: 0 PID: 0 at arch/x86/kernel/alternative.c:211 optimize_nop= s.isra.13 due to that function verifying whether all of the following bytes really are single-byte NOPs. Therefore, carve out the NOP padding into a separate function and call it for each NOP range beginning with a single-byte NOP. Fixes: 23c1ad538f4f ("x86/alternatives: Optimize optimize_nops()") Reported-by: Richard Narron Signed-off-by: Borislav Petkov Acked-by: Peter Zijlstra (Intel) Link: https://bugzilla.kernel.org/show_bug.cgi?id=3D213301 Link: https://lkml.kernel.org/r/20210601212125.17145-1-bp@alien8.de Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/alternative.c | 64 ++++++++++++++++++++++++++++++-------= ----- 1 file changed, 46 insertions(+), 18 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -338,41 +338,69 @@ done: } =20 /* + * optimize_nops_range() - Optimize a sequence of single byte NOPs (0x90) + * + * @instr: instruction byte stream + * @instrlen: length of the above + * @off: offset within @instr where the first NOP has been detected + * + * Return: number of NOPs found (and replaced). + */ +static __always_inline int optimize_nops_range(u8 *instr, u8 instrlen, int= off) +{ + unsigned long flags; + int i =3D off, nnops; + + while (i < instrlen) { + if (instr[i] !=3D 0x90) + break; + + i++; + } + + nnops =3D i - off; + + if (nnops <=3D 1) + return nnops; + + local_irq_save(flags); + add_nops(instr + off, nnops); + local_irq_restore(flags); + + DUMP_BYTES(instr, instrlen, "%px: [%d:%d) optimized NOPs: ", instr, off, = i); + + return nnops; +} + +/* * "noinline" to cause control flow change and thus invalidate I$ and * cause refetch after modification. */ static void __init_or_module noinline optimize_nops(struct alt_instr *a, u= 8 *instr) { - unsigned long flags; struct insn insn; - int nop, i =3D 0; + int i =3D 0; =20 /* - * Jump over the non-NOP insns, the remaining bytes must be single-byte - * NOPs, optimize them. + * Jump over the non-NOP insns and optimize single-byte NOPs into bigger + * ones. */ for (;;) { if (insn_decode_kernel(&insn, &instr[i])) return; =20 + /* + * See if this and any potentially following NOPs can be + * optimized. + */ if (insn.length =3D=3D 1 && insn.opcode.bytes[0] =3D=3D 0x90) - break; + i +=3D optimize_nops_range(instr, a->instrlen, i); + else + i +=3D insn.length; =20 - if ((i +=3D insn.length) >=3D a->instrlen) + if (i >=3D a->instrlen) return; } - - for (nop =3D i; i < a->instrlen; i++) { - if (WARN_ONCE(instr[i] !=3D 0x90, "Not a NOP at 0x%px\n", &instr[i])) - return; - } - - local_irq_save(flags); - add_nops(instr + nop, i - nop); - local_irq_restore(flags); - - DUMP_BYTES(instr, a->instrlen, "%px: [%d:%d) optimized NOPs: ", - instr, nop, a->instrlen); } =20 /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BE69C43334 for ; Sat, 23 Jul 2022 10:01:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237412AbiGWKBB (ORCPT ); Sat, 23 Jul 2022 06:01:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237838AbiGWKAI (ORCPT ); Sat, 23 Jul 2022 06:00:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F76E76E85; Sat, 23 Jul 2022 02:58:19 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BA52E611CD; Sat, 23 Jul 2022 09:58:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9E20C341CA; Sat, 23 Jul 2022 09:58:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570298; bh=OPAVYdq/Nb9TUWPRrzva7ewq9dIjGGbmoWDHPMaNNOQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U2DfzOs+JrKeusYswmHCCga+9HJDJ41zIxYDcNx1mYN9OFVDrk1vfp2X51aZIgZu/ KpbIIjKFaAoVls0tJomRDSoAZQ2vCKZRbgzCsnTaCtiIAaMKehZpcGAf1Gk4QcsZRu puWU3IvU7eFxcsVHZhLMwiGPrhc6eBAtq/+7lqP0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Nick Desaulniers , Fangrui Song , "Peter Zijlstra (Intel)" , Ben Hutchings Subject: [PATCH 5.10 037/148] objtool: Fix .symtab_shndx handling for elf_create_undef_symbol() Date: Sat, 23 Jul 2022 11:54:09 +0200 Message-Id: <20220723095234.704707161@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 584fd3b31889852d0d6f3dd1e3d8e9619b660d2c upstream. When an ELF object uses extended symbol section indexes (IOW it has a .symtab_shndx section), these must be kept in sync with the regular symbol table (.symtab). So for every new symbol we emit, make sure to also emit a .symtab_shndx value to keep the arrays of equal size. Note: since we're writing an UNDEF symbol, most GElf_Sym fields will be 0 and we can repurpose one (st_size) to host the 0 for the xshndx value. Fixes: 2f2f7e47f052 ("objtool: Add elf_create_undef_symbol()") Reported-by: Nick Desaulniers Suggested-by: Fangrui Song Signed-off-by: Peter Zijlstra (Intel) Tested-by: Nick Desaulniers Link: https://lkml.kernel.org/r/YL3q1qFO9QIRL/BA@hirez.programming.kicks-as= s.net Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/elf.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -768,7 +768,7 @@ static int elf_add_string(struct elf *el =20 struct symbol *elf_create_undef_symbol(struct elf *elf, const char *name) { - struct section *symtab; + struct section *symtab, *symtab_shndx; struct symbol *sym; Elf_Data *data; Elf_Scn *s; @@ -819,6 +819,29 @@ struct symbol *elf_create_undef_symbol(s symtab->len +=3D data->d_size; symtab->changed =3D true; =20 + symtab_shndx =3D find_section_by_name(elf, ".symtab_shndx"); + if (symtab_shndx) { + s =3D elf_getscn(elf->elf, symtab_shndx->idx); + if (!s) { + WARN_ELF("elf_getscn"); + return NULL; + } + + data =3D elf_newdata(s); + if (!data) { + WARN_ELF("elf_newdata"); + return NULL; + } + + data->d_buf =3D &sym->sym.st_size; /* conveniently 0 */ + data->d_size =3D sizeof(Elf32_Word); + data->d_align =3D 4; + data->d_type =3D ELF_T_WORD; + + symtab_shndx->len +=3D 4; + symtab_shndx->changed =3D true; + } + sym->sec =3D find_section_by_index(elf, 0); =20 elf_add_symbol(elf, sym); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D09ADC433EF for ; Sat, 23 Jul 2022 10:01:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237572AbiGWKBU (ORCPT ); Sat, 23 Jul 2022 06:01:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237777AbiGWKAW (ORCPT ); Sat, 23 Jul 2022 06:00:22 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13BAC77A75; Sat, 23 Jul 2022 02:58:23 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4B2D5B82C1A; Sat, 23 Jul 2022 09:58:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0A3DC341C0; Sat, 23 Jul 2022 09:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570301; bh=AAi8L56AzRyySQ+Kplk53X8AoGVDe7CwzbFO9ViCi28=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bfhNoVtt1BBjBuiWhgvB/L6lzVZ2Kz7OR2KT9efPT58vsC5KS4rmBj6Q2M+L5N5Od Yq8PiZvYMjxCEtc+mGZL8BgihBEaHDyMqKvqL8rjiPbIEZd9Znh0ELuB0l2qYhqvm9 9i68vozD8knXZp5Vpf7pu+oDKaEjzz8Nu7Su//QU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Lukasz Majczak , Nathan Chancellor , "Peter Zijlstra (Intel)" , Ben Hutchings Subject: [PATCH 5.10 038/148] objtool: Only rewrite unconditional retpoline thunk calls Date: Sat, 23 Jul 2022 11:54:10 +0200 Message-Id: <20220723095235.007621512@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 2d49b721dc18c113d5221f4cf5a6104eb66cb7f2 upstream. It turns out that the compilers generate conditional branches to the retpoline thunks like: 5d5: 0f 85 00 00 00 00 jne 5db 5d7: R_X86_64_PLT32 __x86_indirect_thunk_r11-0x4 while the rewrite can only handle JMP/CALL to the thunks. The result is the alternative wrecking the code. Make sure to skip writing the alternatives for conditional branches. Fixes: 9bc0bb50727c ("objtool/x86: Rewrite retpoline thunk calls") Reported-by: Lukasz Majczak Reported-by: Nathan Chancellor Signed-off-by: Peter Zijlstra (Intel) Tested-by: Nathan Chancellor Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch/x86/decode.c | 4 ++++ 1 file changed, 4 insertions(+) --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -674,6 +674,10 @@ int arch_rewrite_retpolines(struct objto =20 list_for_each_entry(insn, &file->retpoline_call_list, call_node) { =20 + if (insn->type !=3D INSN_JUMP_DYNAMIC && + insn->type !=3D INSN_CALL_DYNAMIC) + continue; + if (!strcmp(insn->sec->name, ".text.__x86.indirect_thunk")) continue; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2562C433EF for ; Sat, 23 Jul 2022 10:01:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237909AbiGWKBZ (ORCPT ); Sat, 23 Jul 2022 06:01:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237888AbiGWKAX (ORCPT ); Sat, 23 Jul 2022 06:00:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 197AE785B1; Sat, 23 Jul 2022 02:58:24 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5764C611BD; Sat, 23 Jul 2022 09:58:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 690C5C341C0; Sat, 23 Jul 2022 09:58:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570303; bh=b3T8UY6FZP2wcxf8N/fTReXZewBuYrszqPfDeP+Iw4Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YKTx3NReB94/BFtQmE/WZ7MMImvZ48fBtQCmwNnd69Tzea9tbbHKU4zLw7t3QuP4/ 3fL/n6S8HHhkXu3N4KVy/kWv660iZlxthlp49c/GQohjNgxkrqXn78cIH1pWLsw9Ol 5RLMqecUuwcaQOxDJgsl2or59OdaMTKWgweMCYSc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Ingo Molnar , Ben Hutchings Subject: [PATCH 5.10 039/148] objtool/x86: Ignore __x86_indirect_alt_* symbols Date: Sat, 23 Jul 2022 11:54:11 +0200 Message-Id: <20220723095235.268633228@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 31197d3a0f1caeb60fb01f6755e28347e4f44037 upstream. Because the __x86_indirect_alt* symbols are just that, objtool will try and validate them as regular symbols, instead of the alternative replacements that they are. This goes sideways for FRAME_POINTER=3Dy builds; which generate a fair amount of warnings. Fixes: 9bc0bb50727c ("objtool/x86: Rewrite retpoline thunk calls") Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/YNCgxwLBiK9wclYJ@hirez.programming.kicks-as= s.net Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/lib/retpoline.S | 4 ++++ 1 file changed, 4 insertions(+) --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -58,12 +58,16 @@ SYM_FUNC_START_NOALIGN(__x86_indirect_al 2: .skip 5-(2b-1b), 0x90 SYM_FUNC_END(__x86_indirect_alt_call_\reg) =20 +STACK_FRAME_NON_STANDARD(__x86_indirect_alt_call_\reg) + SYM_FUNC_START_NOALIGN(__x86_indirect_alt_jmp_\reg) ANNOTATE_RETPOLINE_SAFE 1: jmp *%\reg 2: .skip 5-(2b-1b), 0x90 SYM_FUNC_END(__x86_indirect_alt_jmp_\reg) =20 +STACK_FRAME_NON_STANDARD(__x86_indirect_alt_jmp_\reg) + .endm =20 /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A247CC43334 for ; Sat, 23 Jul 2022 10:01:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237996AbiGWKBk (ORCPT ); Sat, 23 Jul 2022 06:01:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237914AbiGWKAh (ORCPT ); Sat, 23 Jul 2022 06:00:37 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E0693E760; Sat, 23 Jul 2022 02:58:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3B3C7611BF; Sat, 23 Jul 2022 09:58:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 43D87C341C0; Sat, 23 Jul 2022 09:58:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570306; bh=Cc2FMcbo41U7/t4HCIrRJ9iCyx8ihS/uOqE18E+9374=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tf/6n7YuvNVqLV+XK3KH/ubMnw0U3JFyVLrIlLwfv4JZg05WORjeDd37nZ4Sv+vYN 7/PHQjqVbVtfDgyDf7+5tkPm3Y/HLLNJCw5dZS1uzaVhmYmMEBvtBc9s4dYckv+lge 07ugfkBTJFehBVNfoCeACXIAX4SQsq0cmfRX2338= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Joe Lawrence , Josh Poimboeuf , Ingo Molnar , Ben Hutchings Subject: [PATCH 5.10 040/148] objtool: Dont make .altinstructions writable Date: Sat, 23 Jul 2022 11:54:12 +0200 Message-Id: <20220723095235.534043721@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit e31694e0a7a709293319475d8001e05e31f2178c upstream. When objtool creates the .altinstructions section, it sets the SHF_WRITE flag to make the section writable -- unless the section had already been previously created by the kernel. The mismatch between kernel-created and objtool-created section flags can cause failures with external tooling (kpatch-build). And the section doesn't need to be writable anyway. Make the section flags consistent with the kernel's. Fixes: 9bc0bb50727c ("objtool/x86: Rewrite retpoline thunk calls") Reported-by: Joe Lawrence Signed-off-by: Josh Poimboeuf Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/6c284ae89717889ea136f9f0064d914cd8329d31.16= 24462939.git.jpoimboe@redhat.com Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch/x86/decode.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -611,7 +611,7 @@ static int elf_add_alternative(struct el sec =3D find_section_by_name(elf, ".altinstructions"); if (!sec) { sec =3D elf_create_section(elf, ".altinstructions", - SHF_WRITE, size, 0); + SHF_ALLOC, size, 0); =20 if (!sec) { WARN_ELF("elf_create_section"); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B7DEC43334 for ; Sat, 23 Jul 2022 10:01:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237536AbiGWKBz (ORCPT ); Sat, 23 Jul 2022 06:01:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237938AbiGWKAx (ORCPT ); Sat, 23 Jul 2022 06:00:53 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 479AB3F334; Sat, 23 Jul 2022 02:58:32 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C45CEB82C1A; Sat, 23 Jul 2022 09:58:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 34642C341C7; Sat, 23 Jul 2022 09:58:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570309; bh=XZe6mZDywRYuCTJTB4/Kt7NN12giWZC5dpHCMsUZakg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0EqX2Ay7Lks1q1ATRNEyWTXw7sifxlaXRThxS1iYgoSiM2qr3jaPcPko40HloScyD JAoZJAaouiz38pu1fKR+6cvWOSauuJ7teBUFPeoEh5CioPoa3S7OHrb4d+Xr1rOcjd CZB2oXzPqx1NJIVP5fTHV76OD7eOZgn+SHBQBweE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Stephen Rothwell , Borislav Petkov , "Peter Zijlstra (Intel)" , Josh Poimboeuf , Nathan Chancellor , Ben Hutchings Subject: [PATCH 5.10 041/148] objtool: Teach get_alt_entry() about more relocation types Date: Sat, 23 Jul 2022 11:54:13 +0200 Message-Id: <20220723095235.834263566@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 24ff652573754fe4c03213ebd26b17e86842feb3 upstream. Occasionally objtool encounters symbol (as opposed to section) relocations in .altinstructions. Typically they are the alternatives written by elf_add_alternative() as encountered on a noinstr validation run on vmlinux after having already ran objtool on the individual .o files. Basically this is the counterpart of commit 44f6a7c0755d ("objtool: Fix seg fault with Clang non-section symbols"), because when these new assemblers (binutils now also does this) strip the section symbols, elf_add_reloc_to_insn() is forced to emit symbol based relocations. As such, teach get_alt_entry() about different relocation types. Fixes: 9bc0bb50727c ("objtool/x86: Rewrite retpoline thunk calls") Reported-by: Stephen Rothwell Reported-by: Borislav Petkov Signed-off-by: Peter Zijlstra (Intel) Acked-by: Josh Poimboeuf Tested-by: Nathan Chancellor Link: https://lore.kernel.org/r/YVWUvknIEVNkPvnP@hirez.programming.kicks-as= s.net Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/special.c | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -55,6 +55,24 @@ void __weak arch_handle_alternative(unsi { } =20 +static bool reloc2sec_off(struct reloc *reloc, struct section **sec, unsig= ned long *off) +{ + switch (reloc->sym->type) { + case STT_FUNC: + *sec =3D reloc->sym->sec; + *off =3D reloc->sym->offset + reloc->addend; + return true; + + case STT_SECTION: + *sec =3D reloc->sym->sec; + *off =3D reloc->addend; + return true; + + default: + return false; + } +} + static int get_alt_entry(struct elf *elf, struct special_entry *entry, struct section *sec, int idx, struct special_alt *alt) @@ -87,15 +105,12 @@ static int get_alt_entry(struct elf *elf WARN_FUNC("can't find orig reloc", sec, offset + entry->orig); return -1; } - if (orig_reloc->sym->type !=3D STT_SECTION) { - WARN_FUNC("don't know how to handle non-section reloc symbol %s", + if (!reloc2sec_off(orig_reloc, &alt->orig_sec, &alt->orig_off)) { + WARN_FUNC("don't know how to handle reloc symbol type: %s", sec, offset + entry->orig, orig_reloc->sym->name); return -1; } =20 - alt->orig_sec =3D orig_reloc->sym->sec; - alt->orig_off =3D orig_reloc->addend; - if (!entry->group || alt->new_len) { new_reloc =3D find_reloc_by_dest(elf, sec, offset + entry->new); if (!new_reloc) { @@ -112,8 +127,11 @@ static int get_alt_entry(struct elf *elf if (arch_is_retpoline(new_reloc->sym)) return 1; =20 - alt->new_sec =3D new_reloc->sym->sec; - alt->new_off =3D (unsigned int)new_reloc->addend; + if (!reloc2sec_off(new_reloc, &alt->new_sec, &alt->new_off)) { + WARN_FUNC("don't know how to handle reloc symbol type: %s", + sec, offset + entry->new, new_reloc->sym->name); + return -1; + } =20 /* _ASM_EXTABLE_EX hack */ if (alt->new_off >=3D 0x7ffffff0) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3455C433EF for ; Sat, 23 Jul 2022 10:02:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237941AbiGWKCA (ORCPT ); Sat, 23 Jul 2022 06:02:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237762AbiGWKAy (ORCPT ); Sat, 23 Jul 2022 06:00:54 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 686287AB37; Sat, 23 Jul 2022 02:58:33 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EF174611BF; Sat, 23 Jul 2022 09:58:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0BA59C341C0; Sat, 23 Jul 2022 09:58:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570312; bh=p69Pkhj7FSTSQGKzGUI/7ybWeLAgcbtPAlo52bDLKgs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o1+vpE3mkCX65VqBzLyYz2IrLovWokugbYS95yBDdQtyag2Sos65nsLeJVxfVn33C sdw7b6af6NqEpaYsqwAQjpFM6pN9Jd8umAG6L4AmZFwgqbclBvkigvNTI96g6vmO3A Y/8YThA/McvYNI+mKDGrFJ6qytyI3+9kAuCTaqUk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Linus Torvalds , Ben Hutchings Subject: [PATCH 5.10 042/148] objtool: print out the symbol type when complaining about it Date: Sat, 23 Jul 2022 11:54:14 +0200 Message-Id: <20220723095236.117871422@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Linus Torvalds commit 7fab1c12bde926c5a8c7d5984c551d0854d7e0b3 upstream. The objtool warning that the kvm instruction emulation code triggered wasn't very useful: arch/x86/kvm/emulate.o: warning: objtool: __ex_table+0x4: don't know ho= w to handle reloc symbol type: kvm_fastop_exception in that it helpfully tells you which symbol name it had trouble figuring out the relocation for, but it doesn't actually say what the unknown symbol type was that triggered it all. In this case it was because of missing type information (type 0, aka STT_NOTYPE), but on the whole it really should just have printed that out as part of the message. Because if this warning triggers, that's very much the first thing you want to know - why did reloc2sec_off() return failure for that symbol? So rather than just saying you can't handle some type of symbol without saying what the type _was_, just print out the type number too. Fixes: 24ff65257375 ("objtool: Teach get_alt_entry() about more relocation = types") Link: https://lore.kernel.org/lkml/CAHk-=3DwiZwq-0LknKhXN4M+T8jbxn_2i9mcKpO= +OaBSSq_Eh7tg@mail.gmail.com/ Signed-off-by: Linus Torvalds Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/special.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -106,8 +106,10 @@ static int get_alt_entry(struct elf *elf return -1; } if (!reloc2sec_off(orig_reloc, &alt->orig_sec, &alt->orig_off)) { - WARN_FUNC("don't know how to handle reloc symbol type: %s", - sec, offset + entry->orig, orig_reloc->sym->name); + WARN_FUNC("don't know how to handle reloc symbol type %d: %s", + sec, offset + entry->orig, + orig_reloc->sym->type, + orig_reloc->sym->name); return -1; } =20 @@ -128,8 +130,10 @@ static int get_alt_entry(struct elf *elf return 1; =20 if (!reloc2sec_off(new_reloc, &alt->new_sec, &alt->new_off)) { - WARN_FUNC("don't know how to handle reloc symbol type: %s", - sec, offset + entry->new, new_reloc->sym->name); + WARN_FUNC("don't know how to handle reloc symbol type %d: %s", + sec, offset + entry->new, + new_reloc->sym->type, + new_reloc->sym->name); return -1; } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12B57CCA483 for ; Sat, 23 Jul 2022 10:02:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238031AbiGWKCE (ORCPT ); Sat, 23 Jul 2022 06:02:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237850AbiGWKBF (ORCPT ); Sat, 23 Jul 2022 06:01:05 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A9BC7969C; Sat, 23 Jul 2022 02:58:37 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7F1ADB82B92; Sat, 23 Jul 2022 09:58:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DE17DC341C7; Sat, 23 Jul 2022 09:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570315; bh=Z0Ife+YnzVRgmcIv2ueknAX3OEj7I9mFJI5lNXufmyA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hKNtpT5iN2pBTr9/NWVfBbwMooEQOs1dpHhE9myPA5GDcgAs4o737JX983j2697pv 5DMzO5RmONqlil3N8Oi9NL0gxp1RGY2MiVElfhmPIkoJa5lh4bYKYsKcjmOW4q3nSO j+f6TGTOxDrUxi0tKM21lWbPO5vqHG/ykYm33QTw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Linus Torvalds , Josh Poimboeuf , Peter Zijlstra , x86@kernel.org, Miroslav Benes , Ben Hutchings Subject: [PATCH 5.10 043/148] objtool: Remove reloc symbol type checks in get_alt_entry() Date: Sat, 23 Jul 2022 11:54:15 +0200 Message-Id: <20220723095236.390915078@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 4d8b35968bbf9e42b6b202eedb510e2c82ad8b38 upstream. Converting a special section's relocation reference to a symbol is straightforward. No need for objtool to complain that it doesn't know how to handle it. Just handle it. This fixes the following warning: arch/x86/kvm/emulate.o: warning: objtool: __ex_table+0x4: don't know how = to handle reloc symbol type: kvm_fastop_exception Fixes: 24ff65257375 ("objtool: Teach get_alt_entry() about more relocation = types") Reported-by: Linus Torvalds Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/feadbc3dfb3440d973580fad8d3db873cbfe1694.16= 33367242.git.jpoimboe@redhat.com Cc: Peter Zijlstra Cc: x86@kernel.org Cc: Miroslav Benes Cc: linux-kernel@vger.kernel.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/special.c | 36 +++++++----------------------------- 1 file changed, 7 insertions(+), 29 deletions(-) --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -55,22 +55,11 @@ void __weak arch_handle_alternative(unsi { } =20 -static bool reloc2sec_off(struct reloc *reloc, struct section **sec, unsig= ned long *off) +static void reloc_to_sec_off(struct reloc *reloc, struct section **sec, + unsigned long *off) { - switch (reloc->sym->type) { - case STT_FUNC: - *sec =3D reloc->sym->sec; - *off =3D reloc->sym->offset + reloc->addend; - return true; - - case STT_SECTION: - *sec =3D reloc->sym->sec; - *off =3D reloc->addend; - return true; - - default: - return false; - } + *sec =3D reloc->sym->sec; + *off =3D reloc->sym->offset + reloc->addend; } =20 static int get_alt_entry(struct elf *elf, struct special_entry *entry, @@ -105,13 +94,8 @@ static int get_alt_entry(struct elf *elf WARN_FUNC("can't find orig reloc", sec, offset + entry->orig); return -1; } - if (!reloc2sec_off(orig_reloc, &alt->orig_sec, &alt->orig_off)) { - WARN_FUNC("don't know how to handle reloc symbol type %d: %s", - sec, offset + entry->orig, - orig_reloc->sym->type, - orig_reloc->sym->name); - return -1; - } + + reloc_to_sec_off(orig_reloc, &alt->orig_sec, &alt->orig_off); =20 if (!entry->group || alt->new_len) { new_reloc =3D find_reloc_by_dest(elf, sec, offset + entry->new); @@ -129,13 +113,7 @@ static int get_alt_entry(struct elf *elf if (arch_is_retpoline(new_reloc->sym)) return 1; =20 - if (!reloc2sec_off(new_reloc, &alt->new_sec, &alt->new_off)) { - WARN_FUNC("don't know how to handle reloc symbol type %d: %s", - sec, offset + entry->new, - new_reloc->sym->type, - new_reloc->sym->name); - return -1; - } + reloc_to_sec_off(new_reloc, &alt->new_sec, &alt->new_off); =20 /* _ASM_EXTABLE_EX hack */ if (alt->new_off >=3D 0x7ffffff0) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C4ABCCA483 for ; Sat, 23 Jul 2022 10:02:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237762AbiGWKCY (ORCPT ); Sat, 23 Jul 2022 06:02:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237538AbiGWKBy (ORCPT ); Sat, 23 Jul 2022 06:01:54 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 637087D1FA; Sat, 23 Jul 2022 02:58:40 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 61B39B82C1A; Sat, 23 Jul 2022 09:58:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9840C341C0; Sat, 23 Jul 2022 09:58:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570318; bh=jvHwMZhTHBgi2667p9dTXzJhBqmd7pBBtH9FrE3/mks=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BmCPnCpuBau/vwhRAO/vR5BWQQV0i+6cwOVjX/rQjrBtJmAe6atbwRJ9E72RQul4G 1/tg7S4rAaHUM+/bZJO9JQL1kd13q4nigqxeLOC5JZxLxNZhteHt0fdmlbn73zCCT2 9oS6EZvyx7+3F5mA0TNcK8R3ZKaHBTGY5lDatR1M= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Joe Lawrence , Miroslav Benes , Josh Poimboeuf , Andy Lavr , Peter Zijlstra , x86@kernel.org, Ben Hutchings Subject: [PATCH 5.10 044/148] objtool: Make .altinstructions section entry size consistent Date: Sat, 23 Jul 2022 11:54:16 +0200 Message-Id: <20220723095236.664921242@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Joe Lawrence commit dc02368164bd0ec603e3f5b3dd8252744a667b8a upstream. Commit e31694e0a7a7 ("objtool: Don't make .altinstructions writable") aligned objtool-created and kernel-created .altinstructions section flags, but there remains a minor discrepency in their use of a section entry size: objtool sets one while the kernel build does not. While sh_entsize of sizeof(struct alt_instr) seems intuitive, this small deviation can cause failures with external tooling (kpatch-build). Fix this by creating new .altinstructions sections with sh_entsize of 0 and then later updating sec->sh_size as alternatives are added to the section. An added benefit is avoiding the data descriptor and buffer created by elf_create_section(), but previously unused by elf_add_alternative(). Fixes: 9bc0bb50727c ("objtool/x86: Rewrite retpoline thunk calls") Signed-off-by: Joe Lawrence Reviewed-by: Miroslav Benes Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/20210822225037.54620-2-joe.lawrence@redhat.= com Cc: Andy Lavr Cc: Peter Zijlstra Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch/x86/decode.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -611,7 +611,7 @@ static int elf_add_alternative(struct el sec =3D find_section_by_name(elf, ".altinstructions"); if (!sec) { sec =3D elf_create_section(elf, ".altinstructions", - SHF_ALLOC, size, 0); + SHF_ALLOC, 0, 0); =20 if (!sec) { WARN_ELF("elf_create_section"); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0AC8C433EF for ; Sat, 23 Jul 2022 10:00:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237526AbiGWKAi (ORCPT ); Sat, 23 Jul 2022 06:00:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237522AbiGWJ7n (ORCPT ); Sat, 23 Jul 2022 05:59:43 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84ECE72EFC; Sat, 23 Jul 2022 02:58:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DB754B82C1B; Sat, 23 Jul 2022 09:58:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4E689C341C0; Sat, 23 Jul 2022 09:58:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570289; bh=GyAi/z9kd5x4+CY3oG+36WFHdd/w1Bu8yktqoqmX19g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=1oXE2bS6OxDSj5AXD9brB9OJJ2wMpmwcUqjtxWbFEVOeT9nhPURz96KAESHwXVbVa x3706/w52/ncodRo1WWtq8/0AcsATOpQAKzIiXl7EWq4RpzptvCsVtNCSjo7stgUX8 MB3UvRwYSFLV51NDybspJt8M8NbfaMNzMseItQ7k= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Andi Kleen , "Peter Zijlstra (Intel)" , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 045/148] objtool: Introduce CFI hash Date: Sat, 23 Jul 2022 11:54:17 +0200 Message-Id: <20220723095236.920893541@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 8b946cc38e063f0f7bb67789478c38f6d7d457c9 upstream. Andi reported that objtool on vmlinux.o consumes more memory than his system has, leading to horrific performance. This is in part because we keep a struct instruction for every instruction in the file in-memory. Shrink struct instruction by removing the CFI state (which includes full register state) from it and demand allocating it. Given most instructions don't actually change CFI state, there's lots of repetition there, so add a hash table to find previous CFI instances. Reduces memory consumption (and runtime) for processing an x86_64-allyesconfig: pre: 4:40.84 real, 143.99 user, 44.18 sys, 30624988 mem post: 2:14.61 real, 108.58 user, 25.04 sys, 16396184 mem Suggested-by: Andi Kleen Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/20210624095147.756759107@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: - Don't use bswap_if_needed() since we don't have any of the other fixes for mixed-endian cross-compilation - Since we don't have "objtool: Rewrite hashtable sizing", make cfi_hash_alloc() set the number of bits similarly to elf_hash_bits() - objtool doesn't have any mcount handling - Adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch.h | 2=20 tools/objtool/arch/x86/decode.c | 20 ++--- tools/objtool/cfi.h | 2=20 tools/objtool/check.c | 154 +++++++++++++++++++++++++++++++++++= ----- tools/objtool/check.h | 2=20 tools/objtool/orc_gen.c | 15 ++- 6 files changed, 160 insertions(+), 35 deletions(-) --- a/tools/objtool/arch.h +++ b/tools/objtool/arch.h @@ -84,7 +84,7 @@ unsigned long arch_dest_reloc_offset(int =20 const char *arch_nop_insn(int len); =20 -int arch_decode_hint_reg(struct instruction *insn, u8 sp_reg); +int arch_decode_hint_reg(u8 sp_reg, int *base); =20 bool arch_is_retpoline(struct symbol *sym); =20 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -706,34 +706,32 @@ int arch_rewrite_retpolines(struct objto return 0; } =20 -int arch_decode_hint_reg(struct instruction *insn, u8 sp_reg) +int arch_decode_hint_reg(u8 sp_reg, int *base) { - struct cfi_reg *cfa =3D &insn->cfi.cfa; - switch (sp_reg) { case ORC_REG_UNDEFINED: - cfa->base =3D CFI_UNDEFINED; + *base =3D CFI_UNDEFINED; break; case ORC_REG_SP: - cfa->base =3D CFI_SP; + *base =3D CFI_SP; break; case ORC_REG_BP: - cfa->base =3D CFI_BP; + *base =3D CFI_BP; break; case ORC_REG_SP_INDIRECT: - cfa->base =3D CFI_SP_INDIRECT; + *base =3D CFI_SP_INDIRECT; break; case ORC_REG_R10: - cfa->base =3D CFI_R10; + *base =3D CFI_R10; break; case ORC_REG_R13: - cfa->base =3D CFI_R13; + *base =3D CFI_R13; break; case ORC_REG_DI: - cfa->base =3D CFI_DI; + *base =3D CFI_DI; break; case ORC_REG_DX: - cfa->base =3D CFI_DX; + *base =3D CFI_DX; break; default: return -1; --- a/tools/objtool/cfi.h +++ b/tools/objtool/cfi.h @@ -7,6 +7,7 @@ #define _OBJTOOL_CFI_H =20 #include "cfi_regs.h" +#include =20 #define CFI_UNDEFINED -1 #define CFI_CFA -2 @@ -24,6 +25,7 @@ struct cfi_init_state { }; =20 struct cfi_state { + struct hlist_node hash; /* must be first, cficmp() */ struct cfi_reg regs[CFI_NUM_REGS]; struct cfi_reg vals[CFI_NUM_REGS]; struct cfi_reg cfa; --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -5,6 +5,7 @@ =20 #include #include +#include =20 #include "builtin.h" #include "cfi.h" @@ -25,7 +26,11 @@ struct alternative { bool skip_orig; }; =20 -struct cfi_init_state initial_func_cfi; +static unsigned long nr_cfi, nr_cfi_reused, nr_cfi_cache; + +static struct cfi_init_state initial_func_cfi; +static struct cfi_state init_cfi; +static struct cfi_state func_cfi; =20 struct instruction *find_insn(struct objtool_file *file, struct section *sec, unsigned long offset) @@ -265,6 +270,78 @@ static void init_insn_state(struct insn_ state->noinstr =3D sec->noinstr; } =20 +static struct cfi_state *cfi_alloc(void) +{ + struct cfi_state *cfi =3D calloc(sizeof(struct cfi_state), 1); + if (!cfi) { + WARN("calloc failed"); + exit(1); + } + nr_cfi++; + return cfi; +} + +static int cfi_bits; +static struct hlist_head *cfi_hash; + +static inline bool cficmp(struct cfi_state *cfi1, struct cfi_state *cfi2) +{ + return memcmp((void *)cfi1 + sizeof(cfi1->hash), + (void *)cfi2 + sizeof(cfi2->hash), + sizeof(struct cfi_state) - sizeof(struct hlist_node)); +} + +static inline u32 cfi_key(struct cfi_state *cfi) +{ + return jhash((void *)cfi + sizeof(cfi->hash), + sizeof(*cfi) - sizeof(cfi->hash), 0); +} + +static struct cfi_state *cfi_hash_find_or_add(struct cfi_state *cfi) +{ + struct hlist_head *head =3D &cfi_hash[hash_min(cfi_key(cfi), cfi_bits)]; + struct cfi_state *obj; + + hlist_for_each_entry(obj, head, hash) { + if (!cficmp(cfi, obj)) { + nr_cfi_cache++; + return obj; + } + } + + obj =3D cfi_alloc(); + *obj =3D *cfi; + hlist_add_head(&obj->hash, head); + + return obj; +} + +static void cfi_hash_add(struct cfi_state *cfi) +{ + struct hlist_head *head =3D &cfi_hash[hash_min(cfi_key(cfi), cfi_bits)]; + + hlist_add_head(&cfi->hash, head); +} + +static void *cfi_hash_alloc(void) +{ + cfi_bits =3D vmlinux ? ELF_HASH_BITS - 3 : 13; + cfi_hash =3D mmap(NULL, sizeof(struct hlist_head) << cfi_bits, + PROT_READ|PROT_WRITE, + MAP_PRIVATE|MAP_ANON, -1, 0); + if (cfi_hash =3D=3D (void *)-1L) { + WARN("mmap fail cfi_hash"); + cfi_hash =3D NULL; + } else if (stats) { + printf("cfi_bits: %d\n", cfi_bits); + } + + return cfi_hash; +} + +static unsigned long nr_insns; +static unsigned long nr_insns_visited; + /* * Call the arch-specific instruction decoder for all the instructions and= add * them to the global instruction list. @@ -275,7 +352,6 @@ static int decode_instructions(struct ob struct symbol *func; unsigned long offset; struct instruction *insn; - unsigned long nr_insns =3D 0; int ret; =20 for_each_sec(file, sec) { @@ -301,7 +377,6 @@ static int decode_instructions(struct ob memset(insn, 0, sizeof(*insn)); INIT_LIST_HEAD(&insn->alts); INIT_LIST_HEAD(&insn->stack_ops); - init_cfi_state(&insn->cfi); =20 insn->sec =3D sec; insn->offset =3D offset; @@ -1077,7 +1152,6 @@ static int handle_group_alt(struct objto memset(nop, 0, sizeof(*nop)); INIT_LIST_HEAD(&nop->alts); INIT_LIST_HEAD(&nop->stack_ops); - init_cfi_state(&nop->cfi); =20 nop->sec =3D special_alt->new_sec; nop->offset =3D special_alt->new_off + special_alt->new_len; @@ -1454,10 +1528,11 @@ static void set_func_state(struct cfi_st =20 static int read_unwind_hints(struct objtool_file *file) { + struct cfi_state cfi =3D init_cfi; struct section *sec, *relocsec; - struct reloc *reloc; struct unwind_hint *hint; struct instruction *insn; + struct reloc *reloc; int i; =20 sec =3D find_section_by_name(file->elf, ".discard.unwind_hints"); @@ -1495,19 +1570,24 @@ static int read_unwind_hints(struct objt insn->hint =3D true; =20 if (hint->type =3D=3D UNWIND_HINT_TYPE_FUNC) { - set_func_state(&insn->cfi); + insn->cfi =3D &func_cfi; continue; } =20 - if (arch_decode_hint_reg(insn, hint->sp_reg)) { + if (insn->cfi) + cfi =3D *(insn->cfi); + + if (arch_decode_hint_reg(hint->sp_reg, &cfi.cfa.base)) { WARN_FUNC("unsupported unwind_hint sp base reg %d", insn->sec, insn->offset, hint->sp_reg); return -1; } =20 - insn->cfi.cfa.offset =3D hint->sp_offset; - insn->cfi.type =3D hint->type; - insn->cfi.end =3D hint->end; + cfi.cfa.offset =3D hint->sp_offset; + cfi.type =3D hint->type; + cfi.end =3D hint->end; + + insn->cfi =3D cfi_hash_find_or_add(&cfi); } =20 return 0; @@ -2283,13 +2363,18 @@ static int propagate_alt_cfi(struct objt if (!insn->alt_group) return 0; =20 + if (!insn->cfi) { + WARN("CFI missing"); + return -1; + } + alt_cfi =3D insn->alt_group->cfi; group_off =3D insn->offset - insn->alt_group->first_insn->offset; =20 if (!alt_cfi[group_off]) { - alt_cfi[group_off] =3D &insn->cfi; + alt_cfi[group_off] =3D insn->cfi; } else { - if (memcmp(alt_cfi[group_off], &insn->cfi, sizeof(struct cfi_state))) { + if (cficmp(alt_cfi[group_off], insn->cfi)) { WARN_FUNC("stack layout conflict in alternatives", insn->sec, insn->offset); return -1; @@ -2335,9 +2420,14 @@ static int handle_insn_ops(struct instru =20 static bool insn_cfi_match(struct instruction *insn, struct cfi_state *cfi= 2) { - struct cfi_state *cfi1 =3D &insn->cfi; + struct cfi_state *cfi1 =3D insn->cfi; int i; =20 + if (!cfi1) { + WARN("CFI missing"); + return false; + } + if (memcmp(&cfi1->cfa, &cfi2->cfa, sizeof(cfi1->cfa))) { =20 WARN_FUNC("stack state mismatch: cfa1=3D%d%+d cfa2=3D%d%+d", @@ -2522,7 +2612,7 @@ static int validate_branch(struct objtoo struct instruction *insn, struct insn_state state) { struct alternative *alt; - struct instruction *next_insn; + struct instruction *next_insn, *prev_insn =3D NULL; struct section *sec; u8 visited; int ret; @@ -2551,15 +2641,25 @@ static int validate_branch(struct objtoo =20 if (insn->visited & visited) return 0; + } else { + nr_insns_visited++; } =20 if (state.noinstr) state.instr +=3D insn->instr; =20 - if (insn->hint) - state.cfi =3D insn->cfi; - else - insn->cfi =3D state.cfi; + if (insn->hint) { + state.cfi =3D *insn->cfi; + } else { + /* XXX track if we actually changed state.cfi */ + + if (prev_insn && !cficmp(prev_insn->cfi, &state.cfi)) { + insn->cfi =3D prev_insn->cfi; + nr_cfi_reused++; + } else { + insn->cfi =3D cfi_hash_find_or_add(&state.cfi); + } + } =20 insn->visited |=3D visited; =20 @@ -2709,6 +2809,7 @@ static int validate_branch(struct objtoo return 1; } =20 + prev_insn =3D insn; insn =3D next_insn; } =20 @@ -2964,10 +3065,20 @@ int check(struct objtool_file *file) int ret, warnings =3D 0; =20 arch_initial_func_cfi_state(&initial_func_cfi); + init_cfi_state(&init_cfi); + init_cfi_state(&func_cfi); + set_func_state(&func_cfi); + + if (!cfi_hash_alloc()) + goto out; + + cfi_hash_add(&init_cfi); + cfi_hash_add(&func_cfi); =20 ret =3D decode_sections(file); if (ret < 0) goto out; + warnings +=3D ret; =20 if (list_empty(&file->insn_list)) @@ -3011,6 +3122,13 @@ int check(struct objtool_file *file) goto out; warnings +=3D ret; =20 + if (stats) { + printf("nr_insns_visited: %ld\n", nr_insns_visited); + printf("nr_cfi: %ld\n", nr_cfi); + printf("nr_cfi_reused: %ld\n", nr_cfi_reused); + printf("nr_cfi_cache: %ld\n", nr_cfi_cache); + } + out: /* * For now, don't fail the kernel build on fatal warnings. These --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -59,7 +59,7 @@ struct instruction { struct list_head alts; struct symbol *func; struct list_head stack_ops; - struct cfi_state cfi; + struct cfi_state *cfi; }; =20 static inline bool is_static_jump(struct instruction *insn) --- a/tools/objtool/orc_gen.c +++ b/tools/objtool/orc_gen.c @@ -12,13 +12,19 @@ #include "check.h" #include "warn.h" =20 -static int init_orc_entry(struct orc_entry *orc, struct cfi_state *cfi) +static int init_orc_entry(struct orc_entry *orc, struct cfi_state *cfi, + struct instruction *insn) { - struct instruction *insn =3D container_of(cfi, struct instruction, cfi); struct cfi_reg *bp =3D &cfi->regs[CFI_BP]; =20 memset(orc, 0, sizeof(*orc)); =20 + if (!cfi) { + orc->end =3D 0; + orc->sp_reg =3D ORC_REG_UNDEFINED; + return 0; + } + orc->end =3D cfi->end; =20 if (cfi->cfa.base =3D=3D CFI_UNDEFINED) { @@ -159,7 +165,7 @@ int orc_create(struct objtool_file *file int i; =20 if (!alt_group) { - if (init_orc_entry(&orc, &insn->cfi)) + if (init_orc_entry(&orc, insn->cfi, insn)) return -1; if (!memcmp(&prev_orc, &orc, sizeof(orc))) continue; @@ -183,7 +189,8 @@ int orc_create(struct objtool_file *file struct cfi_state *cfi =3D alt_group->cfi[i]; if (!cfi) continue; - if (init_orc_entry(&orc, cfi)) + /* errors are reported on the original insn */ + if (init_orc_entry(&orc, cfi, insn)) return -1; if (!memcmp(&prev_orc, &orc, sizeof(orc))) continue; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FE91C433EF for ; Sat, 23 Jul 2022 10:00:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237747AbiGWKAn (ORCPT ); Sat, 23 Jul 2022 06:00:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237708AbiGWJ7v (ORCPT ); Sat, 23 Jul 2022 05:59:51 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D948B7170B; Sat, 23 Jul 2022 02:58:13 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2F4D96116A; Sat, 23 Jul 2022 09:58:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3B5D1C341C0; Sat, 23 Jul 2022 09:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570292; bh=tKSV3e90dPUNedIPkdfEv0/qdUpBA/3raO94IT2xlr4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nY/bO+lveZN1ol59tEoM5UliuF4qnAupDVdBb33pHR2GknXy+PYbvFFHhcom4Snzk lubSgNUwvDPZFio+i9W2swkwbYZ72fLjhmH44bo4u9RCJM+pJI22Bdc+TpT70quj4a V9WoMoOvmgyVb/vx86GUqGXnIuJvDsxxZYwCi2fs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Marco Elver , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 046/148] objtool: Handle __sanitize_cov*() tail calls Date: Sat, 23 Jul 2022 11:54:18 +0200 Message-Id: <20220723095237.197522022@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit f56dae88a81fded66adf2bea9922d1d98d1da14f upstream. Turns out the compilers also generate tail calls to __sanitize_cov*(), make sure to also patch those out in noinstr code. Fixes: 0f1441b44e82 ("objtool: Fix noinstr vs KCOV") Signed-off-by: Peter Zijlstra (Intel) Acked-by: Marco Elver Link: https://lore.kernel.org/r/20210624095147.818783799@infradead.org Signed-off-by: Sasha Levin [bwh: Backported to 5.10: - objtool doesn't have any mcount handling - Write the NOPs as hex literals since we can't use ] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch.h | 1=20 tools/objtool/arch/x86/decode.c | 20 ++++++ tools/objtool/check.c | 123 +++++++++++++++++++++--------------= ----- 3 files changed, 86 insertions(+), 58 deletions(-) --- a/tools/objtool/arch.h +++ b/tools/objtool/arch.h @@ -83,6 +83,7 @@ unsigned long arch_jump_destination(stru unsigned long arch_dest_reloc_offset(int addend); =20 const char *arch_nop_insn(int len); +const char *arch_ret_insn(int len); =20 int arch_decode_hint_reg(u8 sp_reg, int *base); =20 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -586,6 +586,26 @@ const char *arch_nop_insn(int len) return nops[len-1]; } =20 +#define BYTE_RET 0xC3 + +const char *arch_ret_insn(int len) +{ + static const char ret[5][5] =3D { + { BYTE_RET }, + { BYTE_RET, 0x90 }, + { BYTE_RET, 0x66, 0x90 }, + { BYTE_RET, 0x0f, 0x1f, 0x00 }, + { BYTE_RET, 0x0f, 0x1f, 0x40, 0x00 }, + }; + + if (len < 1 || len > 5) { + WARN("invalid RET size: %d\n", len); + return NULL; + } + + return ret[len-1]; +} + /* asm/alternative.h ? */ =20 #define ALTINSTR_FLAG_INV (1 << 15) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -860,6 +860,60 @@ static struct reloc *insn_reloc(struct o return insn->reloc; } =20 +static void remove_insn_ops(struct instruction *insn) +{ + struct stack_op *op, *tmp; + + list_for_each_entry_safe(op, tmp, &insn->stack_ops, list) { + list_del(&op->list); + free(op); + } +} + +static void add_call_dest(struct objtool_file *file, struct instruction *i= nsn, + struct symbol *dest, bool sibling) +{ + struct reloc *reloc =3D insn_reloc(file, insn); + + insn->call_dest =3D dest; + if (!dest) + return; + + if (insn->call_dest->static_call_tramp) { + list_add_tail(&insn->call_node, + &file->static_call_list); + } + + /* + * Many compilers cannot disable KCOV with a function attribute + * so they need a little help, NOP out any KCOV calls from noinstr + * text. + */ + if (insn->sec->noinstr && + !strncmp(insn->call_dest->name, "__sanitizer_cov_", 16)) { + if (reloc) { + reloc->type =3D R_NONE; + elf_write_reloc(file->elf, reloc); + } + + elf_write_insn(file->elf, insn->sec, + insn->offset, insn->len, + sibling ? arch_ret_insn(insn->len) + : arch_nop_insn(insn->len)); + + insn->type =3D sibling ? INSN_RETURN : INSN_NOP; + } + + /* + * Whatever stack impact regular CALLs have, should be undone + * by the RETURN of the called function. + * + * Annotated intra-function calls retain the stack_ops but + * are converted to JUMP, see read_intra_function_calls(). + */ + remove_insn_ops(insn); +} + /* * Find the destination instructions for all jumps. */ @@ -898,11 +952,7 @@ static int add_jump_destinations(struct continue; } else if (insn->func) { /* internal or external sibling call (with reloc) */ - insn->call_dest =3D reloc->sym; - if (insn->call_dest->static_call_tramp) { - list_add_tail(&insn->call_node, - &file->static_call_list); - } + add_call_dest(file, insn, reloc->sym, true); continue; } else if (reloc->sym->sec->idx) { dest_sec =3D reloc->sym->sec; @@ -958,13 +1008,8 @@ static int add_jump_destinations(struct =20 } else if (insn->jump_dest->func->pfunc !=3D insn->func->pfunc && insn->jump_dest->offset =3D=3D insn->jump_dest->func->offset) { - /* internal sibling call (without reloc) */ - insn->call_dest =3D insn->jump_dest->func; - if (insn->call_dest->static_call_tramp) { - list_add_tail(&insn->call_node, - &file->static_call_list); - } + add_call_dest(file, insn, insn->jump_dest->func, true); } } } @@ -972,16 +1017,6 @@ static int add_jump_destinations(struct return 0; } =20 -static void remove_insn_ops(struct instruction *insn) -{ - struct stack_op *op, *tmp; - - list_for_each_entry_safe(op, tmp, &insn->stack_ops, list) { - list_del(&op->list); - free(op); - } -} - static struct symbol *find_call_destination(struct section *sec, unsigned = long offset) { struct symbol *call_dest; @@ -1000,6 +1035,7 @@ static int add_call_destinations(struct { struct instruction *insn; unsigned long dest_off; + struct symbol *dest; struct reloc *reloc; =20 for_each_insn(file, insn) { @@ -1009,7 +1045,9 @@ static int add_call_destinations(struct reloc =3D insn_reloc(file, insn); if (!reloc) { dest_off =3D arch_jump_destination(insn); - insn->call_dest =3D find_call_destination(insn->sec, dest_off); + dest =3D find_call_destination(insn->sec, dest_off); + + add_call_dest(file, insn, dest, false); =20 if (insn->ignore) continue; @@ -1027,9 +1065,8 @@ static int add_call_destinations(struct =20 } else if (reloc->sym->type =3D=3D STT_SECTION) { dest_off =3D arch_dest_reloc_offset(reloc->addend); - insn->call_dest =3D find_call_destination(reloc->sym->sec, - dest_off); - if (!insn->call_dest) { + dest =3D find_call_destination(reloc->sym->sec, dest_off); + if (!dest) { WARN_FUNC("can't find call dest symbol at %s+0x%lx", insn->sec, insn->offset, reloc->sym->sec->name, @@ -1037,6 +1074,8 @@ static int add_call_destinations(struct return -1; } =20 + add_call_dest(file, insn, dest, false); + } else if (arch_is_retpoline(reloc->sym)) { /* * Retpoline calls are really dynamic calls in @@ -1052,39 +1091,7 @@ static int add_call_destinations(struct continue; =20 } else - insn->call_dest =3D reloc->sym; - - if (insn->call_dest && insn->call_dest->static_call_tramp) { - list_add_tail(&insn->call_node, - &file->static_call_list); - } - - /* - * Many compilers cannot disable KCOV with a function attribute - * so they need a little help, NOP out any KCOV calls from noinstr - * text. - */ - if (insn->sec->noinstr && - !strncmp(insn->call_dest->name, "__sanitizer_cov_", 16)) { - if (reloc) { - reloc->type =3D R_NONE; - elf_write_reloc(file->elf, reloc); - } - - elf_write_insn(file->elf, insn->sec, - insn->offset, insn->len, - arch_nop_insn(insn->len)); - insn->type =3D INSN_NOP; - } - - /* - * Whatever stack impact regular CALLs have, should be undone - * by the RETURN of the called function. - * - * Annotated intra-function calls retain the stack_ops but - * are converted to JUMP, see read_intra_function_calls(). - */ - remove_insn_ops(insn); + add_call_dest(file, insn, reloc->sym, false); } =20 return 0; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE538C433EF for ; Sat, 23 Jul 2022 10:01:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237647AbiGWKBM (ORCPT ); Sat, 23 Jul 2022 06:01:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237776AbiGWKAM (ORCPT ); Sat, 23 Jul 2022 06:00:12 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52F7C774B7; Sat, 23 Jul 2022 02:58:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 4B64BCE0DBE; Sat, 23 Jul 2022 09:58:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07D53C341C0; Sat, 23 Jul 2022 09:58:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570295; bh=P8df0VZgaba5kE5vIVSyLhZJRZN76d9+sQs8YKCUaeM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u98AwpspS2bGKTe9C7RbO1xLFhLhj01cIFU39h8GXZuZWuFNJ3LbOcflnEEstTyy2 7+yss9dv/3FWHTwr2DYKA+Vm4ScZqrawY2iiFf0hx1cWFMrESYegAmQwQ5Rc2jqYhe 5Z3WmYn603EpLlyXsEtJSnpQoZ/eJdyigEK7ThAU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 047/148] objtool: Classify symbols Date: Sat, 23 Jul 2022 11:54:19 +0200 Message-Id: <20220723095237.497630005@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 1739c66eb7bd5f27f1b69a5a26e10e8327d1e136 upstream. In order to avoid calling str*cmp() on symbol names, over and over, do them all once upfront and store the result. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120309.658539311@infradead.org [cascardo: no pv_target on struct symbol, because of missing db2b0c5d7b6f19b3c2cab08c531b65342eb5252b] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: objtool doesn't have any mcount handling] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 32 +++++++++++++++++++++----------- tools/objtool/elf.h | 7 +++++-- 2 files changed, 26 insertions(+), 13 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -889,8 +889,7 @@ static void add_call_dest(struct objtool * so they need a little help, NOP out any KCOV calls from noinstr * text. */ - if (insn->sec->noinstr && - !strncmp(insn->call_dest->name, "__sanitizer_cov_", 16)) { + if (insn->sec->noinstr && insn->call_dest->kcov) { if (reloc) { reloc->type =3D R_NONE; elf_write_reloc(file->elf, reloc); @@ -935,7 +934,7 @@ static int add_jump_destinations(struct } else if (reloc->sym->type =3D=3D STT_SECTION) { dest_sec =3D reloc->sym->sec; dest_off =3D arch_dest_reloc_offset(reloc->addend); - } else if (arch_is_retpoline(reloc->sym)) { + } else if (reloc->sym->retpoline_thunk) { /* * Retpoline jumps are really dynamic jumps in * disguise, so convert them accordingly. @@ -1076,7 +1075,7 @@ static int add_call_destinations(struct =20 add_call_dest(file, insn, dest, false); =20 - } else if (arch_is_retpoline(reloc->sym)) { + } else if (reloc->sym->retpoline_thunk) { /* * Retpoline calls are really dynamic calls in * disguise, so convert them accordingly. @@ -1733,17 +1732,28 @@ static int read_intra_function_calls(str return 0; } =20 -static int read_static_call_tramps(struct objtool_file *file) +static int classify_symbols(struct objtool_file *file) { struct section *sec; struct symbol *func; =20 for_each_sec(file, sec) { list_for_each_entry(func, &sec->symbol_list, list) { - if (func->bind =3D=3D STB_GLOBAL && - !strncmp(func->name, STATIC_CALL_TRAMP_PREFIX_STR, + if (func->bind !=3D STB_GLOBAL) + continue; + + if (!strncmp(func->name, STATIC_CALL_TRAMP_PREFIX_STR, strlen(STATIC_CALL_TRAMP_PREFIX_STR))) func->static_call_tramp =3D true; + + if (arch_is_retpoline(func)) + func->retpoline_thunk =3D true; + + if (!strcmp(func->name, "__fentry__")) + func->fentry =3D true; + + if (!strncmp(func->name, "__sanitizer_cov_", 16)) + func->kcov =3D true; } } =20 @@ -1805,7 +1815,7 @@ static int decode_sections(struct objtoo /* * Must be before add_{jump_call}_destination. */ - ret =3D read_static_call_tramps(file); + ret =3D classify_symbols(file); if (ret) return ret; =20 @@ -1863,9 +1873,9 @@ static int decode_sections(struct objtoo =20 static bool is_fentry_call(struct instruction *insn) { - if (insn->type =3D=3D INSN_CALL && insn->call_dest && - insn->call_dest->type =3D=3D STT_NOTYPE && - !strcmp(insn->call_dest->name, "__fentry__")) + if (insn->type =3D=3D INSN_CALL && + insn->call_dest && + insn->call_dest->fentry) return true; =20 return false; --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -55,8 +55,11 @@ struct symbol { unsigned long offset; unsigned int len; struct symbol *pfunc, *cfunc, *alias; - bool uaccess_safe; - bool static_call_tramp; + u8 uaccess_safe : 1; + u8 static_call_tramp : 1; + u8 retpoline_thunk : 1; + u8 fentry : 1; + u8 kcov : 1; }; =20 struct reloc { From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 628E7C43334 for ; Sat, 23 Jul 2022 10:04:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238273AbiGWKD7 (ORCPT ); Sat, 23 Jul 2022 06:03:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237984AbiGWKDc (ORCPT ); Sat, 23 Jul 2022 06:03:32 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1AD18AB09; Sat, 23 Jul 2022 02:59:30 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 2959FCE0DBE; Sat, 23 Jul 2022 09:59:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 15F5AC341C0; Sat, 23 Jul 2022 09:59:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570355; bh=b4lPAu6TIYBuJvohHsBAYnRu4yFmScVttnEgCBbUYFc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qeEAPTGZAHiw/D/G50afSWGdRyQsAwDowupN8hjN4fNVsJNhYuJAnAHmAy3xRgx/I WgiRd5WwllDq12dTLoK7y5x5SzeSpZ+IV455YrqZ5dNVdYVBUb2A63f+WZG4MUGjv9 uFDYj/9epltV5F//86tMeCqS9qwtLDSuMQii0uAk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 048/148] objtool: Explicitly avoid self modifying code in .altinstr_replacement Date: Sat, 23 Jul 2022 11:54:20 +0200 Message-Id: <20220723095237.747602693@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit dd003edeffa3cb87bc9862582004f405d77d7670 upstream. Assume ALTERNATIVE()s know what they're doing and do not change, or cause to change, instructions in .altinstr_replacement sections. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120309.722511775@infradead.org [cascardo: context adjustment] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: objtool doesn't have any mcount handling] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -870,18 +870,27 @@ static void remove_insn_ops(struct instr } } =20 -static void add_call_dest(struct objtool_file *file, struct instruction *i= nsn, - struct symbol *dest, bool sibling) +static void annotate_call_site(struct objtool_file *file, + struct instruction *insn, bool sibling) { struct reloc *reloc =3D insn_reloc(file, insn); + struct symbol *sym =3D insn->call_dest; =20 - insn->call_dest =3D dest; - if (!dest) + if (!sym) + sym =3D reloc->sym; + + /* + * Alternative replacement code is just template code which is + * sometimes copied to the original instruction. For now, don't + * annotate it. (In the future we might consider annotating the + * original instruction if/when it ever makes sense to do so.) + */ + if (!strcmp(insn->sec->name, ".altinstr_replacement")) return; =20 - if (insn->call_dest->static_call_tramp) { - list_add_tail(&insn->call_node, - &file->static_call_list); + if (sym->static_call_tramp) { + list_add_tail(&insn->call_node, &file->static_call_list); + return; } =20 /* @@ -889,7 +898,7 @@ static void add_call_dest(struct objtool * so they need a little help, NOP out any KCOV calls from noinstr * text. */ - if (insn->sec->noinstr && insn->call_dest->kcov) { + if (insn->sec->noinstr && sym->kcov) { if (reloc) { reloc->type =3D R_NONE; elf_write_reloc(file->elf, reloc); @@ -901,7 +910,16 @@ static void add_call_dest(struct objtool : arch_nop_insn(insn->len)); =20 insn->type =3D sibling ? INSN_RETURN : INSN_NOP; + return; } +} + +static void add_call_dest(struct objtool_file *file, struct instruction *i= nsn, + struct symbol *dest, bool sibling) +{ + insn->call_dest =3D dest; + if (!dest) + return; =20 /* * Whatever stack impact regular CALLs have, should be undone @@ -911,6 +929,8 @@ static void add_call_dest(struct objtool * are converted to JUMP, see read_intra_function_calls(). */ remove_insn_ops(insn); + + annotate_call_site(file, insn, sibling); } =20 /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F8BC433EF for ; Sat, 23 Jul 2022 10:02:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238067AbiGWKC2 (ORCPT ); Sat, 23 Jul 2022 06:02:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237939AbiGWKB5 (ORCPT ); Sat, 23 Jul 2022 06:01:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08F4F61D48; Sat, 23 Jul 2022 02:58:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 773F6611BF; Sat, 23 Jul 2022 09:58:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7C742C341C0; Sat, 23 Jul 2022 09:58:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570323; bh=YK9+XOi9SbioXxUkEVO0zL8ub18V/83umVoV5u7euKI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SEgiZguqLqA3dI1W/jKy09C9XvXs5ro2yRNDt7zwFo1my9H8uf2k4m1HyxIP5tat7 UeeH6iacXB6lJ6JEglQcbxIzApOIBG3N83ywDDUvsb4ti6Ei2Ogk+cfZmI2dAhupSd fChCWRuArhkc7tR2zCL9RuscEb8Ea2vICWgKimjo= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 049/148] objtool,x86: Replace alternatives with .retpoline_sites Date: Sat, 23 Jul 2022 11:54:21 +0200 Message-Id: <20220723095238.041412869@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 134ab5bd1883312d7a4b3033b05c6b5a1bb8889b upstream. Instead of writing complete alternatives, simply provide a list of all the retpoline thunk calls. Then the kernel is free to do with them as it pleases. Simpler code all-round. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120309.850007165@infradead.org [cascardo: fixed conflict because of missing 8b946cc38e063f0f7bb67789478c38f6d7d457c9] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: deleted functions had slightly different code] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/vmlinux.lds.S | 14 ++++ tools/objtool/arch/x86/decode.c | 120 ------------------------------------ tools/objtool/check.c | 132 ++++++++++++++++++++++++++++-------= ----- tools/objtool/elf.c | 83 ------------------------- tools/objtool/elf.h | 1=20 tools/objtool/special.c | 8 -- 6 files changed, 107 insertions(+), 251 deletions(-) --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -272,6 +272,20 @@ SECTIONS __parainstructions_end =3D .; } =20 +#ifdef CONFIG_RETPOLINE + /* + * List of instructions that call/jmp/jcc to retpoline thunks + * __x86_indirect_thunk_*(). These instructions can be patched along + * with alternatives, after which the section can be freed. + */ + . =3D ALIGN(8); + .retpoline_sites : AT(ADDR(.retpoline_sites) - LOAD_OFFSET) { + __retpoline_sites =3D .; + *(.retpoline_sites) + __retpoline_sites_end =3D .; + } +#endif + /* * struct alt_inst entries. From the header (alternative.h): * "Alternative instructions for different CPU types or capabilities" --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -606,126 +606,6 @@ const char *arch_ret_insn(int len) return ret[len-1]; } =20 -/* asm/alternative.h ? */ - -#define ALTINSTR_FLAG_INV (1 << 15) -#define ALT_NOT(feat) ((feat) | ALTINSTR_FLAG_INV) - -struct alt_instr { - s32 instr_offset; /* original instruction */ - s32 repl_offset; /* offset to replacement instruction */ - u16 cpuid; /* cpuid bit set for replacement */ - u8 instrlen; /* length of original instruction */ - u8 replacementlen; /* length of new instruction */ -} __packed; - -static int elf_add_alternative(struct elf *elf, - struct instruction *orig, struct symbol *sym, - int cpuid, u8 orig_len, u8 repl_len) -{ - const int size =3D sizeof(struct alt_instr); - struct alt_instr *alt; - struct section *sec; - Elf_Scn *s; - - sec =3D find_section_by_name(elf, ".altinstructions"); - if (!sec) { - sec =3D elf_create_section(elf, ".altinstructions", - SHF_ALLOC, 0, 0); - - if (!sec) { - WARN_ELF("elf_create_section"); - return -1; - } - } - - s =3D elf_getscn(elf->elf, sec->idx); - if (!s) { - WARN_ELF("elf_getscn"); - return -1; - } - - sec->data =3D elf_newdata(s); - if (!sec->data) { - WARN_ELF("elf_newdata"); - return -1; - } - - sec->data->d_size =3D size; - sec->data->d_align =3D 1; - - alt =3D sec->data->d_buf =3D malloc(size); - if (!sec->data->d_buf) { - perror("malloc"); - return -1; - } - memset(sec->data->d_buf, 0, size); - - if (elf_add_reloc_to_insn(elf, sec, sec->sh.sh_size, - R_X86_64_PC32, orig->sec, orig->offset)) { - WARN("elf_create_reloc: alt_instr::instr_offset"); - return -1; - } - - if (elf_add_reloc(elf, sec, sec->sh.sh_size + 4, - R_X86_64_PC32, sym, 0)) { - WARN("elf_create_reloc: alt_instr::repl_offset"); - return -1; - } - - alt->cpuid =3D cpuid; - alt->instrlen =3D orig_len; - alt->replacementlen =3D repl_len; - - sec->sh.sh_size +=3D size; - sec->changed =3D true; - - return 0; -} - -#define X86_FEATURE_RETPOLINE ( 7*32+12) - -int arch_rewrite_retpolines(struct objtool_file *file) -{ - struct instruction *insn; - struct reloc *reloc; - struct symbol *sym; - char name[32] =3D ""; - - list_for_each_entry(insn, &file->retpoline_call_list, call_node) { - - if (insn->type !=3D INSN_JUMP_DYNAMIC && - insn->type !=3D INSN_CALL_DYNAMIC) - continue; - - if (!strcmp(insn->sec->name, ".text.__x86.indirect_thunk")) - continue; - - reloc =3D insn->reloc; - - sprintf(name, "__x86_indirect_alt_%s_%s", - insn->type =3D=3D INSN_JUMP_DYNAMIC ? "jmp" : "call", - reloc->sym->name + 21); - - sym =3D find_symbol_by_name(file->elf, name); - if (!sym) { - sym =3D elf_create_undef_symbol(file->elf, name); - if (!sym) { - WARN("elf_create_undef_symbol"); - return -1; - } - } - - if (elf_add_alternative(file->elf, insn, sym, - ALT_NOT(X86_FEATURE_RETPOLINE), 5, 5)) { - WARN("elf_add_alternative"); - return -1; - } - } - - return 0; -} - int arch_decode_hint_reg(u8 sp_reg, int *base) { switch (sp_reg) { --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -606,6 +606,52 @@ static int create_static_call_sections(s return 0; } =20 +static int create_retpoline_sites_sections(struct objtool_file *file) +{ + struct instruction *insn; + struct section *sec; + int idx; + + sec =3D find_section_by_name(file->elf, ".retpoline_sites"); + if (sec) { + WARN("file already has .retpoline_sites, skipping"); + return 0; + } + + idx =3D 0; + list_for_each_entry(insn, &file->retpoline_call_list, call_node) + idx++; + + if (!idx) + return 0; + + sec =3D elf_create_section(file->elf, ".retpoline_sites", 0, + sizeof(int), idx); + if (!sec) { + WARN("elf_create_section: .retpoline_sites"); + return -1; + } + + idx =3D 0; + list_for_each_entry(insn, &file->retpoline_call_list, call_node) { + + int *site =3D (int *)sec->data->d_buf + idx; + *site =3D 0; + + if (elf_add_reloc_to_insn(file->elf, sec, + idx * sizeof(int), + R_X86_64_PC32, + insn->sec, insn->offset)) { + WARN("elf_add_reloc_to_insn: .retpoline_sites"); + return -1; + } + + idx++; + } + + return 0; +} + /* * Warnings shouldn't be reported for ignored functions. */ @@ -893,6 +939,11 @@ static void annotate_call_site(struct ob return; } =20 + if (sym->retpoline_thunk) { + list_add_tail(&insn->call_node, &file->retpoline_call_list); + return; + } + /* * Many compilers cannot disable KCOV with a function attribute * so they need a little help, NOP out any KCOV calls from noinstr @@ -933,6 +984,39 @@ static void add_call_dest(struct objtool annotate_call_site(file, insn, sibling); } =20 +static void add_retpoline_call(struct objtool_file *file, struct instructi= on *insn) +{ + /* + * Retpoline calls/jumps are really dynamic calls/jumps in disguise, + * so convert them accordingly. + */ + switch (insn->type) { + case INSN_CALL: + insn->type =3D INSN_CALL_DYNAMIC; + break; + case INSN_JUMP_UNCONDITIONAL: + insn->type =3D INSN_JUMP_DYNAMIC; + break; + case INSN_JUMP_CONDITIONAL: + insn->type =3D INSN_JUMP_DYNAMIC_CONDITIONAL; + break; + default: + return; + } + + insn->retpoline_safe =3D true; + + /* + * Whatever stack impact regular CALLs have, should be undone + * by the RETURN of the called function. + * + * Annotated intra-function calls retain the stack_ops but + * are converted to JUMP, see read_intra_function_calls(). + */ + remove_insn_ops(insn); + + annotate_call_site(file, insn, false); +} /* * Find the destination instructions for all jumps. */ @@ -955,19 +1039,7 @@ static int add_jump_destinations(struct dest_sec =3D reloc->sym->sec; dest_off =3D arch_dest_reloc_offset(reloc->addend); } else if (reloc->sym->retpoline_thunk) { - /* - * Retpoline jumps are really dynamic jumps in - * disguise, so convert them accordingly. - */ - if (insn->type =3D=3D INSN_JUMP_UNCONDITIONAL) - insn->type =3D INSN_JUMP_DYNAMIC; - else - insn->type =3D INSN_JUMP_DYNAMIC_CONDITIONAL; - - list_add_tail(&insn->call_node, - &file->retpoline_call_list); - - insn->retpoline_safe =3D true; + add_retpoline_call(file, insn); continue; } else if (insn->func) { /* internal or external sibling call (with reloc) */ @@ -1096,18 +1168,7 @@ static int add_call_destinations(struct add_call_dest(file, insn, dest, false); =20 } else if (reloc->sym->retpoline_thunk) { - /* - * Retpoline calls are really dynamic calls in - * disguise, so convert them accordingly. - */ - insn->type =3D INSN_CALL_DYNAMIC; - insn->retpoline_safe =3D true; - - list_add_tail(&insn->call_node, - &file->retpoline_call_list); - - remove_insn_ops(insn); - continue; + add_retpoline_call(file, insn); =20 } else add_call_dest(file, insn, reloc->sym, false); @@ -1806,11 +1867,6 @@ static void mark_rodata(struct objtool_f file->rodata =3D found; } =20 -__weak int arch_rewrite_retpolines(struct objtool_file *file) -{ - return 0; -} - static int decode_sections(struct objtool_file *file) { int ret; @@ -1879,15 +1935,6 @@ static int decode_sections(struct objtoo if (ret) return ret; =20 - /* - * Must be after add_special_section_alts(), since this will emit - * alternatives. Must be after add_{jump,call}_destination(), since - * those create the call insn lists. - */ - ret =3D arch_rewrite_retpolines(file); - if (ret) - return ret; - return 0; } =20 @@ -3159,6 +3206,13 @@ int check(struct objtool_file *file) goto out; warnings +=3D ret; =20 + if (retpoline) { + ret =3D create_retpoline_sites_sections(file); + if (ret < 0) + goto out; + warnings +=3D ret; + } + if (stats) { printf("nr_insns_visited: %ld\n", nr_insns_visited); printf("nr_cfi: %ld\n", nr_cfi); --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -766,89 +766,6 @@ static int elf_add_string(struct elf *el return len; } =20 -struct symbol *elf_create_undef_symbol(struct elf *elf, const char *name) -{ - struct section *symtab, *symtab_shndx; - struct symbol *sym; - Elf_Data *data; - Elf_Scn *s; - - sym =3D malloc(sizeof(*sym)); - if (!sym) { - perror("malloc"); - return NULL; - } - memset(sym, 0, sizeof(*sym)); - - sym->name =3D strdup(name); - - sym->sym.st_name =3D elf_add_string(elf, NULL, sym->name); - if (sym->sym.st_name =3D=3D -1) - return NULL; - - sym->sym.st_info =3D GELF_ST_INFO(STB_GLOBAL, STT_NOTYPE); - // st_other 0 - // st_shndx 0 - // st_value 0 - // st_size 0 - - symtab =3D find_section_by_name(elf, ".symtab"); - if (!symtab) { - WARN("can't find .symtab"); - return NULL; - } - - s =3D elf_getscn(elf->elf, symtab->idx); - if (!s) { - WARN_ELF("elf_getscn"); - return NULL; - } - - data =3D elf_newdata(s); - if (!data) { - WARN_ELF("elf_newdata"); - return NULL; - } - - data->d_buf =3D &sym->sym; - data->d_size =3D sizeof(sym->sym); - data->d_align =3D 1; - - sym->idx =3D symtab->len / sizeof(sym->sym); - - symtab->len +=3D data->d_size; - symtab->changed =3D true; - - symtab_shndx =3D find_section_by_name(elf, ".symtab_shndx"); - if (symtab_shndx) { - s =3D elf_getscn(elf->elf, symtab_shndx->idx); - if (!s) { - WARN_ELF("elf_getscn"); - return NULL; - } - - data =3D elf_newdata(s); - if (!data) { - WARN_ELF("elf_newdata"); - return NULL; - } - - data->d_buf =3D &sym->sym.st_size; /* conveniently 0 */ - data->d_size =3D sizeof(Elf32_Word); - data->d_align =3D 4; - data->d_type =3D ELF_T_WORD; - - symtab_shndx->len +=3D 4; - symtab_shndx->changed =3D true; - } - - sym->sec =3D find_section_by_index(elf, 0); - - elf_add_symbol(elf, sym); - - return sym; -} - struct section *elf_create_section(struct elf *elf, const char *name, unsigned int sh_flags, size_t entsize, int nr) { --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -136,7 +136,6 @@ int elf_write_insn(struct elf *elf, stru unsigned long offset, unsigned int len, const char *insn); int elf_write_reloc(struct elf *elf, struct reloc *reloc); -struct symbol *elf_create_undef_symbol(struct elf *elf, const char *name); int elf_write(struct elf *elf); void elf_close(struct elf *elf); =20 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -105,14 +105,6 @@ static int get_alt_entry(struct elf *elf return -1; } =20 - /* - * Skip retpoline .altinstr_replacement... we already rewrite the - * instructions for retpolines anyway, see arch_is_retpoline() - * usage in add_{call,jump}_destinations(). - */ - if (arch_is_retpoline(new_reloc->sym)) - return 1; - reloc_to_sec_off(new_reloc, &alt->new_sec, &alt->new_off); =20 /* _ASM_EXTABLE_EX hack */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B669C43334 for ; Sat, 23 Jul 2022 10:02:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237830AbiGWKCp (ORCPT ); Sat, 23 Jul 2022 06:02:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237994AbiGWKCQ (ORCPT ); Sat, 23 Jul 2022 06:02:16 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1829804B4; Sat, 23 Jul 2022 02:58:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 24B32611D2; Sat, 23 Jul 2022 09:58:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F6AEC341C0; Sat, 23 Jul 2022 09:58:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570332; bh=nHVd6Mwvbhq8aggXi24Iyi/JP3pj3Tu3p2pzPUSotaM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OCQ43XIXUSmwzBlyHD/Sb+zFlHi/Y3l2kxYRPChqOX/1ydMMvmdUY6PrTtPsNwF/d DzQnQ9IxdGGaTf14i5WsSD0ZkGoFSWxyHw+qKmOeHBkuzoCFK59xEfUr8ki1ABNueH Uu98jYqeVF+htNh+Ac/6Kmk+wSiSo85C+rt9KkD8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 050/148] x86/retpoline: Remove unused replacement symbols Date: Sat, 23 Jul 2022 11:54:22 +0200 Message-Id: <20220723095238.332171280@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 4fe79e710d9574a14993f8b4e16b7252da72d5e8 upstream. Now that objtool no longer creates alternatives, these replacement symbols are no longer needed, remove them. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120309.915051744@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/asm-prototypes.h | 10 -------- arch/x86/lib/retpoline.S | 42 -----------------------------= ----- 2 files changed, 52 deletions(-) --- a/arch/x86/include/asm/asm-prototypes.h +++ b/arch/x86/include/asm/asm-prototypes.h @@ -24,14 +24,4 @@ extern void cmpxchg8b_emu(void); extern asmlinkage void __x86_indirect_thunk_ ## reg (void); #include =20 -#undef GEN -#define GEN(reg) \ - extern asmlinkage void __x86_indirect_alt_call_ ## reg (void); -#include - -#undef GEN -#define GEN(reg) \ - extern asmlinkage void __x86_indirect_alt_jmp_ ## reg (void); -#include - #endif /* CONFIG_RETPOLINE */ --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -41,36 +41,6 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg) .endm =20 /* - * This generates .altinstr_replacement symbols for use by objtool. They, - * however, must not actually live in .altinstr_replacement since that wil= l be - * discarded after init, but module alternatives will also reference these - * symbols. - * - * Their names matches the "__x86_indirect_" prefix to mark them as retpol= ines. - */ -.macro ALT_THUNK reg - - .align 1 - -SYM_FUNC_START_NOALIGN(__x86_indirect_alt_call_\reg) - ANNOTATE_RETPOLINE_SAFE -1: call *%\reg -2: .skip 5-(2b-1b), 0x90 -SYM_FUNC_END(__x86_indirect_alt_call_\reg) - -STACK_FRAME_NON_STANDARD(__x86_indirect_alt_call_\reg) - -SYM_FUNC_START_NOALIGN(__x86_indirect_alt_jmp_\reg) - ANNOTATE_RETPOLINE_SAFE -1: jmp *%\reg -2: .skip 5-(2b-1b), 0x90 -SYM_FUNC_END(__x86_indirect_alt_jmp_\reg) - -STACK_FRAME_NON_STANDARD(__x86_indirect_alt_jmp_\reg) - -.endm - -/* * Despite being an assembler file we can't just use .irp here * because __KSYM_DEPS__ only uses the C preprocessor and would * only see one instance of "__x86_indirect_thunk_\reg" rather @@ -92,15 +62,3 @@ STACK_FRAME_NON_STANDARD(__x86_indirect_ #undef GEN #define GEN(reg) EXPORT_THUNK(reg) #include - -#undef GEN -#define GEN(reg) ALT_THUNK reg -#include - -#undef GEN -#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_call_ ## reg) -#include - -#undef GEN -#define GEN(reg) __EXPORT_THUNK(__x86_indirect_alt_jmp_ ## reg) -#include From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FEC7C433EF for ; Sat, 23 Jul 2022 10:02:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237939AbiGWKCv (ORCPT ); Sat, 23 Jul 2022 06:02:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238008AbiGWKCU (ORCPT ); Sat, 23 Jul 2022 06:02:20 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9AEF63934; Sat, 23 Jul 2022 02:59:00 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9FB74B82C1B; Sat, 23 Jul 2022 09:58:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 137F7C341C7; Sat, 23 Jul 2022 09:58:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570335; bh=l0kO0QEUe8EFVtCdORKJzuiIYU1+sCDa53haTUQKWpE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lVU4/voGGbauQAsJog3B0PCpujVacYyFvP2DUX5FKLnZYHFXyvi5yjV14Y60ovKe3 yMciUafkDHsnRxVq0XNEvc19s43lxBvhaY3ThXF6gPo7Ah9HewDeXuQgTS4jIGcgcb oxY/Dt/nNCV/Hg1N7/ihx9IRRR0EdyXwyyHVMtr4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 051/148] x86/asm: Fix register order Date: Sat, 23 Jul 2022 11:54:23 +0200 Message-Id: <20220723095238.624255292@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit a92ede2d584a2e070def59c7e47e6b6f6341c55c upstream. Ensure the register order is correct; this allows for easy translation between register number and trampoline and vice-versa. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120309.978573921@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/GEN-for-each-reg.h | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) --- a/arch/x86/include/asm/GEN-for-each-reg.h +++ b/arch/x86/include/asm/GEN-for-each-reg.h @@ -1,11 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * These are in machine order; things rely on that. + */ #ifdef CONFIG_64BIT GEN(rax) -GEN(rbx) GEN(rcx) GEN(rdx) +GEN(rbx) +GEN(rsp) +GEN(rbp) GEN(rsi) GEN(rdi) -GEN(rbp) GEN(r8) GEN(r9) GEN(r10) @@ -16,10 +21,11 @@ GEN(r14) GEN(r15) #else GEN(eax) -GEN(ebx) GEN(ecx) GEN(edx) +GEN(ebx) +GEN(esp) +GEN(ebp) GEN(esi) GEN(edi) -GEN(ebp) #endif From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 683E9C43334 for ; Sat, 23 Jul 2022 10:03:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238165AbiGWKDJ (ORCPT ); Sat, 23 Jul 2022 06:03:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237801AbiGWKCi (ORCPT ); Sat, 23 Jul 2022 06:02:38 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3653A6717F; Sat, 23 Jul 2022 02:59:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E7010611D4; Sat, 23 Jul 2022 09:58:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01EB6C36AE7; Sat, 23 Jul 2022 09:58:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570338; bh=QLUaL63/ytWAUClgmAMMnqPtqDpfAEvNGe4wTT9p+hI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g3NGkrIrSDrJW+OdygsPvqbLkrkIO3jOcCwOMfF3FCRWQwPiDWZiafFDLkKLwjV/1 9SEwq5gVtH8wnprfgFsIMqibHNDquqez8Y9S9cB1PayV7HjTjKxXcNgMzyQdjlwCIL BzlBnRj9OECxAJPDC2vDiwBtOLo4BrkE3MnTInzw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 052/148] x86/asm: Fixup odd GEN-for-each-reg.h usage Date: Sat, 23 Jul 2022 11:54:24 +0200 Message-Id: <20220723095238.874709180@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit b6d3d9944bd7c9e8c06994ead3c9952f673f2a66 upstream. Currently GEN-for-each-reg.h usage leaves GEN defined, relying on any subsequent usage to start with #undef, which is rude. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.041792350@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/asm-prototypes.h | 2 +- arch/x86/lib/retpoline.S | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) --- a/arch/x86/include/asm/asm-prototypes.h +++ b/arch/x86/include/asm/asm-prototypes.h @@ -19,9 +19,9 @@ extern void cmpxchg8b_emu(void); =20 #ifdef CONFIG_RETPOLINE =20 -#undef GEN #define GEN(reg) \ extern asmlinkage void __x86_indirect_thunk_ ## reg (void); #include +#undef GEN =20 #endif /* CONFIG_RETPOLINE */ --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -55,10 +55,10 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg) #define __EXPORT_THUNK(sym) _ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym) #define EXPORT_THUNK(reg) __EXPORT_THUNK(__x86_indirect_thunk_ ## reg) =20 -#undef GEN #define GEN(reg) THUNK reg #include - #undef GEN + #define GEN(reg) EXPORT_THUNK(reg) #include +#undef GEN From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49E36C43334 for ; Sat, 23 Jul 2022 10:02:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238137AbiGWKCz (ORCPT ); Sat, 23 Jul 2022 06:02:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238054AbiGWKCW (ORCPT ); Sat, 23 Jul 2022 06:02:22 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5969181B3E; Sat, 23 Jul 2022 02:59:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B614B61240; Sat, 23 Jul 2022 09:59:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC5B6C341C0; Sat, 23 Jul 2022 09:59:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570341; bh=UJTwM9IbOrqoHdkWkc3PKEHG1lSMjA/Rmh5gdrCnikc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wF2IMf2TWyol6RkRGe4cOhVYFBanpc0LGnyJ5cGYnrGu2y2e0VhFArS6oP1WIX4FY kT/FKUohAG+VBd0nJWJPNQlX+q94YV7RGN4TMV7fgq6l2OOx9UfA/93Qd88mq12Zro WWtniH9eEjDE17n0LQcCiXb4sMbs51L+bsk6JUGI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 053/148] x86/retpoline: Move the retpoline thunk declarations to nospec-branch.h Date: Sat, 23 Jul 2022 11:54:25 +0200 Message-Id: <20220723095239.143471969@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 6fda8a38865607db739be3e567a2387376222dbd upstream. Because it makes no sense to split the retpoline gunk over multiple headers. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.106290934@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/asm-prototypes.h | 8 -------- arch/x86/include/asm/nospec-branch.h | 7 +++++++ arch/x86/net/bpf_jit_comp.c | 1 - 3 files changed, 7 insertions(+), 9 deletions(-) --- a/arch/x86/include/asm/asm-prototypes.h +++ b/arch/x86/include/asm/asm-prototypes.h @@ -17,11 +17,3 @@ extern void cmpxchg8b_emu(void); #endif =20 -#ifdef CONFIG_RETPOLINE - -#define GEN(reg) \ - extern asmlinkage void __x86_indirect_thunk_ ## reg (void); -#include -#undef GEN - -#endif /* CONFIG_RETPOLINE */ --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -5,6 +5,7 @@ =20 #include #include +#include =20 #include #include @@ -118,6 +119,12 @@ ".popsection\n\t" =20 #ifdef CONFIG_RETPOLINE + +#define GEN(reg) \ + extern asmlinkage void __x86_indirect_thunk_ ## reg (void); +#include +#undef GEN + #ifdef CONFIG_X86_64 =20 /* --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -15,7 +15,6 @@ #include #include #include -#include =20 static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B0FCC433EF for ; Sat, 23 Jul 2022 10:03:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238152AbiGWKDD (ORCPT ); Sat, 23 Jul 2022 06:03:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237958AbiGWKCg (ORCPT ); Sat, 23 Jul 2022 06:02:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD43382FAA; Sat, 23 Jul 2022 02:59:05 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9AFF76125D; Sat, 23 Jul 2022 09:59:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A9578C341C0; Sat, 23 Jul 2022 09:59:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570344; bh=D6Pxm5LuSsNkFmaR1etmsshRJ2ZUFrmrGMIFvboJej0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E2EJhavHrsh1X4XVZYT+dekFr9aI6wpXP4F4agk6C7Zf4edwIXKvrmSO+4qoFp3md u1QA6xha4pS1l4WypdztQKazTFXBSV9XDeP5ggu7s9qoKynbz8QMYmF/qoPOtCuZgt 937/OIY11YBBPwTMfKUVCobL2wmw9s7EPjGzHWc8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 054/148] x86/retpoline: Create a retpoline thunk array Date: Sat, 23 Jul 2022 11:54:26 +0200 Message-Id: <20220723095239.410889364@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 1a6f74429c42a3854980359a758e222005712aee upstream. Stick all the retpolines in a single symbol and have the individual thunks as inner labels, this should guarantee thunk order and layout. Previously there were 16 (or rather 15 without rsp) separate symbols and a toolchain might reasonably expect it could displace them however it liked, with disregard for their relative position. However, now they're part of a larger symbol. Any change to their relative position would disrupt this larger _array symbol and thus not be sound. This is the same reasoning used for data symbols. On their own there is no guarantee about their relative position wrt to one aonther, but we're still able to do arrays because an array as a whole is a single larger symbol. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.169659320@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 8 +++++++- arch/x86/lib/retpoline.S | 14 +++++++++----- 2 files changed, 16 insertions(+), 6 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -12,6 +12,8 @@ #include #include =20 +#define RETPOLINE_THUNK_SIZE 32 + /* * Fill the CPU return stack buffer. * @@ -120,11 +122,15 @@ =20 #ifdef CONFIG_RETPOLINE =20 +typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE]; + #define GEN(reg) \ - extern asmlinkage void __x86_indirect_thunk_ ## reg (void); + extern retpoline_thunk_t __x86_indirect_thunk_ ## reg; #include #undef GEN =20 +extern retpoline_thunk_t __x86_indirect_thunk_array[]; + #ifdef CONFIG_X86_64 =20 /* --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -28,16 +28,14 @@ =20 .macro THUNK reg =20 - .align 32 - -SYM_FUNC_START(__x86_indirect_thunk_\reg) + .align RETPOLINE_THUNK_SIZE +SYM_INNER_LABEL(__x86_indirect_thunk_\reg, SYM_L_GLOBAL) + UNWIND_HINT_EMPTY =20 ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ __stringify(RETPOLINE \reg), X86_FEATURE_RETPOLINE, \ __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), X86_FEAT= URE_RETPOLINE_LFENCE =20 -SYM_FUNC_END(__x86_indirect_thunk_\reg) - .endm =20 /* @@ -55,10 +53,16 @@ SYM_FUNC_END(__x86_indirect_thunk_\reg) #define __EXPORT_THUNK(sym) _ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym) #define EXPORT_THUNK(reg) __EXPORT_THUNK(__x86_indirect_thunk_ ## reg) =20 + .align RETPOLINE_THUNK_SIZE +SYM_CODE_START(__x86_indirect_thunk_array) + #define GEN(reg) THUNK reg #include #undef GEN =20 + .align RETPOLINE_THUNK_SIZE +SYM_CODE_END(__x86_indirect_thunk_array) + #define GEN(reg) EXPORT_THUNK(reg) #include #undef GEN From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95FECC433EF for ; Sat, 23 Jul 2022 10:03:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237945AbiGWKDX (ORCPT ); Sat, 23 Jul 2022 06:03:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238006AbiGWKCm (ORCPT ); Sat, 23 Jul 2022 06:02:42 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F0046872B; Sat, 23 Jul 2022 02:59:16 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 72A58611CD; Sat, 23 Jul 2022 09:59:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7B01CC341C0; Sat, 23 Jul 2022 09:59:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570346; bh=3BRqObZWF3vDFEP1t/Bx7zNVLrdMn1bUApJzeKpzwVg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fFH4XiacWaHVcXG+5FJaHFnLXgs83wCBTdIY77yPcMmWcVcjDDXEKvT2X0uSezo6r b1QXipFaOogTVpHahk0zr6uTKtHytesBQRNa8/yxDPe3Z8PwI5Gg9ObBjQlwwWN/yk HSpAFceRKah6fP68Py266uYrnK0+rNkIk14gy2G4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 055/148] x86/alternative: Implement .retpoline_sites support Date: Sat, 23 Jul 2022 11:54:27 +0200 Message-Id: <20220723095239.719793913@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 7508500900814d14e2e085cdc4e28142721abbdf upstream. Rewrite retpoline thunk call sites to be indirect calls for spectre_v2=3Doff. This ensures spectre_v2=3Doff is as near to a RETPOLINE=3Dn build as possible. This is the replacement for objtool writing alternative entries to ensure the same and achieves feature-parity with the previous approach. One noteworthy feature is that it relies on the thunks to be in machine order to compute the register index. Specifically, this does not yet address the Jcc __x86_indirect_thunk_* calls generated by clang, a future patch will add this. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.232495794@infradead.org [cascardo: small conflict fixup at arch/x86/kernel/module.c] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: - Use hex literal instead of BYTES_NOP1 - Adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/um/kernel/um_arch.c | 4 + arch/x86/include/asm/alternative.h | 1=20 arch/x86/kernel/alternative.c | 141 ++++++++++++++++++++++++++++++++= +++-- arch/x86/kernel/module.c | 9 ++ 4 files changed, 150 insertions(+), 5 deletions(-) --- a/arch/um/kernel/um_arch.c +++ b/arch/um/kernel/um_arch.c @@ -358,6 +358,10 @@ void __init check_bugs(void) os_check_bugs(); } =20 +void apply_retpolines(s32 *start, s32 *end) +{ +} + void apply_alternatives(struct alt_instr *start, struct alt_instr *end) { } --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -75,6 +75,7 @@ extern int alternatives_patched; =20 extern void alternative_instructions(void); extern void apply_alternatives(struct alt_instr *start, struct alt_instr *= end); +extern void apply_retpolines(s32 *start, s32 *end); =20 struct module; =20 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -28,6 +28,7 @@ #include #include #include +#include =20 int __read_mostly alternatives_patched; =20 @@ -268,6 +269,7 @@ static void __init_or_module add_nops(vo } } =20 +extern s32 __retpoline_sites[], __retpoline_sites_end[]; extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; void text_poke_early(void *addr, const void *opcode, size_t len); @@ -376,7 +378,7 @@ static __always_inline int optimize_nops * "noinline" to cause control flow change and thus invalidate I$ and * cause refetch after modification. */ -static void __init_or_module noinline optimize_nops(struct alt_instr *a, u= 8 *instr) +static void __init_or_module noinline optimize_nops(u8 *instr, size_t len) { struct insn insn; int i =3D 0; @@ -394,11 +396,11 @@ static void __init_or_module noinline op * optimized. */ if (insn.length =3D=3D 1 && insn.opcode.bytes[0] =3D=3D 0x90) - i +=3D optimize_nops_range(instr, a->instrlen, i); + i +=3D optimize_nops_range(instr, len, i); else i +=3D insn.length; =20 - if (i >=3D a->instrlen) + if (i >=3D len) return; } } @@ -486,10 +488,135 @@ void __init_or_module noinline apply_alt text_poke_early(instr, insn_buff, insn_buff_sz); =20 next: - optimize_nops(a, instr); + optimize_nops(instr, a->instrlen); } } =20 +#if defined(CONFIG_RETPOLINE) && defined(CONFIG_STACK_VALIDATION) + +/* + * CALL/JMP *%\reg + */ +static int emit_indirect(int op, int reg, u8 *bytes) +{ + int i =3D 0; + u8 modrm; + + switch (op) { + case CALL_INSN_OPCODE: + modrm =3D 0x10; /* Reg =3D 2; CALL r/m */ + break; + + case JMP32_INSN_OPCODE: + modrm =3D 0x20; /* Reg =3D 4; JMP r/m */ + break; + + default: + WARN_ON_ONCE(1); + return -1; + } + + if (reg >=3D 8) { + bytes[i++] =3D 0x41; /* REX.B prefix */ + reg -=3D 8; + } + + modrm |=3D 0xc0; /* Mod =3D 3 */ + modrm +=3D reg; + + bytes[i++] =3D 0xff; /* opcode */ + bytes[i++] =3D modrm; + + return i; +} + +/* + * Rewrite the compiler generated retpoline thunk calls. + * + * For spectre_v2=3Doff (!X86_FEATURE_RETPOLINE), rewrite them into immedi= ate + * indirect instructions, avoiding the extra indirection. + * + * For example, convert: + * + * CALL __x86_indirect_thunk_\reg + * + * into: + * + * CALL *%\reg + * + */ +static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes) +{ + retpoline_thunk_t *target; + int reg, i =3D 0; + + target =3D addr + insn->length + insn->immediate.value; + reg =3D target - __x86_indirect_thunk_array; + + if (WARN_ON_ONCE(reg & ~0xf)) + return -1; + + /* If anyone ever does: CALL/JMP *%rsp, we're in deep trouble. */ + BUG_ON(reg =3D=3D 4); + + if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) + return -1; + + i =3D emit_indirect(insn->opcode.bytes[0], reg, bytes); + if (i < 0) + return i; + + for (; i < insn->length;) + bytes[i++] =3D 0x90; + + return i; +} + +/* + * Generated by 'objtool --retpoline'. + */ +void __init_or_module noinline apply_retpolines(s32 *start, s32 *end) +{ + s32 *s; + + for (s =3D start; s < end; s++) { + void *addr =3D (void *)s + *s; + struct insn insn; + int len, ret; + u8 bytes[16]; + u8 op1, op2; + + ret =3D insn_decode_kernel(&insn, addr); + if (WARN_ON_ONCE(ret < 0)) + continue; + + op1 =3D insn.opcode.bytes[0]; + op2 =3D insn.opcode.bytes[1]; + + switch (op1) { + case CALL_INSN_OPCODE: + case JMP32_INSN_OPCODE: + break; + + default: + WARN_ON_ONCE(1); + continue; + } + + len =3D patch_retpoline(addr, &insn, bytes); + if (len =3D=3D insn.length) { + optimize_nops(bytes, len); + text_poke_early(addr, bytes, len); + } + } +} + +#else /* !RETPOLINES || !CONFIG_STACK_VALIDATION */ + +void __init_or_module noinline apply_retpolines(s32 *start, s32 *end) { } + +#endif /* CONFIG_RETPOLINE && CONFIG_STACK_VALIDATION */ + #ifdef CONFIG_SMP static void alternatives_smp_lock(const s32 *start, const s32 *end, u8 *text, u8 *text_end) @@ -774,6 +901,12 @@ void __init alternative_instructions(voi * patching. */ =20 + /* + * Rewrite the retpolines, must be done before alternatives since + * those can rewrite the retpoline thunks. + */ + apply_retpolines(__retpoline_sites, __retpoline_sites_end); + apply_alternatives(__alt_instructions, __alt_instructions_end); =20 #ifdef CONFIG_SMP --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -251,7 +251,8 @@ int module_finalize(const Elf_Ehdr *hdr, struct module *me) { const Elf_Shdr *s, *text =3D NULL, *alt =3D NULL, *locks =3D NULL, - *para =3D NULL, *orc =3D NULL, *orc_ip =3D NULL; + *para =3D NULL, *orc =3D NULL, *orc_ip =3D NULL, + *retpolines =3D NULL; char *secstrings =3D (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; =20 for (s =3D sechdrs; s < sechdrs + hdr->e_shnum; s++) { @@ -267,8 +268,14 @@ int module_finalize(const Elf_Ehdr *hdr, orc =3D s; if (!strcmp(".orc_unwind_ip", secstrings + s->sh_name)) orc_ip =3D s; + if (!strcmp(".retpoline_sites", secstrings + s->sh_name)) + retpolines =3D s; } =20 + if (retpolines) { + void *rseg =3D (void *)retpolines->sh_addr; + apply_retpolines(rseg, rseg + retpolines->sh_size); + } if (alt) { /* patch .altinstructions */ void *aseg =3D (void *)alt->sh_addr; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A9DDC43334 for ; Sat, 23 Jul 2022 10:03:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238212AbiGWKDc (ORCPT ); Sat, 23 Jul 2022 06:03:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237872AbiGWKDC (ORCPT ); Sat, 23 Jul 2022 06:03:02 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54A1868DC8; Sat, 23 Jul 2022 02:59:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 0F5B7B82C1A; Sat, 23 Jul 2022 09:59:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 688F3C341C0; Sat, 23 Jul 2022 09:59:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570349; bh=7U7iZoqe4+dtDPPJp4LSzYXjPFIQ+BxwPO0m8HF420E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E+IoxH5m5EwraZAcSHtxAIAdOkGMuO1M04bacNnNAA1b6QzEaqZP1tJ3uj6l9JTuS El9JPhBrQXE3kX/fKnJj23wcdPnnvifediS9UOKmBxKZ98mtHOAkLrUj+CZoxsCEYd 2ybrtZ+lIt6h9dwQ83OrRXWvDv8wMLdikYGzrY7M= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 056/148] x86/alternative: Handle Jcc __x86_indirect_thunk_\reg Date: Sat, 23 Jul 2022 11:54:28 +0200 Message-Id: <20220723095239.974816312@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 2f0cbb2a8e5bbf101e9de118fc0eb168111a5e1e upstream. Handle the rare cases where the compiler (clang) does an indirect conditional tail-call using: Jcc __x86_indirect_thunk_\reg For the !RETPOLINE case this can be rewritten to fit the original (6 byte) instruction like: Jncc.d8 1f JMP *%\reg NOP 1: Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.296470217@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/alternative.c | 40 ++++++++++++++++++++++++++++++++++++-= --- 1 file changed, 36 insertions(+), 4 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -548,7 +548,8 @@ static int emit_indirect(int op, int reg static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes) { retpoline_thunk_t *target; - int reg, i =3D 0; + int reg, ret, i =3D 0; + u8 op, cc; =20 target =3D addr + insn->length + insn->immediate.value; reg =3D target - __x86_indirect_thunk_array; @@ -562,9 +563,36 @@ static int patch_retpoline(void *addr, s if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) return -1; =20 - i =3D emit_indirect(insn->opcode.bytes[0], reg, bytes); - if (i < 0) - return i; + op =3D insn->opcode.bytes[0]; + + /* + * Convert: + * + * Jcc.d32 __x86_indirect_thunk_\reg + * + * into: + * + * Jncc.d8 1f + * JMP *%\reg + * NOP + * 1: + */ + /* Jcc.d32 second opcode byte is in the range: 0x80-0x8f */ + if (op =3D=3D 0x0f && (insn->opcode.bytes[1] & 0xf0) =3D=3D 0x80) { + cc =3D insn->opcode.bytes[1] & 0xf; + cc ^=3D 1; /* invert condition */ + + bytes[i++] =3D 0x70 + cc; /* Jcc.d8 */ + bytes[i++] =3D insn->length - 2; /* sizeof(Jcc.d8) =3D=3D 2 */ + + /* Continue as if: JMP.d32 __x86_indirect_thunk_\reg */ + op =3D JMP32_INSN_OPCODE; + } + + ret =3D emit_indirect(op, reg, bytes + i); + if (ret < 0) + return ret; + i +=3D ret; =20 for (; i < insn->length;) bytes[i++] =3D 0x90; @@ -598,6 +626,10 @@ void __init_or_module noinline apply_ret case JMP32_INSN_OPCODE: break; =20 + case 0x0f: /* escape */ + if (op2 >=3D 0x80 && op2 <=3D 0x8f) + break; + fallthrough; default: WARN_ON_ONCE(1); continue; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62290C433EF for ; Sat, 23 Jul 2022 10:03:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238092AbiGWKDT (ORCPT ); Sat, 23 Jul 2022 06:03:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237746AbiGWKCk (ORCPT ); Sat, 23 Jul 2022 06:02:40 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA83B3C8D7; Sat, 23 Jul 2022 02:59:13 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 31275611BD; Sat, 23 Jul 2022 09:59:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D495C341C0; Sat, 23 Jul 2022 09:59:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570352; bh=OqfwUX60UBlfoRQ1m3NNXPSM4sIy5u0tsQRZI9rcZ6c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=raggF41zeXAY5aBCegC/3QrHiBfZflCpT6bVEWWEPu9r9f6ZO2WUW1dZayABIJ9iT tO/E+d1+W4Wo+odK755rTW4tz+37i/SZRyDk3HGNaRg/h+XN7exVK29FvxcJoiqYVN KlskujkLvr5K8JfqpXKA7ARVBmtiaUYp0Btp25pc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 057/148] x86/alternative: Try inline spectre_v2=retpoline,amd Date: Sat, 23 Jul 2022 11:54:29 +0200 Message-Id: <20220723095240.221456201@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit bbe2df3f6b6da7848398d55b1311d58a16ec21e4 upstream. Try and replace retpoline thunk calls with: LFENCE CALL *%\reg for spectre_v2=3Dretpoline,amd. Specifically, the sequence above is 5 bytes for the low 8 registers, but 6 bytes for the high 8 registers. This means that unless the compilers prefix stuff the call with higher registers this replacement will fail. Luckily GCC strongly favours RAX for the indirect calls and most (95%+ for defconfig-x86_64) will be converted. OTOH clang strongly favours R11 and almost nothing gets converted. Note: it will also generate a correct replacement for the Jcc.d32 case, except unless the compilers start to prefix stuff that, it'll never fit. Specifically: Jncc.d8 1f LFENCE JMP *%\reg 1: is 7-8 bytes long, where the original instruction in unpadded form is only 6 bytes. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.359986601@infradead.org [cascardo: RETPOLINE_AMD was renamed to RETPOLINE_LFENCE] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/alternative.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -544,6 +544,7 @@ static int emit_indirect(int op, int reg * * CALL *%\reg * + * It also tries to inline spectre_v2=3Dretpoline,amd when size permits. */ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes) { @@ -560,7 +561,8 @@ static int patch_retpoline(void *addr, s /* If anyone ever does: CALL/JMP *%rsp, we're in deep trouble. */ BUG_ON(reg =3D=3D 4); =20 - if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) + if (cpu_feature_enabled(X86_FEATURE_RETPOLINE) && + !cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) return -1; =20 op =3D insn->opcode.bytes[0]; @@ -573,8 +575,9 @@ static int patch_retpoline(void *addr, s * into: * * Jncc.d8 1f + * [ LFENCE ] * JMP *%\reg - * NOP + * [ NOP ] * 1: */ /* Jcc.d32 second opcode byte is in the range: 0x80-0x8f */ @@ -589,6 +592,15 @@ static int patch_retpoline(void *addr, s op =3D JMP32_INSN_OPCODE; } =20 + /* + * For RETPOLINE_AMD: prepend the indirect CALL/JMP with an LFENCE. + */ + if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { + bytes[i++] =3D 0x0f; + bytes[i++] =3D 0xae; + bytes[i++] =3D 0xe8; /* LFENCE */ + } + ret =3D emit_indirect(op, reg, bytes + i); if (ret < 0) return ret; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6884C43334 for ; Sat, 23 Jul 2022 10:02:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237997AbiGWKCm (ORCPT ); Sat, 23 Jul 2022 06:02:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237723AbiGWKCP (ORCPT ); Sat, 23 Jul 2022 06:02:15 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7288A66AE5; Sat, 23 Jul 2022 02:58:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 65737611BD; Sat, 23 Jul 2022 09:58:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4A2E6C341C0; Sat, 23 Jul 2022 09:58:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570326; bh=Z9v+W6Ypo/TtsuKDyVQ7GeOlltbq63xMAp1hYKg1CZM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B13Slh3MIH/hbHJG/7JBDCPR8FKD1HhpfF1Z0JaNOl3/t3LDXfvqioPcwOjbQvoPc nfbHoiivnlcuo+l+p3YcNkYICY5JhxgFgzH/z8DhmAAXhMgip0j3gp7MUiUBFRr/Gb 1oh3gdkT+OcdG8M4dRxxNLti4jHqcWbsRDjio1XA= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Alexei Starovoitov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 058/148] x86/alternative: Add debug prints to apply_retpolines() Date: Sat, 23 Jul 2022 11:54:30 +0200 Message-Id: <20220723095240.500142143@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit d4b5a5c993009ffeb5febe3b701da3faab6adb96 upstream. Make sure we can see the text changes when booting with 'debug-alternative'. Example output: [ ] SMP alternatives: retpoline at: __traceiter_initcall_level+0x1f/0x30 (= ffffffff8100066f) len: 5 to: __x86_indirect_thunk_rax+0x0/0x20 [ ] SMP alternatives: ffffffff82603e58: [2:5) optimized NOPs: ff d0 0f 1f = 00 [ ] SMP alternatives: ffffffff8100066f: orig: e8 cc 30 00 01 [ ] SMP alternatives: ffffffff8100066f: repl: ff d0 0f 1f 00 Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.422273830@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/alternative.c | 6 ++++++ 1 file changed, 6 insertions(+) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -647,9 +647,15 @@ void __init_or_module noinline apply_ret continue; } =20 + DPRINTK("retpoline at: %pS (%px) len: %d to: %pS", + addr, addr, insn.length, + addr + insn.length + insn.immediate.value); + len =3D patch_retpoline(addr, &insn, bytes); if (len =3D=3D insn.length) { optimize_nops(bytes, len); + DUMP_BYTES(((u8*)addr), len, "%px: orig: ", addr); + DUMP_BYTES(((u8*)bytes), len, "%px: repl: ", addr); text_poke_early(addr, bytes, len); } } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4442EC433EF for ; Sat, 23 Jul 2022 10:02:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238116AbiGWKCt (ORCPT ); Sat, 23 Jul 2022 06:02:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237814AbiGWKCQ (ORCPT ); Sat, 23 Jul 2022 06:02:16 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 577C4804BB; Sat, 23 Jul 2022 02:58:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DC9B2B82B92; Sat, 23 Jul 2022 09:58:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4DF5EC341C0; Sat, 23 Jul 2022 09:58:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570329; bh=0N1LzbyNDavrV5k+6A090X+J0FMWV2xO3y75l1OYMoI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JZIHL0IKyYB7UzfuN9HSIgo00dVaEDu6unpSN+XWhU/NRrZyJvnKWZUMSyKNM/RZb Ppb7768G6rySfe7Bq83EoBqZB/nPBOcKmgIOh4pbT//T46MGm1WdydkE6KkO57ts9g OXm8LsuDrotfuXt0xNKjh2yDDlCMFng4w/Gvx4VY= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Alexei Starovoitov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 059/148] bpf,x86: Simplify computing label offsets Date: Sat, 23 Jul 2022 11:54:31 +0200 Message-Id: <20220723095240.777639683@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit dceba0817ca329868a15e2e1dd46eb6340b69206 upstream. Take an idea from the 32bit JIT, which uses the multi-pass nature of the JIT to compute the instruction offsets on a prior pass in order to compute the relative jump offsets on a later pass. Application to the x86_64 JIT is slightly more involved because the offsets depend on program variables (such as callee_regs_used and stack_depth) and hence the computed offsets need to be kept in the context of the JIT. This removes, IMO quite fragile, code that hard-codes the offsets and tries to compute the length of variable parts of it. Convert both emit_bpf_tail_call_*() functions which have an out: label at the end. Additionally emit_bpt_tail_call_direct() also has a poke table entry, for which it computes the offset from the end (and thus already relies on the previous pass to have computed addrs[i]), also convert this to be a forward based offset. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Alexei Starovoitov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.552304864@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: keep the cnt variable in emit_bpf_tail_call_{,in}direct()] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/net/bpf_jit_comp.c | 125 ++++++++++++++-------------------------= ----- 1 file changed, 42 insertions(+), 83 deletions(-) --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -212,6 +212,14 @@ static void jit_fill_hole(void *area, un =20 struct jit_context { int cleanup_addr; /* Epilogue code offset */ + + /* + * Program specific offsets of labels in the code; these rely on the + * JIT doing at least 2 passes, recording the position on the first + * pass, only to generate the correct offset on the second pass. + */ + int tail_call_direct_label; + int tail_call_indirect_label; }; =20 /* Maximum number of bytes emitted while JITing one eBPF insn */ @@ -371,22 +379,6 @@ int bpf_arch_text_poke(void *ip, enum bp return __bpf_arch_text_poke(ip, t, old_addr, new_addr, true); } =20 -static int get_pop_bytes(bool *callee_regs_used) -{ - int bytes =3D 0; - - if (callee_regs_used[3]) - bytes +=3D 2; - if (callee_regs_used[2]) - bytes +=3D 2; - if (callee_regs_used[1]) - bytes +=3D 2; - if (callee_regs_used[0]) - bytes +=3D 1; - - return bytes; -} - /* * Generate the following code: * @@ -402,30 +394,12 @@ static int get_pop_bytes(bool *callee_re * out: */ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, - u32 stack_depth) + u32 stack_depth, u8 *ip, + struct jit_context *ctx) { int tcc_off =3D -4 - round_up(stack_depth, 8); - u8 *prog =3D *pprog; - int pop_bytes =3D 0; - int off1 =3D 42; - int off2 =3D 31; - int off3 =3D 9; - int cnt =3D 0; - - /* count the additional bytes used for popping callee regs from stack - * that need to be taken into account for each of the offsets that - * are used for bailing out of the tail call - */ - pop_bytes =3D get_pop_bytes(callee_regs_used); - off1 +=3D pop_bytes; - off2 +=3D pop_bytes; - off3 +=3D pop_bytes; - - if (stack_depth) { - off1 +=3D 7; - off2 +=3D 7; - off3 +=3D 7; - } + u8 *prog =3D *pprog, *start =3D *pprog; + int cnt =3D 0, offset; =20 /* * rdi - pointer to ctx @@ -440,8 +414,9 @@ static void emit_bpf_tail_call_indirect( EMIT2(0x89, 0xD2); /* mov edx, edx */ EMIT3(0x39, 0x56, /* cmp dword ptr [rsi + 16], ed= x */ offsetof(struct bpf_array, map.max_entries)); -#define OFFSET1 (off1 + RETPOLINE_RCX_BPF_JIT_SIZE) /* Number of bytes to = jump */ - EMIT2(X86_JBE, OFFSET1); /* jbe out */ + + offset =3D ctx->tail_call_indirect_label - (prog + 2 - start); + EMIT2(X86_JBE, offset); /* jbe out */ =20 /* * if (tail_call_cnt > MAX_TAIL_CALL_CNT) @@ -449,8 +424,9 @@ static void emit_bpf_tail_call_indirect( */ EMIT2_off32(0x8B, 0x85, tcc_off); /* mov eax, dword ptr [rbp - tc= c_off] */ EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT); /* cmp eax, MAX_TAIL_CALL_CNT */ -#define OFFSET2 (off2 + RETPOLINE_RCX_BPF_JIT_SIZE) - EMIT2(X86_JA, OFFSET2); /* ja out */ + + offset =3D ctx->tail_call_indirect_label - (prog + 2 - start); + EMIT2(X86_JA, offset); /* ja out */ EMIT3(0x83, 0xC0, 0x01); /* add eax, 1 */ EMIT2_off32(0x89, 0x85, tcc_off); /* mov dword ptr [rbp - tcc_off= ], eax */ =20 @@ -463,12 +439,11 @@ static void emit_bpf_tail_call_indirect( * goto out; */ EMIT3(0x48, 0x85, 0xC9); /* test rcx,rcx */ -#define OFFSET3 (off3 + RETPOLINE_RCX_BPF_JIT_SIZE) - EMIT2(X86_JE, OFFSET3); /* je out */ =20 - *pprog =3D prog; - pop_callee_regs(pprog, callee_regs_used); - prog =3D *pprog; + offset =3D ctx->tail_call_indirect_label - (prog + 2 - start); + EMIT2(X86_JE, offset); /* je out */ + + pop_callee_regs(&prog, callee_regs_used); =20 EMIT1(0x58); /* pop rax */ if (stack_depth) @@ -488,39 +463,18 @@ static void emit_bpf_tail_call_indirect( RETPOLINE_RCX_BPF_JIT(); =20 /* out: */ + ctx->tail_call_indirect_label =3D prog - start; *pprog =3D prog; } =20 static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke, - u8 **pprog, int addr, u8 *image, - bool *callee_regs_used, u32 stack_depth) + u8 **pprog, u8 *ip, + bool *callee_regs_used, u32 stack_depth, + struct jit_context *ctx) { int tcc_off =3D -4 - round_up(stack_depth, 8); - u8 *prog =3D *pprog; - int pop_bytes =3D 0; - int off1 =3D 20; - int poke_off; - int cnt =3D 0; - - /* count the additional bytes used for popping callee regs to stack - * that need to be taken into account for jump offset that is used for - * bailing out from of the tail call when limit is reached - */ - pop_bytes =3D get_pop_bytes(callee_regs_used); - off1 +=3D pop_bytes; - - /* - * total bytes for: - * - nop5/ jmpq $off - * - pop callee regs - * - sub rsp, $val if depth > 0 - * - pop rax - */ - poke_off =3D X86_PATCH_SIZE + pop_bytes + 1; - if (stack_depth) { - poke_off +=3D 7; - off1 +=3D 7; - } + u8 *prog =3D *pprog, *start =3D *pprog; + int cnt =3D 0, offset; =20 /* * if (tail_call_cnt > MAX_TAIL_CALL_CNT) @@ -528,28 +482,30 @@ static void emit_bpf_tail_call_direct(st */ EMIT2_off32(0x8B, 0x85, tcc_off); /* mov eax, dword ptr [rbp = - tcc_off] */ EMIT3(0x83, 0xF8, MAX_TAIL_CALL_CNT); /* cmp eax, MAX_TAIL_CALL_C= NT */ - EMIT2(X86_JA, off1); /* ja out */ + + offset =3D ctx->tail_call_direct_label - (prog + 2 - start); + EMIT2(X86_JA, offset); /* ja out */ EMIT3(0x83, 0xC0, 0x01); /* add eax, 1 */ EMIT2_off32(0x89, 0x85, tcc_off); /* mov dword ptr [rbp - tcc= _off], eax */ =20 - poke->tailcall_bypass =3D image + (addr - poke_off - X86_PATCH_SIZE); + poke->tailcall_bypass =3D ip + (prog - start); poke->adj_off =3D X86_TAIL_CALL_OFFSET; - poke->tailcall_target =3D image + (addr - X86_PATCH_SIZE); + poke->tailcall_target =3D ip + ctx->tail_call_direct_label - X86_PATCH_SI= ZE; poke->bypass_addr =3D (u8 *)poke->tailcall_target + X86_PATCH_SIZE; =20 emit_jump(&prog, (u8 *)poke->tailcall_target + X86_PATCH_SIZE, poke->tailcall_bypass); =20 - *pprog =3D prog; - pop_callee_regs(pprog, callee_regs_used); - prog =3D *pprog; + pop_callee_regs(&prog, callee_regs_used); EMIT1(0x58); /* pop rax */ if (stack_depth) EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8)); =20 memcpy(prog, ideal_nops[NOP_ATOMIC5], X86_PATCH_SIZE); prog +=3D X86_PATCH_SIZE; + /* out: */ + ctx->tail_call_direct_label =3D prog - start; =20 *pprog =3D prog; } @@ -1274,13 +1230,16 @@ xadd: if (is_imm8(insn->off)) case BPF_JMP | BPF_TAIL_CALL: if (imm32) emit_bpf_tail_call_direct(&bpf_prog->aux->poke_tab[imm32 - 1], - &prog, addrs[i], image, + &prog, image + addrs[i - 1], callee_regs_used, - bpf_prog->aux->stack_depth); + bpf_prog->aux->stack_depth, + ctx); else emit_bpf_tail_call_indirect(&prog, callee_regs_used, - bpf_prog->aux->stack_depth); + bpf_prog->aux->stack_depth, + image + addrs[i - 1], + ctx); break; =20 /* cond jump */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D260EC43334 for ; Sat, 23 Jul 2022 10:05:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238329AbiGWKFN (ORCPT ); Sat, 23 Jul 2022 06:05:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238389AbiGWKEU (ORCPT ); Sat, 23 Jul 2022 06:04:20 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B63316EEA8; Sat, 23 Jul 2022 02:59:55 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AAF296121F; Sat, 23 Jul 2022 09:59:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B33D8C341C0; Sat, 23 Jul 2022 09:59:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570393; bh=nJKapeQMlrIeJZpFilw2PfRWT6K3ylI1DSqHQel0aj8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wUbz4JuSVkSEgOsugH4x1xYSwJrZPcEepMT1RSfx4lKcOdqy+qwnu51i4fPRVbzvZ gOEAWWU4BTISIJRfos0GMiTBFlPX2CVgMa9zj2Nwyv9FzUSQy2fnKCUY2KC46+U9zB XnSHrqMmCepD930ch+rol440jsSoOJIoR4fHl5sM= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Alexei Starovoitov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 060/148] bpf,x86: Respect X86_FEATURE_RETPOLINE* Date: Sat, 23 Jul 2022 11:54:32 +0200 Message-Id: <20220723095241.027856833@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 87c87ecd00c54ecd677798cb49ef27329e0fab41 upstream. Current BPF codegen doesn't respect X86_FEATURE_RETPOLINE* flags and unconditionally emits a thunk call, this is sub-optimal and doesn't match the regular, compiler generated, code. Update the i386 JIT to emit code equal to what the compiler emits for the regular kernel text (IOW. a plain THUNK call). Update the x86_64 JIT to emit code similar to the result of compiler and kernel rewrites as according to X86_FEATURE_RETPOLINE* flags. Inlining RETPOLINE_AMD (lfence; jmp *%reg) and !RETPOLINE (jmp *%reg), while doing a THUNK call for RETPOLINE. This removes the hard-coded retpoline thunks and shrinks the generated code. Leaving a single retpoline thunk definition in the kernel. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Borislav Petkov Acked-by: Alexei Starovoitov Acked-by: Josh Poimboeuf Tested-by: Alexei Starovoitov Link: https://lore.kernel.org/r/20211026120310.614772675@infradead.org [cascardo: RETPOLINE_AMD was renamed to RETPOLINE_LFENCE] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: add the necessary cnt variable to emit_indirect_jump()] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 59 ------------------------------= ----- arch/x86/net/bpf_jit_comp.c | 49 +++++++++++++---------------- arch/x86/net/bpf_jit_comp32.c | 22 +++++++++++-- 3 files changed, 42 insertions(+), 88 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -318,63 +318,4 @@ static inline void mds_idle_clear_cpu_bu =20 #endif /* __ASSEMBLY__ */ =20 -/* - * Below is used in the eBPF JIT compiler and emits the byte sequence - * for the following assembly: - * - * With retpolines configured: - * - * callq do_rop - * spec_trap: - * pause - * lfence - * jmp spec_trap - * do_rop: - * mov %rcx,(%rsp) for x86_64 - * mov %edx,(%esp) for x86_32 - * retq - * - * Without retpolines configured: - * - * jmp *%rcx for x86_64 - * jmp *%edx for x86_32 - */ -#ifdef CONFIG_RETPOLINE -# ifdef CONFIG_X86_64 -# define RETPOLINE_RCX_BPF_JIT_SIZE 17 -# define RETPOLINE_RCX_BPF_JIT() \ -do { \ - EMIT1_off32(0xE8, 7); /* callq do_rop */ \ - /* spec_trap: */ \ - EMIT2(0xF3, 0x90); /* pause */ \ - EMIT3(0x0F, 0xAE, 0xE8); /* lfence */ \ - EMIT2(0xEB, 0xF9); /* jmp spec_trap */ \ - /* do_rop: */ \ - EMIT4(0x48, 0x89, 0x0C, 0x24); /* mov %rcx,(%rsp) */ \ - EMIT1(0xC3); /* retq */ \ -} while (0) -# else /* !CONFIG_X86_64 */ -# define RETPOLINE_EDX_BPF_JIT() \ -do { \ - EMIT1_off32(0xE8, 7); /* call do_rop */ \ - /* spec_trap: */ \ - EMIT2(0xF3, 0x90); /* pause */ \ - EMIT3(0x0F, 0xAE, 0xE8); /* lfence */ \ - EMIT2(0xEB, 0xF9); /* jmp spec_trap */ \ - /* do_rop: */ \ - EMIT3(0x89, 0x14, 0x24); /* mov %edx,(%esp) */ \ - EMIT1(0xC3); /* ret */ \ -} while (0) -# endif -#else /* !CONFIG_RETPOLINE */ -# ifdef CONFIG_X86_64 -# define RETPOLINE_RCX_BPF_JIT_SIZE 2 -# define RETPOLINE_RCX_BPF_JIT() \ - EMIT2(0xFF, 0xE1); /* jmp *%rcx */ -# else /* !CONFIG_X86_64 */ -# define RETPOLINE_EDX_BPF_JIT() \ - EMIT2(0xFF, 0xE2) /* jmp *%edx */ -# endif -#endif - #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -379,6 +379,26 @@ int bpf_arch_text_poke(void *ip, enum bp return __bpf_arch_text_poke(ip, t, old_addr, new_addr, true); } =20 +#define EMIT_LFENCE() EMIT3(0x0F, 0xAE, 0xE8) + +static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip) +{ + u8 *prog =3D *pprog; + int cnt =3D 0; + +#ifdef CONFIG_RETPOLINE + if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { + EMIT_LFENCE(); + EMIT2(0xFF, 0xE0 + reg); + } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) { + emit_jump(&prog, &__x86_indirect_thunk_array[reg], ip); + } else +#endif + EMIT2(0xFF, 0xE0 + reg); + + *pprog =3D prog; +} + /* * Generate the following code: * @@ -460,7 +480,7 @@ static void emit_bpf_tail_call_indirect( * rdi =3D=3D ctx (1st arg) * rcx =3D=3D prog->bpf_func + X86_TAIL_CALL_OFFSET */ - RETPOLINE_RCX_BPF_JIT(); + emit_indirect_jump(&prog, 1 /* rcx */, ip + (prog - start)); =20 /* out: */ ctx->tail_call_indirect_label =3D prog - start; @@ -1099,8 +1119,7 @@ static int do_jit(struct bpf_prog *bpf_p /* speculation barrier */ case BPF_ST | BPF_NOSPEC: if (boot_cpu_has(X86_FEATURE_XMM2)) - /* Emit 'lfence' */ - EMIT3(0x0F, 0xAE, 0xE8); + EMIT_LFENCE(); break; =20 /* ST: *(u8*)(dst_reg + off) =3D imm */ @@ -1878,26 +1897,6 @@ cleanup: return ret; } =20 -static int emit_fallback_jump(u8 **pprog) -{ - u8 *prog =3D *pprog; - int err =3D 0; - -#ifdef CONFIG_RETPOLINE - /* Note that this assumes the the compiler uses external - * thunks for indirect calls. Both clang and GCC use the same - * naming convention for external thunks. - */ - err =3D emit_jump(&prog, __x86_indirect_thunk_rdx, prog); -#else - int cnt =3D 0; - - EMIT2(0xFF, 0xE2); /* jmp rdx */ -#endif - *pprog =3D prog; - return err; -} - static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs) { u8 *jg_reloc, *prog =3D *pprog; @@ -1919,9 +1918,7 @@ static int emit_bpf_dispatcher(u8 **ppro if (err) return err; =20 - err =3D emit_fallback_jump(&prog); /* jmp thunk/indirect */ - if (err) - return err; + emit_indirect_jump(&prog, 2 /* rdx */, prog); =20 *pprog =3D prog; return 0; --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -15,6 +15,7 @@ #include #include #include +#include #include =20 /* @@ -1267,6 +1268,21 @@ static void emit_epilogue(u8 **pprog, u3 *pprog =3D prog; } =20 +static int emit_jmp_edx(u8 **pprog, u8 *ip) +{ + u8 *prog =3D *pprog; + int cnt =3D 0; + +#ifdef CONFIG_RETPOLINE + EMIT1_off32(0xE9, (u8 *)__x86_indirect_thunk_edx - (ip + 5)); +#else + EMIT2(0xFF, 0xE2); +#endif + *pprog =3D prog; + + return cnt; +} + /* * Generate the following code: * ... bpf_tail_call(void *ctx, struct bpf_array *array, u64 index) ... @@ -1280,7 +1296,7 @@ static void emit_epilogue(u8 **pprog, u3 * goto *(prog->bpf_func + prologue_size); * out: */ -static void emit_bpf_tail_call(u8 **pprog) +static void emit_bpf_tail_call(u8 **pprog, u8 *ip) { u8 *prog =3D *pprog; int cnt =3D 0; @@ -1362,7 +1378,7 @@ static void emit_bpf_tail_call(u8 **ppro * eax =3D=3D ctx (1st arg) * edx =3D=3D prog->bpf_func + prologue_size */ - RETPOLINE_EDX_BPF_JIT(); + cnt +=3D emit_jmp_edx(&prog, ip + cnt); =20 if (jmp_label1 =3D=3D -1) jmp_label1 =3D cnt; @@ -1929,7 +1945,7 @@ static int do_jit(struct bpf_prog *bpf_p break; } case BPF_JMP | BPF_TAIL_CALL: - emit_bpf_tail_call(&prog); + emit_bpf_tail_call(&prog, image + addrs[i - 1]); break; =20 /* cond jump */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6953CC433EF for ; Sat, 23 Jul 2022 10:03:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238217AbiGWKDg (ORCPT ); Sat, 23 Jul 2022 06:03:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237736AbiGWKDQ (ORCPT ); Sat, 23 Jul 2022 06:03:16 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9728687205; Sat, 23 Jul 2022 02:59:24 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 827A2B82C24; Sat, 23 Jul 2022 09:59:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D6994C341C0; Sat, 23 Jul 2022 09:59:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570358; bh=l9Rc68ra/4bHVh2wqbrdbsWHOSZSdMEmpI128/zN7Yg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=coLRuZAqJy2PquUev9j3WJJhRozDDh/+FoJ6HLZNV/zJzRWlPKBgPauVvfq24XBXf ZqZTRU4TRuqb+6CUTGuiCifzytOQQ+LvcCD6pEBXifx9trZtjqwBTUrNvbJscLzNKa J3A02C/ezAfeG4uN9g29a1Va3vn1rDThmGNq+ItA= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 061/148] x86/lib/atomic64_386_32: Rename things Date: Sat, 23 Jul 2022 11:54:33 +0200 Message-Id: <20220723095241.371425867@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 22da5a07c75e1104caf6a42f189c97b83d070073 upstream. Principally, in order to get rid of #define RET in this code to make place for a new RET, but also to clarify the code, rename a bunch of things: s/UNLOCK/IRQ_RESTORE/ s/LOCK/IRQ_SAVE/ s/BEGIN/BEGIN_IRQ_SAVE/ s/\/RET_IRQ_RESTORE/ s/RET_ENDP/\tRET_IRQ_RESTORE\rENDP/ which then leaves RET unused so it can be removed. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/20211204134907.841623970@infradead.org Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/lib/atomic64_386_32.S | 84 ++++++++++++++++++++++--------------= ----- 1 file changed, 46 insertions(+), 38 deletions(-) --- a/arch/x86/lib/atomic64_386_32.S +++ b/arch/x86/lib/atomic64_386_32.S @@ -9,81 +9,83 @@ #include =20 /* if you want SMP support, implement these with real spinlocks */ -.macro LOCK reg +.macro IRQ_SAVE reg pushfl cli .endm =20 -.macro UNLOCK reg +.macro IRQ_RESTORE reg popfl .endm =20 -#define BEGIN(op) \ +#define BEGIN_IRQ_SAVE(op) \ .macro endp; \ SYM_FUNC_END(atomic64_##op##_386); \ .purgem endp; \ .endm; \ SYM_FUNC_START(atomic64_##op##_386); \ - LOCK v; + IRQ_SAVE v; =20 #define ENDP endp =20 -#define RET \ - UNLOCK v; \ +#define RET_IRQ_RESTORE \ + IRQ_RESTORE v; \ ret =20 -#define RET_ENDP \ - RET; \ - ENDP - #define v %ecx -BEGIN(read) +BEGIN_IRQ_SAVE(read) movl (v), %eax movl 4(v), %edx -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %esi -BEGIN(set) +BEGIN_IRQ_SAVE(set) movl %ebx, (v) movl %ecx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %esi -BEGIN(xchg) +BEGIN_IRQ_SAVE(xchg) movl (v), %eax movl 4(v), %edx movl %ebx, (v) movl %ecx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %ecx -BEGIN(add) +BEGIN_IRQ_SAVE(add) addl %eax, (v) adcl %edx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %ecx -BEGIN(add_return) +BEGIN_IRQ_SAVE(add_return) addl (v), %eax adcl 4(v), %edx movl %eax, (v) movl %edx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %ecx -BEGIN(sub) +BEGIN_IRQ_SAVE(sub) subl %eax, (v) sbbl %edx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %ecx -BEGIN(sub_return) +BEGIN_IRQ_SAVE(sub_return) negl %edx negl %eax sbbl $0, %edx @@ -91,47 +93,52 @@ BEGIN(sub_return) adcl 4(v), %edx movl %eax, (v) movl %edx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %esi -BEGIN(inc) +BEGIN_IRQ_SAVE(inc) addl $1, (v) adcl $0, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %esi -BEGIN(inc_return) +BEGIN_IRQ_SAVE(inc_return) movl (v), %eax movl 4(v), %edx addl $1, %eax adcl $0, %edx movl %eax, (v) movl %edx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %esi -BEGIN(dec) +BEGIN_IRQ_SAVE(dec) subl $1, (v) sbbl $0, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %esi -BEGIN(dec_return) +BEGIN_IRQ_SAVE(dec_return) movl (v), %eax movl 4(v), %edx subl $1, %eax sbbl $0, %edx movl %eax, (v) movl %edx, 4(v) -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v =20 #define v %esi -BEGIN(add_unless) +BEGIN_IRQ_SAVE(add_unless) addl %eax, %ecx adcl %edx, %edi addl (v), %eax @@ -143,7 +150,7 @@ BEGIN(add_unless) movl %edx, 4(v) movl $1, %eax 2: - RET + RET_IRQ_RESTORE 3: cmpl %edx, %edi jne 1b @@ -153,7 +160,7 @@ ENDP #undef v =20 #define v %esi -BEGIN(inc_not_zero) +BEGIN_IRQ_SAVE(inc_not_zero) movl (v), %eax movl 4(v), %edx testl %eax, %eax @@ -165,7 +172,7 @@ BEGIN(inc_not_zero) movl %edx, 4(v) movl $1, %eax 2: - RET + RET_IRQ_RESTORE 3: testl %edx, %edx jne 1b @@ -174,7 +181,7 @@ ENDP #undef v =20 #define v %esi -BEGIN(dec_if_positive) +BEGIN_IRQ_SAVE(dec_if_positive) movl (v), %eax movl 4(v), %edx subl $1, %eax @@ -183,5 +190,6 @@ BEGIN(dec_if_positive) movl %eax, (v) movl %edx, 4(v) 1: -RET_ENDP + RET_IRQ_RESTORE +ENDP #undef v From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA5FCC43334 for ; Sat, 23 Jul 2022 10:04:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238043AbiGWKED (ORCPT ); Sat, 23 Jul 2022 06:04:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238221AbiGWKDl (ORCPT ); Sat, 23 Jul 2022 06:03:41 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 053E98BA98; Sat, 23 Jul 2022 02:59:31 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D13CE6121F; Sat, 23 Jul 2022 09:59:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 749B3C341CA; Sat, 23 Jul 2022 09:59:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570370; bh=2ycqqBemgitgmqLwude8UasV5oIG76MSHxW0apQqCKQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AWoxHP4x7XvOEKULHExm6RQOYCQIoaZk+1nQeGxdS0j95Rz9WEs/OP0kr9AH/0+oM +6MTbRLdsVaDTILZgbJF1o+2MyyRBgeohuH3lbbWgJkU0CJWoksc+CPPqjlJAcdsib usZNe1M8Y6xCLSuuWDTHS9RP2CIzx7xC043rQl5A= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 062/148] x86: Prepare asm files for straight-line-speculation Date: Sat, 23 Jul 2022 11:54:34 +0200 Message-Id: <20220723095241.630661983@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit f94909ceb1ed4bfdb2ada72f93236305e6d6951f upstream. Replace all ret/retq instructions with RET in preparation of making RET a macro. Since AS is case insensitive it's a big no-op without RET defined. find arch/x86/ -name \*.S | while read file do sed -i 's/\/RET/' $file done Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/20211204134907.905503893@infradead.org [bwh: Backported to 5.10: ran the above command] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/boot/compressed/efi_thunk_64.S | 2 - arch/x86/boot/compressed/head_64.S | 4 +- arch/x86/boot/compressed/mem_encrypt.S | 4 +- arch/x86/crypto/aegis128-aesni-asm.S | 48 ++++++++++++----------= -- arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 2 - arch/x86/crypto/aesni-intel_asm.S | 52 +++++++++++++---------= ----- arch/x86/crypto/aesni-intel_avx-x86_64.S | 40 ++++++++++---------- arch/x86/crypto/blake2s-core.S | 4 +- arch/x86/crypto/blowfish-x86_64-asm_64.S | 12 +++--- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 18 ++++----- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 18 ++++----- arch/x86/crypto/camellia-x86_64-asm_64.S | 12 +++--- arch/x86/crypto/cast5-avx-x86_64-asm_64.S | 12 +++--- arch/x86/crypto/cast6-avx-x86_64-asm_64.S | 16 ++++---- arch/x86/crypto/chacha-avx2-x86_64.S | 6 +-- arch/x86/crypto/chacha-avx512vl-x86_64.S | 6 +-- arch/x86/crypto/chacha-ssse3-x86_64.S | 8 ++-- arch/x86/crypto/crc32-pclmul_asm.S | 2 - arch/x86/crypto/crc32c-pcl-intel-asm_64.S | 2 - arch/x86/crypto/crct10dif-pcl-asm_64.S | 2 - arch/x86/crypto/des3_ede-asm_64.S | 4 +- arch/x86/crypto/ghash-clmulni-intel_asm.S | 6 +-- arch/x86/crypto/nh-avx2-x86_64.S | 2 - arch/x86/crypto/nh-sse2-x86_64.S | 2 - arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 16 ++++---- arch/x86/crypto/serpent-avx2-asm_64.S | 16 ++++---- arch/x86/crypto/serpent-sse2-i586-asm_32.S | 6 +-- arch/x86/crypto/serpent-sse2-x86_64-asm_64.S | 6 +-- arch/x86/crypto/sha1_avx2_x86_64_asm.S | 2 - arch/x86/crypto/sha1_ni_asm.S | 2 - arch/x86/crypto/sha1_ssse3_asm.S | 2 - arch/x86/crypto/sha256-avx-asm.S | 2 - arch/x86/crypto/sha256-avx2-asm.S | 2 - arch/x86/crypto/sha256-ssse3-asm.S | 2 - arch/x86/crypto/sha256_ni_asm.S | 2 - arch/x86/crypto/sha512-avx-asm.S | 2 - arch/x86/crypto/sha512-avx2-asm.S | 2 - arch/x86/crypto/sha512-ssse3-asm.S | 2 - arch/x86/crypto/twofish-avx-x86_64-asm_64.S | 16 ++++---- arch/x86/crypto/twofish-i586-asm_32.S | 4 +- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 6 +-- arch/x86/crypto/twofish-x86_64-asm_64.S | 4 +- arch/x86/entry/entry_32.S | 2 - arch/x86/entry/entry_64.S | 12 +++--- arch/x86/entry/thunk_32.S | 2 - arch/x86/entry/thunk_64.S | 2 - arch/x86/entry/vdso/vdso32/system_call.S | 2 - arch/x86/entry/vsyscall/vsyscall_emu_64.S | 6 +-- arch/x86/kernel/acpi/wakeup_32.S | 6 +-- arch/x86/kernel/ftrace_32.S | 6 +-- arch/x86/kernel/ftrace_64.S | 10 ++--- arch/x86/kernel/head_32.S | 2 - arch/x86/kernel/irqflags.S | 4 +- arch/x86/kernel/relocate_kernel_32.S | 10 ++--- arch/x86/kernel/relocate_kernel_64.S | 10 ++--- arch/x86/kernel/sev_verify_cbit.S | 2 - arch/x86/kernel/verify_cpu.S | 4 +- arch/x86/kvm/svm/vmenter.S | 2 - arch/x86/kvm/vmx/vmenter.S | 14 +++---- arch/x86/lib/atomic64_386_32.S | 2 - arch/x86/lib/atomic64_cx8_32.S | 16 ++++---- arch/x86/lib/checksum_32.S | 8 ++-- arch/x86/lib/clear_page_64.S | 6 +-- arch/x86/lib/cmpxchg16b_emu.S | 4 +- arch/x86/lib/cmpxchg8b_emu.S | 4 +- arch/x86/lib/copy_mc_64.S | 6 +-- arch/x86/lib/copy_page_64.S | 4 +- arch/x86/lib/copy_user_64.S | 12 +++--- arch/x86/lib/csum-copy_64.S | 2 - arch/x86/lib/getuser.S | 22 +++++------ arch/x86/lib/hweight.S | 6 +-- arch/x86/lib/iomap_copy_64.S | 2 - arch/x86/lib/memcpy_64.S | 12 +++--- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 6 +-- arch/x86/lib/msr-reg.S | 4 +- arch/x86/lib/putuser.S | 6 +-- arch/x86/lib/retpoline.S | 2 - arch/x86/math-emu/div_Xsig.S | 2 - arch/x86/math-emu/div_small.S | 2 - arch/x86/math-emu/mul_Xsig.S | 6 +-- arch/x86/math-emu/polynom_Xsig.S | 2 - arch/x86/math-emu/reg_norm.S | 6 +-- arch/x86/math-emu/reg_round.S | 2 - arch/x86/math-emu/reg_u_add.S | 2 - arch/x86/math-emu/reg_u_div.S | 2 - arch/x86/math-emu/reg_u_mul.S | 2 - arch/x86/math-emu/reg_u_sub.S | 2 - arch/x86/math-emu/round_Xsig.S | 4 +- arch/x86/math-emu/shr_Xsig.S | 8 ++-- arch/x86/math-emu/wm_shrx.S | 16 ++++---- arch/x86/mm/mem_encrypt_boot.S | 4 +- arch/x86/platform/efi/efi_stub_32.S | 2 - arch/x86/platform/efi/efi_stub_64.S | 2 - arch/x86/platform/efi/efi_thunk_64.S | 2 - arch/x86/platform/olpc/xo1-wakeup.S | 6 +-- arch/x86/power/hibernate_asm_32.S | 4 +- arch/x86/power/hibernate_asm_64.S | 4 +- arch/x86/um/checksum_32.S | 4 +- arch/x86/um/setjmp_32.S | 2 - arch/x86/um/setjmp_64.S | 2 - arch/x86/xen/xen-asm.S | 14 +++---- arch/x86/xen/xen-head.S | 2 - 103 files changed, 353 insertions(+), 353 deletions(-) --- a/arch/x86/boot/compressed/efi_thunk_64.S +++ b/arch/x86/boot/compressed/efi_thunk_64.S @@ -89,7 +89,7 @@ SYM_FUNC_START(__efi64_thunk) =20 pop %rbx pop %rbp - ret + RET SYM_FUNC_END(__efi64_thunk) =20 .code32 --- a/arch/x86/boot/compressed/head_64.S +++ b/arch/x86/boot/compressed/head_64.S @@ -786,7 +786,7 @@ SYM_FUNC_START(efi32_pe_entry) 2: popl %edi // restore callee-save registers popl %ebx leave - ret + RET SYM_FUNC_END(efi32_pe_entry) =20 .section ".rodata" @@ -868,7 +868,7 @@ SYM_FUNC_START(startup32_check_sev_cbit) popl %ebx popl %eax #endif - ret + RET SYM_FUNC_END(startup32_check_sev_cbit) =20 /* --- a/arch/x86/boot/compressed/mem_encrypt.S +++ b/arch/x86/boot/compressed/mem_encrypt.S @@ -58,7 +58,7 @@ SYM_FUNC_START(get_sev_encryption_bit) =20 #endif /* CONFIG_AMD_MEM_ENCRYPT */ =20 - ret + RET SYM_FUNC_END(get_sev_encryption_bit) =20 .code64 @@ -99,7 +99,7 @@ SYM_FUNC_START(set_sev_encryption_mask) #endif =20 xor %rax, %rax - ret + RET SYM_FUNC_END(set_sev_encryption_mask) =20 .data --- a/arch/x86/crypto/aegis128-aesni-asm.S +++ b/arch/x86/crypto/aegis128-aesni-asm.S @@ -122,7 +122,7 @@ SYM_FUNC_START_LOCAL(__load_partial) pxor T0, MSG =20 .Lld_partial_8: - ret + RET SYM_FUNC_END(__load_partial) =20 /* @@ -180,7 +180,7 @@ SYM_FUNC_START_LOCAL(__store_partial) mov %r10b, (%r9) =20 .Lst_partial_1: - ret + RET SYM_FUNC_END(__store_partial) =20 /* @@ -225,7 +225,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ini movdqu STATE4, 0x40(STATEP) =20 FRAME_END - ret + RET SYM_FUNC_END(crypto_aegis128_aesni_init) =20 /* @@ -337,7 +337,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad) movdqu STATE3, 0x30(STATEP) movdqu STATE4, 0x40(STATEP) FRAME_END - ret + RET =20 .Lad_out_1: movdqu STATE4, 0x00(STATEP) @@ -346,7 +346,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad) movdqu STATE2, 0x30(STATEP) movdqu STATE3, 0x40(STATEP) FRAME_END - ret + RET =20 .Lad_out_2: movdqu STATE3, 0x00(STATEP) @@ -355,7 +355,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad) movdqu STATE1, 0x30(STATEP) movdqu STATE2, 0x40(STATEP) FRAME_END - ret + RET =20 .Lad_out_3: movdqu STATE2, 0x00(STATEP) @@ -364,7 +364,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad) movdqu STATE0, 0x30(STATEP) movdqu STATE1, 0x40(STATEP) FRAME_END - ret + RET =20 .Lad_out_4: movdqu STATE1, 0x00(STATEP) @@ -373,11 +373,11 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad) movdqu STATE4, 0x30(STATEP) movdqu STATE0, 0x40(STATEP) FRAME_END - ret + RET =20 .Lad_out: FRAME_END - ret + RET SYM_FUNC_END(crypto_aegis128_aesni_ad) =20 .macro encrypt_block a s0 s1 s2 s3 s4 i @@ -452,7 +452,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc movdqu STATE2, 0x30(STATEP) movdqu STATE3, 0x40(STATEP) FRAME_END - ret + RET =20 .Lenc_out_1: movdqu STATE3, 0x00(STATEP) @@ -461,7 +461,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc movdqu STATE1, 0x30(STATEP) movdqu STATE2, 0x40(STATEP) FRAME_END - ret + RET =20 .Lenc_out_2: movdqu STATE2, 0x00(STATEP) @@ -470,7 +470,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc movdqu STATE0, 0x30(STATEP) movdqu STATE1, 0x40(STATEP) FRAME_END - ret + RET =20 .Lenc_out_3: movdqu STATE1, 0x00(STATEP) @@ -479,7 +479,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc movdqu STATE4, 0x30(STATEP) movdqu STATE0, 0x40(STATEP) FRAME_END - ret + RET =20 .Lenc_out_4: movdqu STATE0, 0x00(STATEP) @@ -488,11 +488,11 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc movdqu STATE3, 0x30(STATEP) movdqu STATE4, 0x40(STATEP) FRAME_END - ret + RET =20 .Lenc_out: FRAME_END - ret + RET SYM_FUNC_END(crypto_aegis128_aesni_enc) =20 /* @@ -532,7 +532,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_enc movdqu STATE3, 0x40(STATEP) =20 FRAME_END - ret + RET SYM_FUNC_END(crypto_aegis128_aesni_enc_tail) =20 .macro decrypt_block a s0 s1 s2 s3 s4 i @@ -606,7 +606,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec movdqu STATE2, 0x30(STATEP) movdqu STATE3, 0x40(STATEP) FRAME_END - ret + RET =20 .Ldec_out_1: movdqu STATE3, 0x00(STATEP) @@ -615,7 +615,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec movdqu STATE1, 0x30(STATEP) movdqu STATE2, 0x40(STATEP) FRAME_END - ret + RET =20 .Ldec_out_2: movdqu STATE2, 0x00(STATEP) @@ -624,7 +624,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec movdqu STATE0, 0x30(STATEP) movdqu STATE1, 0x40(STATEP) FRAME_END - ret + RET =20 .Ldec_out_3: movdqu STATE1, 0x00(STATEP) @@ -633,7 +633,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec movdqu STATE4, 0x30(STATEP) movdqu STATE0, 0x40(STATEP) FRAME_END - ret + RET =20 .Ldec_out_4: movdqu STATE0, 0x00(STATEP) @@ -642,11 +642,11 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec movdqu STATE3, 0x30(STATEP) movdqu STATE4, 0x40(STATEP) FRAME_END - ret + RET =20 .Ldec_out: FRAME_END - ret + RET SYM_FUNC_END(crypto_aegis128_aesni_dec) =20 /* @@ -696,7 +696,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_dec movdqu STATE3, 0x40(STATEP) =20 FRAME_END - ret + RET SYM_FUNC_END(crypto_aegis128_aesni_dec_tail) =20 /* @@ -743,5 +743,5 @@ SYM_FUNC_START(crypto_aegis128_aesni_fin movdqu MSG, (%rsi) =20 FRAME_END - ret + RET SYM_FUNC_END(crypto_aegis128_aesni_final) --- a/arch/x86/crypto/aes_ctrby8_avx-x86_64.S +++ b/arch/x86/crypto/aes_ctrby8_avx-x86_64.S @@ -525,7 +525,7 @@ ddq_add_8: /* return updated IV */ vpshufb xbyteswap, xcounter, xcounter vmovdqu xcounter, (p_iv) - ret + RET .endm =20 /* --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -1598,7 +1598,7 @@ SYM_FUNC_START(aesni_gcm_dec) GCM_ENC_DEC dec GCM_COMPLETE arg10, arg11 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_dec) =20 =20 @@ -1687,7 +1687,7 @@ SYM_FUNC_START(aesni_gcm_enc) =20 GCM_COMPLETE arg10, arg11 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_enc) =20 /*************************************************************************= **** @@ -1705,7 +1705,7 @@ SYM_FUNC_START(aesni_gcm_init) FUNC_SAVE GCM_INIT %arg3, %arg4,%arg5, %arg6 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_init) =20 /*************************************************************************= **** @@ -1720,7 +1720,7 @@ SYM_FUNC_START(aesni_gcm_enc_update) FUNC_SAVE GCM_ENC_DEC enc FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_enc_update) =20 /*************************************************************************= **** @@ -1735,7 +1735,7 @@ SYM_FUNC_START(aesni_gcm_dec_update) FUNC_SAVE GCM_ENC_DEC dec FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_dec_update) =20 /*************************************************************************= **** @@ -1750,7 +1750,7 @@ SYM_FUNC_START(aesni_gcm_finalize) FUNC_SAVE GCM_COMPLETE %arg3 %arg4 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_finalize) =20 #endif @@ -1766,7 +1766,7 @@ SYM_FUNC_START_LOCAL(_key_expansion_256a pxor %xmm1, %xmm0 movaps %xmm0, (TKEYP) add $0x10, TKEYP - ret + RET SYM_FUNC_END(_key_expansion_256a) SYM_FUNC_END_ALIAS(_key_expansion_128) =20 @@ -1791,7 +1791,7 @@ SYM_FUNC_START_LOCAL(_key_expansion_192a shufps $0b01001110, %xmm2, %xmm1 movaps %xmm1, 0x10(TKEYP) add $0x20, TKEYP - ret + RET SYM_FUNC_END(_key_expansion_192a) =20 SYM_FUNC_START_LOCAL(_key_expansion_192b) @@ -1810,7 +1810,7 @@ SYM_FUNC_START_LOCAL(_key_expansion_192b =20 movaps %xmm0, (TKEYP) add $0x10, TKEYP - ret + RET SYM_FUNC_END(_key_expansion_192b) =20 SYM_FUNC_START_LOCAL(_key_expansion_256b) @@ -1822,7 +1822,7 @@ SYM_FUNC_START_LOCAL(_key_expansion_256b pxor %xmm1, %xmm2 movaps %xmm2, (TKEYP) add $0x10, TKEYP - ret + RET SYM_FUNC_END(_key_expansion_256b) =20 /* @@ -1937,7 +1937,7 @@ SYM_FUNC_START(aesni_set_key) popl KEYP #endif FRAME_END - ret + RET SYM_FUNC_END(aesni_set_key) =20 /* @@ -1961,7 +1961,7 @@ SYM_FUNC_START(aesni_enc) popl KEYP #endif FRAME_END - ret + RET SYM_FUNC_END(aesni_enc) =20 /* @@ -2018,7 +2018,7 @@ SYM_FUNC_START_LOCAL(_aesni_enc1) aesenc KEY, STATE movaps 0x70(TKEYP), KEY aesenclast KEY, STATE - ret + RET SYM_FUNC_END(_aesni_enc1) =20 /* @@ -2126,7 +2126,7 @@ SYM_FUNC_START_LOCAL(_aesni_enc4) aesenclast KEY, STATE2 aesenclast KEY, STATE3 aesenclast KEY, STATE4 - ret + RET SYM_FUNC_END(_aesni_enc4) =20 /* @@ -2151,7 +2151,7 @@ SYM_FUNC_START(aesni_dec) popl KEYP #endif FRAME_END - ret + RET SYM_FUNC_END(aesni_dec) =20 /* @@ -2208,7 +2208,7 @@ SYM_FUNC_START_LOCAL(_aesni_dec1) aesdec KEY, STATE movaps 0x70(TKEYP), KEY aesdeclast KEY, STATE - ret + RET SYM_FUNC_END(_aesni_dec1) =20 /* @@ -2316,7 +2316,7 @@ SYM_FUNC_START_LOCAL(_aesni_dec4) aesdeclast KEY, STATE2 aesdeclast KEY, STATE3 aesdeclast KEY, STATE4 - ret + RET SYM_FUNC_END(_aesni_dec4) =20 /* @@ -2376,7 +2376,7 @@ SYM_FUNC_START(aesni_ecb_enc) popl LEN #endif FRAME_END - ret + RET SYM_FUNC_END(aesni_ecb_enc) =20 /* @@ -2437,7 +2437,7 @@ SYM_FUNC_START(aesni_ecb_dec) popl LEN #endif FRAME_END - ret + RET SYM_FUNC_END(aesni_ecb_dec) =20 /* @@ -2481,7 +2481,7 @@ SYM_FUNC_START(aesni_cbc_enc) popl IVP #endif FRAME_END - ret + RET SYM_FUNC_END(aesni_cbc_enc) =20 /* @@ -2574,7 +2574,7 @@ SYM_FUNC_START(aesni_cbc_dec) popl IVP #endif FRAME_END - ret + RET SYM_FUNC_END(aesni_cbc_dec) =20 #ifdef __x86_64__ @@ -2602,7 +2602,7 @@ SYM_FUNC_START_LOCAL(_aesni_inc_init) mov $1, TCTR_LOW movq TCTR_LOW, INC movq CTR, TCTR_LOW - ret + RET SYM_FUNC_END(_aesni_inc_init) =20 /* @@ -2630,7 +2630,7 @@ SYM_FUNC_START_LOCAL(_aesni_inc) .Linc_low: movaps CTR, IV pshufb BSWAP_MASK, IV - ret + RET SYM_FUNC_END(_aesni_inc) =20 /* @@ -2693,7 +2693,7 @@ SYM_FUNC_START(aesni_ctr_enc) movups IV, (IVP) .Lctr_enc_just_ret: FRAME_END - ret + RET SYM_FUNC_END(aesni_ctr_enc) =20 /* @@ -2778,7 +2778,7 @@ SYM_FUNC_START(aesni_xts_encrypt) movups IV, (IVP) =20 FRAME_END - ret + RET SYM_FUNC_END(aesni_xts_encrypt) =20 /* @@ -2846,7 +2846,7 @@ SYM_FUNC_START(aesni_xts_decrypt) movups IV, (IVP) =20 FRAME_END - ret + RET SYM_FUNC_END(aesni_xts_decrypt) =20 #endif --- a/arch/x86/crypto/aesni-intel_avx-x86_64.S +++ b/arch/x86/crypto/aesni-intel_avx-x86_64.S @@ -1777,7 +1777,7 @@ SYM_FUNC_START(aesni_gcm_init_avx_gen2) FUNC_SAVE INIT GHASH_MUL_AVX, PRECOMPUTE_AVX FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_init_avx_gen2) =20 ##########################################################################= ##### @@ -1798,15 +1798,15 @@ SYM_FUNC_START(aesni_gcm_enc_update_avx_ # must be 192 GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GH= ASH_LAST_8_AVX, GHASH_MUL_AVX, ENC, 11 FUNC_RESTORE - ret + RET key_128_enc_update: GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GH= ASH_LAST_8_AVX, GHASH_MUL_AVX, ENC, 9 FUNC_RESTORE - ret + RET key_256_enc_update: GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GH= ASH_LAST_8_AVX, GHASH_MUL_AVX, ENC, 13 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_enc_update_avx_gen2) =20 ##########################################################################= ##### @@ -1827,15 +1827,15 @@ SYM_FUNC_START(aesni_gcm_dec_update_avx_ # must be 192 GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GH= ASH_LAST_8_AVX, GHASH_MUL_AVX, DEC, 11 FUNC_RESTORE - ret + RET key_128_dec_update: GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GH= ASH_LAST_8_AVX, GHASH_MUL_AVX, DEC, 9 FUNC_RESTORE - ret + RET key_256_dec_update: GCM_ENC_DEC INITIAL_BLOCKS_AVX, GHASH_8_ENCRYPT_8_PARALLEL_AVX, GH= ASH_LAST_8_AVX, GHASH_MUL_AVX, DEC, 13 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_dec_update_avx_gen2) =20 ##########################################################################= ##### @@ -1856,15 +1856,15 @@ SYM_FUNC_START(aesni_gcm_finalize_avx_ge # must be 192 GCM_COMPLETE GHASH_MUL_AVX, 11, arg3, arg4 FUNC_RESTORE - ret + RET key_128_finalize: GCM_COMPLETE GHASH_MUL_AVX, 9, arg3, arg4 FUNC_RESTORE - ret + RET key_256_finalize: GCM_COMPLETE GHASH_MUL_AVX, 13, arg3, arg4 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_finalize_avx_gen2) =20 ##########################################################################= ##### @@ -2745,7 +2745,7 @@ SYM_FUNC_START(aesni_gcm_init_avx_gen4) FUNC_SAVE INIT GHASH_MUL_AVX2, PRECOMPUTE_AVX2 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_init_avx_gen4) =20 ##########################################################################= ##### @@ -2766,15 +2766,15 @@ SYM_FUNC_START(aesni_gcm_enc_update_avx_ # must be 192 GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, = GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, ENC, 11 FUNC_RESTORE - ret + RET key_128_enc_update4: GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, = GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, ENC, 9 FUNC_RESTORE - ret + RET key_256_enc_update4: GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, = GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, ENC, 13 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_enc_update_avx_gen4) =20 ##########################################################################= ##### @@ -2795,15 +2795,15 @@ SYM_FUNC_START(aesni_gcm_dec_update_avx_ # must be 192 GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, = GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, DEC, 11 FUNC_RESTORE - ret + RET key_128_dec_update4: GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, = GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, DEC, 9 FUNC_RESTORE - ret + RET key_256_dec_update4: GCM_ENC_DEC INITIAL_BLOCKS_AVX2, GHASH_8_ENCRYPT_8_PARALLEL_AVX2, = GHASH_LAST_8_AVX2, GHASH_MUL_AVX2, DEC, 13 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_dec_update_avx_gen4) =20 ##########################################################################= ##### @@ -2824,13 +2824,13 @@ SYM_FUNC_START(aesni_gcm_finalize_avx_ge # must be 192 GCM_COMPLETE GHASH_MUL_AVX2, 11, arg3, arg4 FUNC_RESTORE - ret + RET key_128_finalize4: GCM_COMPLETE GHASH_MUL_AVX2, 9, arg3, arg4 FUNC_RESTORE - ret + RET key_256_finalize4: GCM_COMPLETE GHASH_MUL_AVX2, 13, arg3, arg4 FUNC_RESTORE - ret + RET SYM_FUNC_END(aesni_gcm_finalize_avx_gen4) --- a/arch/x86/crypto/blake2s-core.S +++ b/arch/x86/crypto/blake2s-core.S @@ -171,7 +171,7 @@ SYM_FUNC_START(blake2s_compress_ssse3) movdqu %xmm1,0x10(%rdi) movdqu %xmm14,0x20(%rdi) .Lendofloop: - ret + RET SYM_FUNC_END(blake2s_compress_ssse3) =20 #ifdef CONFIG_AS_AVX512 @@ -251,6 +251,6 @@ SYM_FUNC_START(blake2s_compress_avx512) vmovdqu %xmm1,0x10(%rdi) vmovdqu %xmm4,0x20(%rdi) vzeroupper - retq + RET SYM_FUNC_END(blake2s_compress_avx512) #endif /* CONFIG_AS_AVX512 */ --- a/arch/x86/crypto/blowfish-x86_64-asm_64.S +++ b/arch/x86/crypto/blowfish-x86_64-asm_64.S @@ -135,10 +135,10 @@ SYM_FUNC_START(__blowfish_enc_blk) jnz .L__enc_xor; =20 write_block(); - ret; + RET; .L__enc_xor: xor_block(); - ret; + RET; SYM_FUNC_END(__blowfish_enc_blk) =20 SYM_FUNC_START(blowfish_dec_blk) @@ -170,7 +170,7 @@ SYM_FUNC_START(blowfish_dec_blk) =20 movq %r11, %r12; =20 - ret; + RET; SYM_FUNC_END(blowfish_dec_blk) =20 /********************************************************************** @@ -322,14 +322,14 @@ SYM_FUNC_START(__blowfish_enc_blk_4way) =20 popq %rbx; popq %r12; - ret; + RET; =20 .L__enc_xor4: xor_block4(); =20 popq %rbx; popq %r12; - ret; + RET; SYM_FUNC_END(__blowfish_enc_blk_4way) =20 SYM_FUNC_START(blowfish_dec_blk_4way) @@ -364,5 +364,5 @@ SYM_FUNC_START(blowfish_dec_blk_4way) popq %rbx; popq %r12; =20 - ret; + RET; SYM_FUNC_END(blowfish_dec_blk_4way) --- a/arch/x86/crypto/camellia-aesni-avx-asm_64.S +++ b/arch/x86/crypto/camellia-aesni-avx-asm_64.S @@ -193,7 +193,7 @@ SYM_FUNC_START_LOCAL(roundsm16_x0_x1_x2_ roundsm16(%xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5, %xmm6, %xmm7, %xmm8, %xmm9, %xmm10, %xmm11, %xmm12, %xmm13, %xmm14, %xmm15, %rcx, (%r9)); - ret; + RET; SYM_FUNC_END(roundsm16_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) =20 .align 8 @@ -201,7 +201,7 @@ SYM_FUNC_START_LOCAL(roundsm16_x4_x5_x6_ roundsm16(%xmm4, %xmm5, %xmm6, %xmm7, %xmm0, %xmm1, %xmm2, %xmm3, %xmm12, %xmm13, %xmm14, %xmm15, %xmm8, %xmm9, %xmm10, %xmm11, %rax, (%r9)); - ret; + RET; SYM_FUNC_END(roundsm16_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) =20 /* @@ -787,7 +787,7 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk1 %xmm15, (key_table)(CTX, %r8, 8), (%rax), 1 * 16(%rax)); =20 FRAME_END - ret; + RET; =20 .align 8 .Lenc_max32: @@ -874,7 +874,7 @@ SYM_FUNC_START_LOCAL(__camellia_dec_blk1 %xmm15, (key_table)(CTX), (%rax), 1 * 16(%rax)); =20 FRAME_END - ret; + RET; =20 .align 8 .Ldec_max32: @@ -915,7 +915,7 @@ SYM_FUNC_START(camellia_ecb_enc_16way) %xmm8, %rsi); =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_ecb_enc_16way) =20 SYM_FUNC_START(camellia_ecb_dec_16way) @@ -945,7 +945,7 @@ SYM_FUNC_START(camellia_ecb_dec_16way) %xmm8, %rsi); =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_ecb_dec_16way) =20 SYM_FUNC_START(camellia_cbc_dec_16way) @@ -996,7 +996,7 @@ SYM_FUNC_START(camellia_cbc_dec_16way) %xmm8, %rsi); =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_cbc_dec_16way) =20 #define inc_le128(x, minus_one, tmp) \ @@ -1109,7 +1109,7 @@ SYM_FUNC_START(camellia_ctr_16way) %xmm8, %rsi); =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_ctr_16way) =20 #define gf128mul_x_ble(iv, mask, tmp) \ @@ -1253,7 +1253,7 @@ SYM_FUNC_START_LOCAL(camellia_xts_crypt_ %xmm8, %rsi); =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_xts_crypt_16way) =20 SYM_FUNC_START(camellia_xts_enc_16way) --- a/arch/x86/crypto/camellia-aesni-avx2-asm_64.S +++ b/arch/x86/crypto/camellia-aesni-avx2-asm_64.S @@ -227,7 +227,7 @@ SYM_FUNC_START_LOCAL(roundsm32_x0_x1_x2_ roundsm32(%ymm0, %ymm1, %ymm2, %ymm3, %ymm4, %ymm5, %ymm6, %ymm7, %ymm8, %ymm9, %ymm10, %ymm11, %ymm12, %ymm13, %ymm14, %ymm15, %rcx, (%r9)); - ret; + RET; SYM_FUNC_END(roundsm32_x0_x1_x2_x3_x4_x5_x6_x7_y0_y1_y2_y3_y4_y5_y6_y7_cd) =20 .align 8 @@ -235,7 +235,7 @@ SYM_FUNC_START_LOCAL(roundsm32_x4_x5_x6_ roundsm32(%ymm4, %ymm5, %ymm6, %ymm7, %ymm0, %ymm1, %ymm2, %ymm3, %ymm12, %ymm13, %ymm14, %ymm15, %ymm8, %ymm9, %ymm10, %ymm11, %rax, (%r9)); - ret; + RET; SYM_FUNC_END(roundsm32_x4_x5_x6_x7_x0_x1_x2_x3_y4_y5_y6_y7_y0_y1_y2_y3_ab) =20 /* @@ -825,7 +825,7 @@ SYM_FUNC_START_LOCAL(__camellia_enc_blk3 %ymm15, (key_table)(CTX, %r8, 8), (%rax), 1 * 32(%rax)); =20 FRAME_END - ret; + RET; =20 .align 8 .Lenc_max32: @@ -912,7 +912,7 @@ SYM_FUNC_START_LOCAL(__camellia_dec_blk3 %ymm15, (key_table)(CTX), (%rax), 1 * 32(%rax)); =20 FRAME_END - ret; + RET; =20 .align 8 .Ldec_max32: @@ -957,7 +957,7 @@ SYM_FUNC_START(camellia_ecb_enc_32way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_ecb_enc_32way) =20 SYM_FUNC_START(camellia_ecb_dec_32way) @@ -991,7 +991,7 @@ SYM_FUNC_START(camellia_ecb_dec_32way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_ecb_dec_32way) =20 SYM_FUNC_START(camellia_cbc_dec_32way) @@ -1059,7 +1059,7 @@ SYM_FUNC_START(camellia_cbc_dec_32way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_cbc_dec_32way) =20 #define inc_le128(x, minus_one, tmp) \ @@ -1199,7 +1199,7 @@ SYM_FUNC_START(camellia_ctr_32way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_ctr_32way) =20 #define gf128mul_x_ble(iv, mask, tmp) \ @@ -1366,7 +1366,7 @@ SYM_FUNC_START_LOCAL(camellia_xts_crypt_ vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(camellia_xts_crypt_32way) =20 SYM_FUNC_START(camellia_xts_enc_32way) --- a/arch/x86/crypto/camellia-x86_64-asm_64.S +++ b/arch/x86/crypto/camellia-x86_64-asm_64.S @@ -213,13 +213,13 @@ SYM_FUNC_START(__camellia_enc_blk) enc_outunpack(mov, RT1); =20 movq RR12, %r12; - ret; + RET; =20 .L__enc_xor: enc_outunpack(xor, RT1); =20 movq RR12, %r12; - ret; + RET; SYM_FUNC_END(__camellia_enc_blk) =20 SYM_FUNC_START(camellia_dec_blk) @@ -257,7 +257,7 @@ SYM_FUNC_START(camellia_dec_blk) dec_outunpack(); =20 movq RR12, %r12; - ret; + RET; SYM_FUNC_END(camellia_dec_blk) =20 /********************************************************************** @@ -448,14 +448,14 @@ SYM_FUNC_START(__camellia_enc_blk_2way) =20 movq RR12, %r12; popq %rbx; - ret; + RET; =20 .L__enc2_xor: enc_outunpack2(xor, RT2); =20 movq RR12, %r12; popq %rbx; - ret; + RET; SYM_FUNC_END(__camellia_enc_blk_2way) =20 SYM_FUNC_START(camellia_dec_blk_2way) @@ -495,5 +495,5 @@ SYM_FUNC_START(camellia_dec_blk_2way) =20 movq RR12, %r12; movq RXOR, %rbx; - ret; + RET; SYM_FUNC_END(camellia_dec_blk_2way) --- a/arch/x86/crypto/cast5-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/cast5-avx-x86_64-asm_64.S @@ -279,7 +279,7 @@ SYM_FUNC_START_LOCAL(__cast5_enc_blk16) outunpack_blocks(RR3, RL3, RTMP, RX, RKM); outunpack_blocks(RR4, RL4, RTMP, RX, RKM); =20 - ret; + RET; SYM_FUNC_END(__cast5_enc_blk16) =20 .align 16 @@ -352,7 +352,7 @@ SYM_FUNC_START_LOCAL(__cast5_dec_blk16) outunpack_blocks(RR3, RL3, RTMP, RX, RKM); outunpack_blocks(RR4, RL4, RTMP, RX, RKM); =20 - ret; + RET; =20 .L__skip_dec: vpsrldq $4, RKR, RKR; @@ -393,7 +393,7 @@ SYM_FUNC_START(cast5_ecb_enc_16way) =20 popq %r15; FRAME_END - ret; + RET; SYM_FUNC_END(cast5_ecb_enc_16way) =20 SYM_FUNC_START(cast5_ecb_dec_16way) @@ -431,7 +431,7 @@ SYM_FUNC_START(cast5_ecb_dec_16way) =20 popq %r15; FRAME_END - ret; + RET; SYM_FUNC_END(cast5_ecb_dec_16way) =20 SYM_FUNC_START(cast5_cbc_dec_16way) @@ -483,7 +483,7 @@ SYM_FUNC_START(cast5_cbc_dec_16way) popq %r15; popq %r12; FRAME_END - ret; + RET; SYM_FUNC_END(cast5_cbc_dec_16way) =20 SYM_FUNC_START(cast5_ctr_16way) @@ -559,5 +559,5 @@ SYM_FUNC_START(cast5_ctr_16way) popq %r15; popq %r12; FRAME_END - ret; + RET; SYM_FUNC_END(cast5_ctr_16way) --- a/arch/x86/crypto/cast6-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/cast6-avx-x86_64-asm_64.S @@ -291,7 +291,7 @@ SYM_FUNC_START_LOCAL(__cast6_enc_blk8) outunpack_blocks(RA1, RB1, RC1, RD1, RTMP, RX, RKRF, RKM); outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); =20 - ret; + RET; SYM_FUNC_END(__cast6_enc_blk8) =20 .align 8 @@ -338,7 +338,7 @@ SYM_FUNC_START_LOCAL(__cast6_dec_blk8) outunpack_blocks(RA1, RB1, RC1, RD1, RTMP, RX, RKRF, RKM); outunpack_blocks(RA2, RB2, RC2, RD2, RTMP, RX, RKRF, RKM); =20 - ret; + RET; SYM_FUNC_END(__cast6_dec_blk8) =20 SYM_FUNC_START(cast6_ecb_enc_8way) @@ -361,7 +361,7 @@ SYM_FUNC_START(cast6_ecb_enc_8way) =20 popq %r15; FRAME_END - ret; + RET; SYM_FUNC_END(cast6_ecb_enc_8way) =20 SYM_FUNC_START(cast6_ecb_dec_8way) @@ -384,7 +384,7 @@ SYM_FUNC_START(cast6_ecb_dec_8way) =20 popq %r15; FRAME_END - ret; + RET; SYM_FUNC_END(cast6_ecb_dec_8way) =20 SYM_FUNC_START(cast6_cbc_dec_8way) @@ -410,7 +410,7 @@ SYM_FUNC_START(cast6_cbc_dec_8way) popq %r15; popq %r12; FRAME_END - ret; + RET; SYM_FUNC_END(cast6_cbc_dec_8way) =20 SYM_FUNC_START(cast6_ctr_8way) @@ -438,7 +438,7 @@ SYM_FUNC_START(cast6_ctr_8way) popq %r15; popq %r12; FRAME_END - ret; + RET; SYM_FUNC_END(cast6_ctr_8way) =20 SYM_FUNC_START(cast6_xts_enc_8way) @@ -465,7 +465,7 @@ SYM_FUNC_START(cast6_xts_enc_8way) =20 popq %r15; FRAME_END - ret; + RET; SYM_FUNC_END(cast6_xts_enc_8way) =20 SYM_FUNC_START(cast6_xts_dec_8way) @@ -492,5 +492,5 @@ SYM_FUNC_START(cast6_xts_dec_8way) =20 popq %r15; FRAME_END - ret; + RET; SYM_FUNC_END(cast6_xts_dec_8way) --- a/arch/x86/crypto/chacha-avx2-x86_64.S +++ b/arch/x86/crypto/chacha-avx2-x86_64.S @@ -193,7 +193,7 @@ SYM_FUNC_START(chacha_2block_xor_avx2) =20 .Ldone2: vzeroupper - ret + RET =20 .Lxorpart2: # xor remaining bytes from partial register into output @@ -498,7 +498,7 @@ SYM_FUNC_START(chacha_4block_xor_avx2) =20 .Ldone4: vzeroupper - ret + RET =20 .Lxorpart4: # xor remaining bytes from partial register into output @@ -992,7 +992,7 @@ SYM_FUNC_START(chacha_8block_xor_avx2) .Ldone8: vzeroupper lea -8(%r10),%rsp - ret + RET =20 .Lxorpart8: # xor remaining bytes from partial register into output --- a/arch/x86/crypto/chacha-avx512vl-x86_64.S +++ b/arch/x86/crypto/chacha-avx512vl-x86_64.S @@ -166,7 +166,7 @@ SYM_FUNC_START(chacha_2block_xor_avx512v =20 .Ldone2: vzeroupper - ret + RET =20 .Lxorpart2: # xor remaining bytes from partial register into output @@ -432,7 +432,7 @@ SYM_FUNC_START(chacha_4block_xor_avx512v =20 .Ldone4: vzeroupper - ret + RET =20 .Lxorpart4: # xor remaining bytes from partial register into output @@ -812,7 +812,7 @@ SYM_FUNC_START(chacha_8block_xor_avx512v =20 .Ldone8: vzeroupper - ret + RET =20 .Lxorpart8: # xor remaining bytes from partial register into output --- a/arch/x86/crypto/chacha-ssse3-x86_64.S +++ b/arch/x86/crypto/chacha-ssse3-x86_64.S @@ -108,7 +108,7 @@ SYM_FUNC_START_LOCAL(chacha_permute) sub $2,%r8d jnz .Ldoubleround =20 - ret + RET SYM_FUNC_END(chacha_permute) =20 SYM_FUNC_START(chacha_block_xor_ssse3) @@ -166,7 +166,7 @@ SYM_FUNC_START(chacha_block_xor_ssse3) =20 .Ldone: FRAME_END - ret + RET =20 .Lxorpart: # xor remaining bytes from partial register into output @@ -217,7 +217,7 @@ SYM_FUNC_START(hchacha_block_ssse3) movdqu %xmm3,0x10(%rsi) =20 FRAME_END - ret + RET SYM_FUNC_END(hchacha_block_ssse3) =20 SYM_FUNC_START(chacha_4block_xor_ssse3) @@ -762,7 +762,7 @@ SYM_FUNC_START(chacha_4block_xor_ssse3) =20 .Ldone4: lea -8(%r10),%rsp - ret + RET =20 .Lxorpart4: # xor remaining bytes from partial register into output --- a/arch/x86/crypto/crc32-pclmul_asm.S +++ b/arch/x86/crypto/crc32-pclmul_asm.S @@ -236,5 +236,5 @@ fold_64: pxor %xmm2, %xmm1 pextrd $0x01, %xmm1, %eax =20 - ret + RET SYM_FUNC_END(crc32_pclmul_le_16) --- a/arch/x86/crypto/crc32c-pcl-intel-asm_64.S +++ b/arch/x86/crypto/crc32c-pcl-intel-asm_64.S @@ -309,7 +309,7 @@ do_return: popq %rsi popq %rdi popq %rbx - ret + RET SYM_FUNC_END(crc_pcl) =20 .section .rodata, "a", @progbits --- a/arch/x86/crypto/crct10dif-pcl-asm_64.S +++ b/arch/x86/crypto/crct10dif-pcl-asm_64.S @@ -257,7 +257,7 @@ SYM_FUNC_START(crc_t10dif_pcl) # Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of xmm0. =20 pextrw $0, %xmm0, %eax - ret + RET =20 .align 16 .Lless_than_256_bytes: --- a/arch/x86/crypto/des3_ede-asm_64.S +++ b/arch/x86/crypto/des3_ede-asm_64.S @@ -243,7 +243,7 @@ SYM_FUNC_START(des3_ede_x86_64_crypt_blk popq %r12; popq %rbx; =20 - ret; + RET; SYM_FUNC_END(des3_ede_x86_64_crypt_blk) =20 /*********************************************************************** @@ -528,7 +528,7 @@ SYM_FUNC_START(des3_ede_x86_64_crypt_blk popq %r12; popq %rbx; =20 - ret; + RET; SYM_FUNC_END(des3_ede_x86_64_crypt_blk_3way) =20 .section .rodata, "a", @progbits --- a/arch/x86/crypto/ghash-clmulni-intel_asm.S +++ b/arch/x86/crypto/ghash-clmulni-intel_asm.S @@ -85,7 +85,7 @@ SYM_FUNC_START_LOCAL(__clmul_gf128mul_bl psrlq $1, T2 pxor T2, T1 pxor T1, DATA - ret + RET SYM_FUNC_END(__clmul_gf128mul_ble) =20 /* void clmul_ghash_mul(char *dst, const u128 *shash) */ @@ -99,7 +99,7 @@ SYM_FUNC_START(clmul_ghash_mul) pshufb BSWAP, DATA movups DATA, (%rdi) FRAME_END - ret + RET SYM_FUNC_END(clmul_ghash_mul) =20 /* @@ -128,5 +128,5 @@ SYM_FUNC_START(clmul_ghash_update) movups DATA, (%rdi) .Lupdate_just_ret: FRAME_END - ret + RET SYM_FUNC_END(clmul_ghash_update) --- a/arch/x86/crypto/nh-avx2-x86_64.S +++ b/arch/x86/crypto/nh-avx2-x86_64.S @@ -153,5 +153,5 @@ SYM_FUNC_START(nh_avx2) vpaddq T1, T0, T0 vpaddq T4, T0, T0 vmovdqu T0, (HASH) - ret + RET SYM_FUNC_END(nh_avx2) --- a/arch/x86/crypto/nh-sse2-x86_64.S +++ b/arch/x86/crypto/nh-sse2-x86_64.S @@ -119,5 +119,5 @@ SYM_FUNC_START(nh_sse2) paddq PASS2_SUMS, T1 movdqu T0, 0x00(HASH) movdqu T1, 0x10(HASH) - ret + RET SYM_FUNC_END(nh_sse2) --- a/arch/x86/crypto/serpent-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/serpent-avx-x86_64-asm_64.S @@ -605,7 +605,7 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk8_ write_blocks(RA1, RB1, RC1, RD1, RK0, RK1, RK2); write_blocks(RA2, RB2, RC2, RD2, RK0, RK1, RK2); =20 - ret; + RET; SYM_FUNC_END(__serpent_enc_blk8_avx) =20 .align 8 @@ -659,7 +659,7 @@ SYM_FUNC_START_LOCAL(__serpent_dec_blk8_ write_blocks(RC1, RD1, RB1, RE1, RK0, RK1, RK2); write_blocks(RC2, RD2, RB2, RE2, RK0, RK1, RK2); =20 - ret; + RET; SYM_FUNC_END(__serpent_dec_blk8_avx) =20 SYM_FUNC_START(serpent_ecb_enc_8way_avx) @@ -677,7 +677,7 @@ SYM_FUNC_START(serpent_ecb_enc_8way_avx) store_8way(%rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_ecb_enc_8way_avx) =20 SYM_FUNC_START(serpent_ecb_dec_8way_avx) @@ -695,7 +695,7 @@ SYM_FUNC_START(serpent_ecb_dec_8way_avx) store_8way(%rsi, RC1, RD1, RB1, RE1, RC2, RD2, RB2, RE2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_ecb_dec_8way_avx) =20 SYM_FUNC_START(serpent_cbc_dec_8way_avx) @@ -713,7 +713,7 @@ SYM_FUNC_START(serpent_cbc_dec_8way_avx) store_cbc_8way(%rdx, %rsi, RC1, RD1, RB1, RE1, RC2, RD2, RB2, RE2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_cbc_dec_8way_avx) =20 SYM_FUNC_START(serpent_ctr_8way_avx) @@ -733,7 +733,7 @@ SYM_FUNC_START(serpent_ctr_8way_avx) store_ctr_8way(%rdx, %rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_ctr_8way_avx) =20 SYM_FUNC_START(serpent_xts_enc_8way_avx) @@ -755,7 +755,7 @@ SYM_FUNC_START(serpent_xts_enc_8way_avx) store_xts_8way(%rsi, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_xts_enc_8way_avx) =20 SYM_FUNC_START(serpent_xts_dec_8way_avx) @@ -777,5 +777,5 @@ SYM_FUNC_START(serpent_xts_dec_8way_avx) store_xts_8way(%rsi, RC1, RD1, RB1, RE1, RC2, RD2, RB2, RE2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_xts_dec_8way_avx) --- a/arch/x86/crypto/serpent-avx2-asm_64.S +++ b/arch/x86/crypto/serpent-avx2-asm_64.S @@ -611,7 +611,7 @@ SYM_FUNC_START_LOCAL(__serpent_enc_blk16 write_blocks(RA1, RB1, RC1, RD1, RK0, RK1, RK2); write_blocks(RA2, RB2, RC2, RD2, RK0, RK1, RK2); =20 - ret; + RET; SYM_FUNC_END(__serpent_enc_blk16) =20 .align 8 @@ -665,7 +665,7 @@ SYM_FUNC_START_LOCAL(__serpent_dec_blk16 write_blocks(RC1, RD1, RB1, RE1, RK0, RK1, RK2); write_blocks(RC2, RD2, RB2, RE2, RK0, RK1, RK2); =20 - ret; + RET; SYM_FUNC_END(__serpent_dec_blk16) =20 SYM_FUNC_START(serpent_ecb_enc_16way) @@ -687,7 +687,7 @@ SYM_FUNC_START(serpent_ecb_enc_16way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_ecb_enc_16way) =20 SYM_FUNC_START(serpent_ecb_dec_16way) @@ -709,7 +709,7 @@ SYM_FUNC_START(serpent_ecb_dec_16way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_ecb_dec_16way) =20 SYM_FUNC_START(serpent_cbc_dec_16way) @@ -732,7 +732,7 @@ SYM_FUNC_START(serpent_cbc_dec_16way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_cbc_dec_16way) =20 SYM_FUNC_START(serpent_ctr_16way) @@ -757,7 +757,7 @@ SYM_FUNC_START(serpent_ctr_16way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_ctr_16way) =20 SYM_FUNC_START(serpent_xts_enc_16way) @@ -783,7 +783,7 @@ SYM_FUNC_START(serpent_xts_enc_16way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_xts_enc_16way) =20 SYM_FUNC_START(serpent_xts_dec_16way) @@ -809,5 +809,5 @@ SYM_FUNC_START(serpent_xts_dec_16way) vzeroupper; =20 FRAME_END - ret; + RET; SYM_FUNC_END(serpent_xts_dec_16way) --- a/arch/x86/crypto/serpent-sse2-i586-asm_32.S +++ b/arch/x86/crypto/serpent-sse2-i586-asm_32.S @@ -553,12 +553,12 @@ SYM_FUNC_START(__serpent_enc_blk_4way) =20 write_blocks(%eax, RA, RB, RC, RD, RT0, RT1, RE); =20 - ret; + RET; =20 .L__enc_xor4: xor_blocks(%eax, RA, RB, RC, RD, RT0, RT1, RE); =20 - ret; + RET; SYM_FUNC_END(__serpent_enc_blk_4way) =20 SYM_FUNC_START(serpent_dec_blk_4way) @@ -612,5 +612,5 @@ SYM_FUNC_START(serpent_dec_blk_4way) movl arg_dst(%esp), %eax; write_blocks(%eax, RC, RD, RB, RE, RT0, RT1, RA); =20 - ret; + RET; SYM_FUNC_END(serpent_dec_blk_4way) --- a/arch/x86/crypto/serpent-sse2-x86_64-asm_64.S +++ b/arch/x86/crypto/serpent-sse2-x86_64-asm_64.S @@ -675,13 +675,13 @@ SYM_FUNC_START(__serpent_enc_blk_8way) write_blocks(%rsi, RA1, RB1, RC1, RD1, RK0, RK1, RK2); write_blocks(%rax, RA2, RB2, RC2, RD2, RK0, RK1, RK2); =20 - ret; + RET; =20 .L__enc_xor8: xor_blocks(%rsi, RA1, RB1, RC1, RD1, RK0, RK1, RK2); xor_blocks(%rax, RA2, RB2, RC2, RD2, RK0, RK1, RK2); =20 - ret; + RET; SYM_FUNC_END(__serpent_enc_blk_8way) =20 SYM_FUNC_START(serpent_dec_blk_8way) @@ -735,5 +735,5 @@ SYM_FUNC_START(serpent_dec_blk_8way) write_blocks(%rsi, RC1, RD1, RB1, RE1, RK0, RK1, RK2); write_blocks(%rax, RC2, RD2, RB2, RE2, RK0, RK1, RK2); =20 - ret; + RET; SYM_FUNC_END(serpent_dec_blk_8way) --- a/arch/x86/crypto/sha1_avx2_x86_64_asm.S +++ b/arch/x86/crypto/sha1_avx2_x86_64_asm.S @@ -674,7 +674,7 @@ _loop3: pop %r12 pop %rbx =20 - ret + RET =20 SYM_FUNC_END(\name) .endm --- a/arch/x86/crypto/sha1_ni_asm.S +++ b/arch/x86/crypto/sha1_ni_asm.S @@ -290,7 +290,7 @@ SYM_FUNC_START(sha1_ni_transform) .Ldone_hash: mov RSPSAVE, %rsp =20 - ret + RET SYM_FUNC_END(sha1_ni_transform) =20 .section .rodata.cst16.PSHUFFLE_BYTE_FLIP_MASK, "aM", @progbits, 16 --- a/arch/x86/crypto/sha1_ssse3_asm.S +++ b/arch/x86/crypto/sha1_ssse3_asm.S @@ -99,7 +99,7 @@ pop %rbp pop %r12 pop %rbx - ret + RET =20 SYM_FUNC_END(\name) .endm --- a/arch/x86/crypto/sha256-avx-asm.S +++ b/arch/x86/crypto/sha256-avx-asm.S @@ -458,7 +458,7 @@ done_hash: popq %r13 popq %r12 popq %rbx - ret + RET SYM_FUNC_END(sha256_transform_avx) =20 .section .rodata.cst256.K256, "aM", @progbits, 256 --- a/arch/x86/crypto/sha256-avx2-asm.S +++ b/arch/x86/crypto/sha256-avx2-asm.S @@ -711,7 +711,7 @@ done_hash: popq %r13 popq %r12 popq %rbx - ret + RET SYM_FUNC_END(sha256_transform_rorx) =20 .section .rodata.cst512.K256, "aM", @progbits, 512 --- a/arch/x86/crypto/sha256-ssse3-asm.S +++ b/arch/x86/crypto/sha256-ssse3-asm.S @@ -472,7 +472,7 @@ done_hash: popq %r12 popq %rbx =20 - ret + RET SYM_FUNC_END(sha256_transform_ssse3) =20 .section .rodata.cst256.K256, "aM", @progbits, 256 --- a/arch/x86/crypto/sha256_ni_asm.S +++ b/arch/x86/crypto/sha256_ni_asm.S @@ -326,7 +326,7 @@ SYM_FUNC_START(sha256_ni_transform) =20 .Ldone_hash: =20 - ret + RET SYM_FUNC_END(sha256_ni_transform) =20 .section .rodata.cst256.K256, "aM", @progbits, 256 --- a/arch/x86/crypto/sha512-avx-asm.S +++ b/arch/x86/crypto/sha512-avx-asm.S @@ -364,7 +364,7 @@ updateblock: mov frame_RSPSAVE(%rsp), %rsp =20 nowork: - ret + RET SYM_FUNC_END(sha512_transform_avx) =20 ######################################################################## --- a/arch/x86/crypto/sha512-avx2-asm.S +++ b/arch/x86/crypto/sha512-avx2-asm.S @@ -681,7 +681,7 @@ done_hash: =20 # Restore Stack Pointer mov frame_RSPSAVE(%rsp), %rsp - ret + RET SYM_FUNC_END(sha512_transform_rorx) =20 ######################################################################## --- a/arch/x86/crypto/sha512-ssse3-asm.S +++ b/arch/x86/crypto/sha512-ssse3-asm.S @@ -366,7 +366,7 @@ updateblock: mov frame_RSPSAVE(%rsp), %rsp =20 nowork: - ret + RET SYM_FUNC_END(sha512_transform_ssse3) =20 ######################################################################## --- a/arch/x86/crypto/twofish-avx-x86_64-asm_64.S +++ b/arch/x86/crypto/twofish-avx-x86_64-asm_64.S @@ -272,7 +272,7 @@ SYM_FUNC_START_LOCAL(__twofish_enc_blk8) outunpack_blocks(RC1, RD1, RA1, RB1, RK1, RX0, RY0, RK2); outunpack_blocks(RC2, RD2, RA2, RB2, RK1, RX0, RY0, RK2); =20 - ret; + RET; SYM_FUNC_END(__twofish_enc_blk8) =20 .align 8 @@ -312,7 +312,7 @@ SYM_FUNC_START_LOCAL(__twofish_dec_blk8) outunpack_blocks(RA1, RB1, RC1, RD1, RK1, RX0, RY0, RK2); outunpack_blocks(RA2, RB2, RC2, RD2, RK1, RX0, RY0, RK2); =20 - ret; + RET; SYM_FUNC_END(__twofish_dec_blk8) =20 SYM_FUNC_START(twofish_ecb_enc_8way) @@ -332,7 +332,7 @@ SYM_FUNC_START(twofish_ecb_enc_8way) store_8way(%r11, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(twofish_ecb_enc_8way) =20 SYM_FUNC_START(twofish_ecb_dec_8way) @@ -352,7 +352,7 @@ SYM_FUNC_START(twofish_ecb_dec_8way) store_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(twofish_ecb_dec_8way) =20 SYM_FUNC_START(twofish_cbc_dec_8way) @@ -377,7 +377,7 @@ SYM_FUNC_START(twofish_cbc_dec_8way) popq %r12; =20 FRAME_END - ret; + RET; SYM_FUNC_END(twofish_cbc_dec_8way) =20 SYM_FUNC_START(twofish_ctr_8way) @@ -404,7 +404,7 @@ SYM_FUNC_START(twofish_ctr_8way) popq %r12; =20 FRAME_END - ret; + RET; SYM_FUNC_END(twofish_ctr_8way) =20 SYM_FUNC_START(twofish_xts_enc_8way) @@ -428,7 +428,7 @@ SYM_FUNC_START(twofish_xts_enc_8way) store_xts_8way(%r11, RC1, RD1, RA1, RB1, RC2, RD2, RA2, RB2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(twofish_xts_enc_8way) =20 SYM_FUNC_START(twofish_xts_dec_8way) @@ -452,5 +452,5 @@ SYM_FUNC_START(twofish_xts_dec_8way) store_xts_8way(%r11, RA1, RB1, RC1, RD1, RA2, RB2, RC2, RD2); =20 FRAME_END - ret; + RET; SYM_FUNC_END(twofish_xts_dec_8way) --- a/arch/x86/crypto/twofish-i586-asm_32.S +++ b/arch/x86/crypto/twofish-i586-asm_32.S @@ -260,7 +260,7 @@ SYM_FUNC_START(twofish_enc_blk) pop %ebx pop %ebp mov $1, %eax - ret + RET SYM_FUNC_END(twofish_enc_blk) =20 SYM_FUNC_START(twofish_dec_blk) @@ -317,5 +317,5 @@ SYM_FUNC_START(twofish_dec_blk) pop %ebx pop %ebp mov $1, %eax - ret + RET SYM_FUNC_END(twofish_dec_blk) --- a/arch/x86/crypto/twofish-x86_64-asm_64-3way.S +++ b/arch/x86/crypto/twofish-x86_64-asm_64-3way.S @@ -258,7 +258,7 @@ SYM_FUNC_START(__twofish_enc_blk_3way) popq %rbx; popq %r12; popq %r13; - ret; + RET; =20 .L__enc_xor3: outunpack_enc3(xor); @@ -266,7 +266,7 @@ SYM_FUNC_START(__twofish_enc_blk_3way) popq %rbx; popq %r12; popq %r13; - ret; + RET; SYM_FUNC_END(__twofish_enc_blk_3way) =20 SYM_FUNC_START(twofish_dec_blk_3way) @@ -301,5 +301,5 @@ SYM_FUNC_START(twofish_dec_blk_3way) popq %rbx; popq %r12; popq %r13; - ret; + RET; SYM_FUNC_END(twofish_dec_blk_3way) --- a/arch/x86/crypto/twofish-x86_64-asm_64.S +++ b/arch/x86/crypto/twofish-x86_64-asm_64.S @@ -252,7 +252,7 @@ SYM_FUNC_START(twofish_enc_blk) =20 popq R1 movl $1,%eax - ret + RET SYM_FUNC_END(twofish_enc_blk) =20 SYM_FUNC_START(twofish_dec_blk) @@ -304,5 +304,5 @@ SYM_FUNC_START(twofish_dec_blk) =20 popq R1 movl $1,%eax - ret + RET SYM_FUNC_END(twofish_dec_blk) --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -821,7 +821,7 @@ SYM_FUNC_START(schedule_tail_wrapper) popl %eax =20 FRAME_END - ret + RET SYM_FUNC_END(schedule_tail_wrapper) .popsection =20 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -740,7 +740,7 @@ SYM_FUNC_START(asm_load_gs_index) 2: ALTERNATIVE "", "mfence", X86_BUG_SWAPGS_FENCE swapgs FRAME_END - ret + RET SYM_FUNC_END(asm_load_gs_index) EXPORT_SYMBOL(asm_load_gs_index) =20 @@ -799,7 +799,7 @@ SYM_INNER_LABEL(asm_call_irq_on_stack, S =20 /* Restore the previous stack pointer from RBP. */ leaveq - ret + RET SYM_FUNC_END(asm_call_on_stack) =20 #ifdef CONFIG_XEN_PV @@ -932,7 +932,7 @@ SYM_CODE_START_LOCAL(paranoid_entry) * is needed here. */ SAVE_AND_SET_GSBASE scratch_reg=3D%rax save_reg=3D%rbx - ret + RET =20 .Lparanoid_entry_checkgs: /* EBX =3D 1 -> kernel GSBASE active, no restore required */ @@ -953,7 +953,7 @@ SYM_CODE_START_LOCAL(paranoid_entry) .Lparanoid_kernel_gsbase: =20 FENCE_SWAPGS_KERNEL_ENTRY - ret + RET SYM_CODE_END(paranoid_entry) =20 /* @@ -1032,7 +1032,7 @@ SYM_CODE_START_LOCAL(error_entry) movq %rax, %rsp /* switch stack */ ENCODE_FRAME_POINTER pushq %r12 - ret + RET =20 /* * There are two places in the kernel that can potentially fault with @@ -1063,7 +1063,7 @@ SYM_CODE_START_LOCAL(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY - ret + RET =20 .Lbstep_iret: /* Fix truncated RIP */ --- a/arch/x86/entry/thunk_32.S +++ b/arch/x86/entry/thunk_32.S @@ -24,7 +24,7 @@ SYM_CODE_START_NOALIGN(\name) popl %edx popl %ecx popl %eax - ret + RET _ASM_NOKPROBE(\name) SYM_CODE_END(\name) .endm --- a/arch/x86/entry/thunk_64.S +++ b/arch/x86/entry/thunk_64.S @@ -55,7 +55,7 @@ SYM_CODE_START_LOCAL_NOALIGN(__thunk_res popq %rsi popq %rdi popq %rbp - ret + RET _ASM_NOKPROBE(__thunk_restore) SYM_CODE_END(__thunk_restore) #endif --- a/arch/x86/entry/vdso/vdso32/system_call.S +++ b/arch/x86/entry/vdso/vdso32/system_call.S @@ -78,7 +78,7 @@ SYM_INNER_LABEL(int80_landing_pad, SYM_L popl %ecx CFI_RESTORE ecx CFI_ADJUST_CFA_OFFSET -4 - ret + RET CFI_ENDPROC =20 .size __kernel_vsyscall,.-__kernel_vsyscall --- a/arch/x86/entry/vsyscall/vsyscall_emu_64.S +++ b/arch/x86/entry/vsyscall/vsyscall_emu_64.S @@ -19,17 +19,17 @@ __vsyscall_page: =20 mov $__NR_gettimeofday, %rax syscall - ret + RET =20 .balign 1024, 0xcc mov $__NR_time, %rax syscall - ret + RET =20 .balign 1024, 0xcc mov $__NR_getcpu, %rax syscall - ret + RET =20 .balign 4096, 0xcc =20 --- a/arch/x86/kernel/acpi/wakeup_32.S +++ b/arch/x86/kernel/acpi/wakeup_32.S @@ -60,7 +60,7 @@ save_registers: popl saved_context_eflags =20 movl $ret_point, saved_eip - ret + RET =20 =20 restore_registers: @@ -70,7 +70,7 @@ restore_registers: movl saved_context_edi, %edi pushl saved_context_eflags popfl - ret + RET =20 SYM_CODE_START(do_suspend_lowlevel) call save_processor_state @@ -86,7 +86,7 @@ SYM_CODE_START(do_suspend_lowlevel) ret_point: call restore_registers call restore_processor_state - ret + RET SYM_CODE_END(do_suspend_lowlevel) =20 .data --- a/arch/x86/kernel/ftrace_32.S +++ b/arch/x86/kernel/ftrace_32.S @@ -19,7 +19,7 @@ #endif =20 SYM_FUNC_START(__fentry__) - ret + RET SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) =20 @@ -84,7 +84,7 @@ ftrace_graph_call: =20 /* This is weak to keep gas from relaxing the jumps */ SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK) - ret + RET SYM_CODE_END(ftrace_caller) =20 SYM_CODE_START(ftrace_regs_caller) @@ -177,7 +177,7 @@ SYM_CODE_START(ftrace_graph_caller) popl %edx popl %ecx popl %eax - ret + RET SYM_CODE_END(ftrace_graph_caller) =20 .globl return_to_handler --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -132,7 +132,7 @@ #ifdef CONFIG_DYNAMIC_FTRACE =20 SYM_FUNC_START(__fentry__) - retq + RET SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) =20 @@ -170,10 +170,10 @@ SYM_INNER_LABEL(ftrace_graph_call, SYM_L =20 /* * This is weak to keep gas from relaxing the jumps. - * It is also used to copy the retq for trampolines. + * It is also used to copy the RET for trampolines. */ SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK) - retq + RET SYM_FUNC_END(ftrace_epilogue) =20 SYM_FUNC_START(ftrace_regs_caller) @@ -287,7 +287,7 @@ fgraph_trace: #endif =20 SYM_INNER_LABEL(ftrace_stub, SYM_L_GLOBAL) - retq + RET =20 trace: /* save_mcount_regs fills in first two parameters */ @@ -319,7 +319,7 @@ SYM_FUNC_START(ftrace_graph_caller) =20 restore_mcount_regs =20 - retq + RET SYM_FUNC_END(ftrace_graph_caller) =20 SYM_CODE_START(return_to_handler) --- a/arch/x86/kernel/head_32.S +++ b/arch/x86/kernel/head_32.S @@ -354,7 +354,7 @@ setup_once: #endif =20 andl $0,setup_once_ref /* Once is enough, thanks */ - ret + RET =20 SYM_FUNC_START(early_idt_handler_array) # 36(%esp) %eflags --- a/arch/x86/kernel/irqflags.S +++ b/arch/x86/kernel/irqflags.S @@ -10,7 +10,7 @@ SYM_FUNC_START(native_save_fl) pushf pop %_ASM_AX - ret + RET SYM_FUNC_END(native_save_fl) EXPORT_SYMBOL(native_save_fl) =20 @@ -21,6 +21,6 @@ EXPORT_SYMBOL(native_save_fl) SYM_FUNC_START(native_restore_fl) push %_ASM_ARG1 popf - ret + RET SYM_FUNC_END(native_restore_fl) EXPORT_SYMBOL(native_restore_fl) --- a/arch/x86/kernel/relocate_kernel_32.S +++ b/arch/x86/kernel/relocate_kernel_32.S @@ -91,7 +91,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel) movl %edi, %eax addl $(identity_mapped - relocate_kernel), %eax pushl %eax - ret + RET SYM_CODE_END(relocate_kernel) =20 SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) @@ -159,7 +159,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma xorl %edx, %edx xorl %esi, %esi xorl %ebp, %ebp - ret + RET 1: popl %edx movl CP_PA_SWAP_PAGE(%edi), %esp @@ -190,7 +190,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma movl %edi, %eax addl $(virtual_mapped - relocate_kernel), %eax pushl %eax - ret + RET SYM_CODE_END(identity_mapped) =20 SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped) @@ -208,7 +208,7 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_map popl %edi popl %esi popl %ebx - ret + RET SYM_CODE_END(virtual_mapped) =20 /* Do the copies */ @@ -271,7 +271,7 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages) popl %edi popl %ebx popl %ebp - ret + RET SYM_CODE_END(swap_pages) =20 .globl kexec_control_code_size --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -104,7 +104,7 @@ SYM_CODE_START_NOALIGN(relocate_kernel) /* jump to identity mapped page */ addq $(identity_mapped - relocate_kernel), %r8 pushq %r8 - ret + RET SYM_CODE_END(relocate_kernel) =20 SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) @@ -191,7 +191,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma xorl %r14d, %r14d xorl %r15d, %r15d =20 - ret + RET =20 1: popq %rdx @@ -210,7 +210,7 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma call swap_pages movq $virtual_mapped, %rax pushq %rax - ret + RET SYM_CODE_END(identity_mapped) =20 SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped) @@ -231,7 +231,7 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_map popq %r12 popq %rbp popq %rbx - ret + RET SYM_CODE_END(virtual_mapped) =20 /* Do the copies */ @@ -288,7 +288,7 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages) lea PAGE_SIZE(%rax), %rsi jmp 0b 3: - ret + RET SYM_CODE_END(swap_pages) =20 .globl kexec_control_code_size --- a/arch/x86/kernel/sev_verify_cbit.S +++ b/arch/x86/kernel/sev_verify_cbit.S @@ -85,5 +85,5 @@ SYM_FUNC_START(sev_verify_cbit) #endif /* Return page-table pointer */ movq %rdi, %rax - ret + RET SYM_FUNC_END(sev_verify_cbit) --- a/arch/x86/kernel/verify_cpu.S +++ b/arch/x86/kernel/verify_cpu.S @@ -132,9 +132,9 @@ SYM_FUNC_START_LOCAL(verify_cpu) .Lverify_cpu_no_longmode: popf # Restore caller passed flags movl $1,%eax - ret + RET .Lverify_cpu_sse_ok: popf # Restore caller passed flags xorl %eax, %eax - ret + RET SYM_FUNC_END(verify_cpu) --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -166,5 +166,5 @@ SYM_FUNC_START(__svm_vcpu_run) pop %edi #endif pop %_ASM_BP - ret + RET SYM_FUNC_END(__svm_vcpu_run) --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -49,14 +49,14 @@ SYM_FUNC_START_LOCAL(vmx_vmenter) je 2f =20 1: vmresume - ret + RET =20 2: vmlaunch - ret + RET =20 3: cmpb $0, kvm_rebooting je 4f - ret + RET 4: ud2 =20 _ASM_EXTABLE(1b, 3b) @@ -89,7 +89,7 @@ SYM_FUNC_START(vmx_vmexit) pop %_ASM_AX .Lvmexit_skip_rsb: #endif - ret + RET SYM_FUNC_END(vmx_vmexit) =20 /** @@ -228,7 +228,7 @@ SYM_FUNC_START(__vmx_vcpu_run) pop %edi #endif pop %_ASM_BP - ret + RET =20 /* VM-Fail. Out-of-line to avoid a taken Jcc after VM-Exit. */ 2: mov $1, %eax @@ -293,7 +293,7 @@ SYM_FUNC_START(vmread_error_trampoline) pop %_ASM_AX pop %_ASM_BP =20 - ret + RET SYM_FUNC_END(vmread_error_trampoline) =20 SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff) @@ -326,5 +326,5 @@ SYM_FUNC_START(vmx_do_interrupt_nmi_irqo */ mov %_ASM_BP, %_ASM_SP pop %_ASM_BP - ret + RET SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff) --- a/arch/x86/lib/atomic64_386_32.S +++ b/arch/x86/lib/atomic64_386_32.S @@ -30,7 +30,7 @@ SYM_FUNC_START(atomic64_##op##_386); \ =20 #define RET_IRQ_RESTORE \ IRQ_RESTORE v; \ - ret + RET =20 #define v %ecx BEGIN_IRQ_SAVE(read) --- a/arch/x86/lib/atomic64_cx8_32.S +++ b/arch/x86/lib/atomic64_cx8_32.S @@ -18,7 +18,7 @@ =20 SYM_FUNC_START(atomic64_read_cx8) read64 %ecx - ret + RET SYM_FUNC_END(atomic64_read_cx8) =20 SYM_FUNC_START(atomic64_set_cx8) @@ -28,7 +28,7 @@ SYM_FUNC_START(atomic64_set_cx8) cmpxchg8b (%esi) jne 1b =20 - ret + RET SYM_FUNC_END(atomic64_set_cx8) =20 SYM_FUNC_START(atomic64_xchg_cx8) @@ -37,7 +37,7 @@ SYM_FUNC_START(atomic64_xchg_cx8) cmpxchg8b (%esi) jne 1b =20 - ret + RET SYM_FUNC_END(atomic64_xchg_cx8) =20 .macro addsub_return func ins insc @@ -68,7 +68,7 @@ SYM_FUNC_START(atomic64_\func\()_return_ popl %esi popl %ebx popl %ebp - ret + RET SYM_FUNC_END(atomic64_\func\()_return_cx8) .endm =20 @@ -93,7 +93,7 @@ SYM_FUNC_START(atomic64_\func\()_return_ movl %ebx, %eax movl %ecx, %edx popl %ebx - ret + RET SYM_FUNC_END(atomic64_\func\()_return_cx8) .endm =20 @@ -118,7 +118,7 @@ SYM_FUNC_START(atomic64_dec_if_positive_ movl %ebx, %eax movl %ecx, %edx popl %ebx - ret + RET SYM_FUNC_END(atomic64_dec_if_positive_cx8) =20 SYM_FUNC_START(atomic64_add_unless_cx8) @@ -149,7 +149,7 @@ SYM_FUNC_START(atomic64_add_unless_cx8) addl $8, %esp popl %ebx popl %ebp - ret + RET 4: cmpl %edx, 4(%esp) jne 2b @@ -176,5 +176,5 @@ SYM_FUNC_START(atomic64_inc_not_zero_cx8 movl $1, %eax 3: popl %ebx - ret + RET SYM_FUNC_END(atomic64_inc_not_zero_cx8) --- a/arch/x86/lib/checksum_32.S +++ b/arch/x86/lib/checksum_32.S @@ -127,7 +127,7 @@ SYM_FUNC_START(csum_partial) 8: popl %ebx popl %esi - ret + RET SYM_FUNC_END(csum_partial) =20 #else @@ -245,7 +245,7 @@ SYM_FUNC_START(csum_partial) 90:=20 popl %ebx popl %esi - ret + RET SYM_FUNC_END(csum_partial) =09 #endif @@ -371,7 +371,7 @@ EXC( movb %cl, (%edi) ) popl %esi popl %edi popl %ecx # equivalent to addl $4,%esp - ret=09 + RET SYM_FUNC_END(csum_partial_copy_generic) =20 #else @@ -447,7 +447,7 @@ EXC( movb %dl, (%edi) ) popl %esi popl %edi popl %ebx - ret + RET SYM_FUNC_END(csum_partial_copy_generic) =09 #undef ROUND --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -17,7 +17,7 @@ SYM_FUNC_START(clear_page_rep) movl $4096/8,%ecx xorl %eax,%eax rep stosq - ret + RET SYM_FUNC_END(clear_page_rep) EXPORT_SYMBOL_GPL(clear_page_rep) =20 @@ -39,7 +39,7 @@ SYM_FUNC_START(clear_page_orig) leaq 64(%rdi),%rdi jnz .Lloop nop - ret + RET SYM_FUNC_END(clear_page_orig) EXPORT_SYMBOL_GPL(clear_page_orig) =20 @@ -47,6 +47,6 @@ SYM_FUNC_START(clear_page_erms) movl $4096,%ecx xorl %eax,%eax rep stosb - ret + RET SYM_FUNC_END(clear_page_erms) EXPORT_SYMBOL_GPL(clear_page_erms) --- a/arch/x86/lib/cmpxchg16b_emu.S +++ b/arch/x86/lib/cmpxchg16b_emu.S @@ -37,11 +37,11 @@ SYM_FUNC_START(this_cpu_cmpxchg16b_emu) =20 popfq mov $1, %al - ret + RET =20 .Lnot_same: popfq xor %al,%al - ret + RET =20 SYM_FUNC_END(this_cpu_cmpxchg16b_emu) --- a/arch/x86/lib/cmpxchg8b_emu.S +++ b/arch/x86/lib/cmpxchg8b_emu.S @@ -32,7 +32,7 @@ SYM_FUNC_START(cmpxchg8b_emu) movl %ecx, 4(%esi) =20 popfl - ret + RET =20 .Lnot_same: movl (%esi), %eax @@ -40,7 +40,7 @@ SYM_FUNC_START(cmpxchg8b_emu) movl 4(%esi), %edx =20 popfl - ret + RET =20 SYM_FUNC_END(cmpxchg8b_emu) EXPORT_SYMBOL(cmpxchg8b_emu) --- a/arch/x86/lib/copy_mc_64.S +++ b/arch/x86/lib/copy_mc_64.S @@ -86,7 +86,7 @@ SYM_FUNC_START(copy_mc_fragile) .L_done_memcpy_trap: xorl %eax, %eax .L_done: - ret + RET SYM_FUNC_END(copy_mc_fragile) EXPORT_SYMBOL_GPL(copy_mc_fragile) =20 @@ -142,7 +142,7 @@ SYM_FUNC_START(copy_mc_enhanced_fast_str rep movsb /* Copy successful. Return zero */ xorl %eax, %eax - ret + RET SYM_FUNC_END(copy_mc_enhanced_fast_string) =20 .section .fixup, "ax" @@ -155,7 +155,7 @@ SYM_FUNC_END(copy_mc_enhanced_fast_strin * user-copy routines. */ movq %rcx, %rax - ret + RET =20 .previous =20 --- a/arch/x86/lib/copy_page_64.S +++ b/arch/x86/lib/copy_page_64.S @@ -17,7 +17,7 @@ SYM_FUNC_START(copy_page) ALTERNATIVE "jmp copy_page_regs", "", X86_FEATURE_REP_GOOD movl $4096/8, %ecx rep movsq - ret + RET SYM_FUNC_END(copy_page) EXPORT_SYMBOL(copy_page) =20 @@ -85,5 +85,5 @@ SYM_FUNC_START_LOCAL(copy_page_regs) movq (%rsp), %rbx movq 1*8(%rsp), %r12 addq $2*8, %rsp - ret + RET SYM_FUNC_END(copy_page_regs) --- a/arch/x86/lib/copy_user_64.S +++ b/arch/x86/lib/copy_user_64.S @@ -105,7 +105,7 @@ SYM_FUNC_START(copy_user_generic_unrolle jnz 21b 23: xor %eax,%eax ASM_CLAC - ret + RET =20 .section .fixup,"ax" 30: shll $6,%ecx @@ -173,7 +173,7 @@ SYM_FUNC_START(copy_user_generic_string) movsb xorl %eax,%eax ASM_CLAC - ret + RET =20 .section .fixup,"ax" 11: leal (%rdx,%rcx,8),%ecx @@ -207,7 +207,7 @@ SYM_FUNC_START(copy_user_enhanced_fast_s movsb xorl %eax,%eax ASM_CLAC - ret + RET =20 .section .fixup,"ax" 12: movl %ecx,%edx /* ecx is zerorest also */ @@ -239,7 +239,7 @@ SYM_CODE_START_LOCAL(.Lcopy_user_handle_ 1: rep movsb 2: mov %ecx,%eax ASM_CLAC - ret + RET =20 /* * Return zero to pretend that this copy succeeded. This @@ -250,7 +250,7 @@ SYM_CODE_START_LOCAL(.Lcopy_user_handle_ */ 3: xorl %eax,%eax ASM_CLAC - ret + RET =20 _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail) @@ -361,7 +361,7 @@ SYM_FUNC_START(__copy_user_nocache) xorl %eax,%eax ASM_CLAC sfence - ret + RET =20 .section .fixup,"ax" .L_fixup_4x8b_copy: --- a/arch/x86/lib/csum-copy_64.S +++ b/arch/x86/lib/csum-copy_64.S @@ -201,7 +201,7 @@ SYM_FUNC_START(csum_partial_copy_generic movq 3*8(%rsp), %r13 movq 4*8(%rsp), %r15 addq $5*8, %rsp - ret + RET .Lshort: movl %ecx, %r10d jmp .L1 --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -57,7 +57,7 @@ SYM_FUNC_START(__get_user_1) 1: movzbl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC - ret + RET SYM_FUNC_END(__get_user_1) EXPORT_SYMBOL(__get_user_1) =20 @@ -71,7 +71,7 @@ SYM_FUNC_START(__get_user_2) 2: movzwl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC - ret + RET SYM_FUNC_END(__get_user_2) EXPORT_SYMBOL(__get_user_2) =20 @@ -85,7 +85,7 @@ SYM_FUNC_START(__get_user_4) 3: movl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC - ret + RET SYM_FUNC_END(__get_user_4) EXPORT_SYMBOL(__get_user_4) =20 @@ -100,7 +100,7 @@ SYM_FUNC_START(__get_user_8) 4: movq (%_ASM_AX),%rdx xor %eax,%eax ASM_CLAC - ret + RET #else LOAD_TASK_SIZE_MINUS_N(7) cmp %_ASM_DX,%_ASM_AX @@ -112,7 +112,7 @@ SYM_FUNC_START(__get_user_8) 5: movl 4(%_ASM_AX),%ecx xor %eax,%eax ASM_CLAC - ret + RET #endif SYM_FUNC_END(__get_user_8) EXPORT_SYMBOL(__get_user_8) @@ -124,7 +124,7 @@ SYM_FUNC_START(__get_user_nocheck_1) 6: movzbl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC - ret + RET SYM_FUNC_END(__get_user_nocheck_1) EXPORT_SYMBOL(__get_user_nocheck_1) =20 @@ -134,7 +134,7 @@ SYM_FUNC_START(__get_user_nocheck_2) 7: movzwl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC - ret + RET SYM_FUNC_END(__get_user_nocheck_2) EXPORT_SYMBOL(__get_user_nocheck_2) =20 @@ -144,7 +144,7 @@ SYM_FUNC_START(__get_user_nocheck_4) 8: movl (%_ASM_AX),%edx xor %eax,%eax ASM_CLAC - ret + RET SYM_FUNC_END(__get_user_nocheck_4) EXPORT_SYMBOL(__get_user_nocheck_4) =20 @@ -159,7 +159,7 @@ SYM_FUNC_START(__get_user_nocheck_8) #endif xor %eax,%eax ASM_CLAC - ret + RET SYM_FUNC_END(__get_user_nocheck_8) EXPORT_SYMBOL(__get_user_nocheck_8) =20 @@ -169,7 +169,7 @@ SYM_CODE_START_LOCAL(.Lbad_get_user_clac bad_get_user: xor %edx,%edx mov $(-EFAULT),%_ASM_AX - ret + RET SYM_CODE_END(.Lbad_get_user_clac) =20 #ifdef CONFIG_X86_32 @@ -179,7 +179,7 @@ bad_get_user_8: xor %edx,%edx xor %ecx,%ecx mov $(-EFAULT),%_ASM_AX - ret + RET SYM_CODE_END(.Lbad_get_user_8_clac) #endif =20 --- a/arch/x86/lib/hweight.S +++ b/arch/x86/lib/hweight.S @@ -32,7 +32,7 @@ SYM_FUNC_START(__sw_hweight32) imull $0x01010101, %eax, %eax # w_tmp *=3D 0x01010101 shrl $24, %eax # w =3D w_tmp >> 24 __ASM_SIZE(pop,) %__ASM_REG(dx) - ret + RET SYM_FUNC_END(__sw_hweight32) EXPORT_SYMBOL(__sw_hweight32) =20 @@ -65,7 +65,7 @@ SYM_FUNC_START(__sw_hweight64) =20 popq %rdx popq %rdi - ret + RET #else /* CONFIG_X86_32 */ /* We're getting an u64 arg in (%eax,%edx): unsigned long hweight64(__u64= w) */ pushl %ecx @@ -77,7 +77,7 @@ SYM_FUNC_START(__sw_hweight64) addl %ecx, %eax # result =20 popl %ecx - ret + RET #endif SYM_FUNC_END(__sw_hweight64) EXPORT_SYMBOL(__sw_hweight64) --- a/arch/x86/lib/iomap_copy_64.S +++ b/arch/x86/lib/iomap_copy_64.S @@ -11,5 +11,5 @@ SYM_FUNC_START(__iowrite32_copy) movl %edx,%ecx rep movsd - ret + RET SYM_FUNC_END(__iowrite32_copy) --- a/arch/x86/lib/memcpy_64.S +++ b/arch/x86/lib/memcpy_64.S @@ -39,7 +39,7 @@ SYM_FUNC_START_WEAK(memcpy) rep movsq movl %edx, %ecx rep movsb - ret + RET SYM_FUNC_END(memcpy) SYM_FUNC_END_ALIAS(__memcpy) EXPORT_SYMBOL(memcpy) @@ -53,7 +53,7 @@ SYM_FUNC_START_LOCAL(memcpy_erms) movq %rdi, %rax movq %rdx, %rcx rep movsb - ret + RET SYM_FUNC_END(memcpy_erms) =20 SYM_FUNC_START_LOCAL(memcpy_orig) @@ -137,7 +137,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movq %r9, 1*8(%rdi) movq %r10, -2*8(%rdi, %rdx) movq %r11, -1*8(%rdi, %rdx) - retq + RET .p2align 4 .Lless_16bytes: cmpl $8, %edx @@ -149,7 +149,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movq -1*8(%rsi, %rdx), %r9 movq %r8, 0*8(%rdi) movq %r9, -1*8(%rdi, %rdx) - retq + RET .p2align 4 .Lless_8bytes: cmpl $4, %edx @@ -162,7 +162,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movl -4(%rsi, %rdx), %r8d movl %ecx, (%rdi) movl %r8d, -4(%rdi, %rdx) - retq + RET .p2align 4 .Lless_3bytes: subl $1, %edx @@ -180,7 +180,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movb %cl, (%rdi) =20 .Lend: - retq + RET SYM_FUNC_END(memcpy_orig) =20 .popsection --- a/arch/x86/lib/memmove_64.S +++ b/arch/x86/lib/memmove_64.S @@ -40,7 +40,7 @@ SYM_FUNC_START(__memmove) /* FSRM implies ERMS =3D> no length checks, do the copy directly */ .Lmemmove_begin_forward: ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM - ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; retq", X86_FEATURE_ERMS + ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; RET", X86_FEATURE_ERMS =20 /* * movsq instruction have many startup latency @@ -205,7 +205,7 @@ SYM_FUNC_START(__memmove) movb (%rsi), %r11b movb %r11b, (%rdi) 13: - retq + RET SYM_FUNC_END(__memmove) SYM_FUNC_END_ALIAS(memmove) EXPORT_SYMBOL(__memmove) --- a/arch/x86/lib/memset_64.S +++ b/arch/x86/lib/memset_64.S @@ -40,7 +40,7 @@ SYM_FUNC_START(__memset) movl %edx,%ecx rep stosb movq %r9,%rax - ret + RET SYM_FUNC_END(__memset) SYM_FUNC_END_ALIAS(memset) EXPORT_SYMBOL(memset) @@ -63,7 +63,7 @@ SYM_FUNC_START_LOCAL(memset_erms) movq %rdx,%rcx rep stosb movq %r9,%rax - ret + RET SYM_FUNC_END(memset_erms) =20 SYM_FUNC_START_LOCAL(memset_orig) @@ -125,7 +125,7 @@ SYM_FUNC_START_LOCAL(memset_orig) =20 .Lende: movq %r10,%rax - ret + RET =20 .Lbad_alignment: cmpq $7,%rdx --- a/arch/x86/lib/msr-reg.S +++ b/arch/x86/lib/msr-reg.S @@ -35,7 +35,7 @@ SYM_FUNC_START(\op\()_safe_regs) movl %edi, 28(%r10) popq %r12 popq %rbx - ret + RET 3: movl $-EIO, %r11d jmp 2b @@ -77,7 +77,7 @@ SYM_FUNC_START(\op\()_safe_regs) popl %esi popl %ebp popl %ebx - ret + RET 3: movl $-EIO, 4(%esp) jmp 2b --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -52,7 +52,7 @@ SYM_INNER_LABEL(__put_user_nocheck_1, SY 1: movb %al,(%_ASM_CX) xor %ecx,%ecx ASM_CLAC - ret + RET SYM_FUNC_END(__put_user_1) EXPORT_SYMBOL(__put_user_1) EXPORT_SYMBOL(__put_user_nocheck_1) @@ -66,7 +66,7 @@ SYM_INNER_LABEL(__put_user_nocheck_2, SY 2: movw %ax,(%_ASM_CX) xor %ecx,%ecx ASM_CLAC - ret + RET SYM_FUNC_END(__put_user_2) EXPORT_SYMBOL(__put_user_2) EXPORT_SYMBOL(__put_user_nocheck_2) @@ -80,7 +80,7 @@ SYM_INNER_LABEL(__put_user_nocheck_4, SY 3: movl %eax,(%_ASM_CX) xor %ecx,%ecx ASM_CLAC - ret + RET SYM_FUNC_END(__put_user_4) EXPORT_SYMBOL(__put_user_4) EXPORT_SYMBOL(__put_user_nocheck_4) --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -23,7 +23,7 @@ .Ldo_rop_\@: mov %\reg, (%_ASM_SP) UNWIND_HINT_FUNC - ret + RET .endm =20 .macro THUNK reg --- a/arch/x86/math-emu/div_Xsig.S +++ b/arch/x86/math-emu/div_Xsig.S @@ -341,7 +341,7 @@ L_exit: popl %esi =20 leave - ret + RET =20 =20 #ifdef PARANOID --- a/arch/x86/math-emu/div_small.S +++ b/arch/x86/math-emu/div_small.S @@ -44,5 +44,5 @@ SYM_FUNC_START(FPU_div_small) popl %esi =20 leave - ret + RET SYM_FUNC_END(FPU_div_small) --- a/arch/x86/math-emu/mul_Xsig.S +++ b/arch/x86/math-emu/mul_Xsig.S @@ -62,7 +62,7 @@ SYM_FUNC_START(mul32_Xsig) =20 popl %esi leave - ret + RET SYM_FUNC_END(mul32_Xsig) =20 =20 @@ -115,7 +115,7 @@ SYM_FUNC_START(mul64_Xsig) =20 popl %esi leave - ret + RET SYM_FUNC_END(mul64_Xsig) =20 =20 @@ -175,5 +175,5 @@ SYM_FUNC_START(mul_Xsig_Xsig) =20 popl %esi leave - ret + RET SYM_FUNC_END(mul_Xsig_Xsig) --- a/arch/x86/math-emu/polynom_Xsig.S +++ b/arch/x86/math-emu/polynom_Xsig.S @@ -133,5 +133,5 @@ L_accum_done: popl %edi popl %esi leave - ret + RET SYM_FUNC_END(polynomial_Xsig) --- a/arch/x86/math-emu/reg_norm.S +++ b/arch/x86/math-emu/reg_norm.S @@ -72,7 +72,7 @@ L_exit_valid: L_exit: popl %ebx leave - ret + RET =20 =20 L_zero: @@ -138,7 +138,7 @@ L_exit_nuo_valid: =20 popl %ebx leave - ret + RET =20 L_exit_nuo_zero: movl TAG_Zero,%eax @@ -146,5 +146,5 @@ L_exit_nuo_zero: =20 popl %ebx leave - ret + RET SYM_FUNC_END(FPU_normalize_nuo) --- a/arch/x86/math-emu/reg_round.S +++ b/arch/x86/math-emu/reg_round.S @@ -437,7 +437,7 @@ fpu_Arith_exit: popl %edi popl %esi leave - ret + RET =20 =20 /* --- a/arch/x86/math-emu/reg_u_add.S +++ b/arch/x86/math-emu/reg_u_add.S @@ -164,6 +164,6 @@ L_exit: popl %edi popl %esi leave - ret + RET #endif /* PARANOID */ SYM_FUNC_END(FPU_u_add) --- a/arch/x86/math-emu/reg_u_div.S +++ b/arch/x86/math-emu/reg_u_div.S @@ -468,7 +468,7 @@ L_exit: popl %esi =20 leave - ret + RET #endif /* PARANOID */=20 =20 SYM_FUNC_END(FPU_u_div) --- a/arch/x86/math-emu/reg_u_mul.S +++ b/arch/x86/math-emu/reg_u_mul.S @@ -144,7 +144,7 @@ L_exit: popl %edi popl %esi leave - ret + RET #endif /* PARANOID */=20 =20 SYM_FUNC_END(FPU_u_mul) --- a/arch/x86/math-emu/reg_u_sub.S +++ b/arch/x86/math-emu/reg_u_sub.S @@ -270,5 +270,5 @@ L_exit: popl %edi popl %esi leave - ret + RET SYM_FUNC_END(FPU_u_sub) --- a/arch/x86/math-emu/round_Xsig.S +++ b/arch/x86/math-emu/round_Xsig.S @@ -78,7 +78,7 @@ L_exit: popl %esi popl %ebx leave - ret + RET SYM_FUNC_END(round_Xsig) =20 =20 @@ -138,5 +138,5 @@ L_n_exit: popl %esi popl %ebx leave - ret + RET SYM_FUNC_END(norm_Xsig) --- a/arch/x86/math-emu/shr_Xsig.S +++ b/arch/x86/math-emu/shr_Xsig.S @@ -45,7 +45,7 @@ SYM_FUNC_START(shr_Xsig) popl %ebx popl %esi leave - ret + RET =20 L_more_than_31: cmpl $64,%ecx @@ -61,7 +61,7 @@ L_more_than_31: movl $0,8(%esi) popl %esi leave - ret + RET =20 L_more_than_63: cmpl $96,%ecx @@ -76,7 +76,7 @@ L_more_than_63: movl %edx,8(%esi) popl %esi leave - ret + RET =20 L_more_than_95: xorl %eax,%eax @@ -85,5 +85,5 @@ L_more_than_95: movl %eax,8(%esi) popl %esi leave - ret + RET SYM_FUNC_END(shr_Xsig) --- a/arch/x86/math-emu/wm_shrx.S +++ b/arch/x86/math-emu/wm_shrx.S @@ -55,7 +55,7 @@ SYM_FUNC_START(FPU_shrx) popl %ebx popl %esi leave - ret + RET =20 L_more_than_31: cmpl $64,%ecx @@ -70,7 +70,7 @@ L_more_than_31: movl $0,4(%esi) popl %esi leave - ret + RET =20 L_more_than_63: cmpl $96,%ecx @@ -84,7 +84,7 @@ L_more_than_63: movl %edx,4(%esi) popl %esi leave - ret + RET =20 L_more_than_95: xorl %eax,%eax @@ -92,7 +92,7 @@ L_more_than_95: movl %eax,4(%esi) popl %esi leave - ret + RET SYM_FUNC_END(FPU_shrx) =20 =20 @@ -146,7 +146,7 @@ SYM_FUNC_START(FPU_shrxs) popl %ebx popl %esi leave - ret + RET =20 /* Shift by [0..31] bits */ Ls_less_than_32: @@ -163,7 +163,7 @@ Ls_less_than_32: popl %ebx popl %esi leave - ret + RET =20 /* Shift by [64..95] bits */ Ls_more_than_63: @@ -189,7 +189,7 @@ Ls_more_than_63: popl %ebx popl %esi leave - ret + RET =20 Ls_more_than_95: /* Shift by [96..inf) bits */ @@ -203,5 +203,5 @@ Ls_more_than_95: popl %ebx popl %esi leave - ret + RET SYM_FUNC_END(FPU_shrxs) --- a/arch/x86/mm/mem_encrypt_boot.S +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -65,7 +65,7 @@ SYM_FUNC_START(sme_encrypt_execute) movq %rbp, %rsp /* Restore original stack pointer */ pop %rbp =20 - ret + RET SYM_FUNC_END(sme_encrypt_execute) =20 SYM_FUNC_START(__enc_copy) @@ -151,6 +151,6 @@ SYM_FUNC_START(__enc_copy) pop %r12 pop %r15 =20 - ret + RET .L__enc_copy_end: SYM_FUNC_END(__enc_copy) --- a/arch/x86/platform/efi/efi_stub_32.S +++ b/arch/x86/platform/efi/efi_stub_32.S @@ -56,5 +56,5 @@ SYM_FUNC_START(efi_call_svam) =20 movl 16(%esp), %ebx leave - ret + RET SYM_FUNC_END(efi_call_svam) --- a/arch/x86/platform/efi/efi_stub_64.S +++ b/arch/x86/platform/efi/efi_stub_64.S @@ -23,5 +23,5 @@ SYM_FUNC_START(__efi_call) mov %rsi, %rcx CALL_NOSPEC rdi leave - ret + RET SYM_FUNC_END(__efi_call) --- a/arch/x86/platform/efi/efi_thunk_64.S +++ b/arch/x86/platform/efi/efi_thunk_64.S @@ -63,7 +63,7 @@ SYM_CODE_START(__efi64_thunk) 1: movq 24(%rsp), %rsp pop %rbx pop %rbp - retq + RET =20 .code32 2: pushl $__KERNEL_CS --- a/arch/x86/platform/olpc/xo1-wakeup.S +++ b/arch/x86/platform/olpc/xo1-wakeup.S @@ -77,7 +77,7 @@ save_registers: pushfl popl saved_context_eflags =20 - ret + RET =20 restore_registers: movl saved_context_ebp, %ebp @@ -88,7 +88,7 @@ restore_registers: pushl saved_context_eflags popfl =20 - ret + RET =20 SYM_CODE_START(do_olpc_suspend_lowlevel) call save_processor_state @@ -109,7 +109,7 @@ ret_point: =20 call restore_registers call restore_processor_state - ret + RET SYM_CODE_END(do_olpc_suspend_lowlevel) =20 .data --- a/arch/x86/power/hibernate_asm_32.S +++ b/arch/x86/power/hibernate_asm_32.S @@ -32,7 +32,7 @@ SYM_FUNC_START(swsusp_arch_suspend) FRAME_BEGIN call swsusp_save FRAME_END - ret + RET SYM_FUNC_END(swsusp_arch_suspend) =20 SYM_CODE_START(restore_image) @@ -108,5 +108,5 @@ SYM_FUNC_START(restore_registers) /* tell the hibernation core that we've just restored the memory */ movl %eax, in_suspend =20 - ret + RET SYM_FUNC_END(restore_registers) --- a/arch/x86/power/hibernate_asm_64.S +++ b/arch/x86/power/hibernate_asm_64.S @@ -49,7 +49,7 @@ SYM_FUNC_START(swsusp_arch_suspend) FRAME_BEGIN call swsusp_save FRAME_END - ret + RET SYM_FUNC_END(swsusp_arch_suspend) =20 SYM_CODE_START(restore_image) @@ -143,5 +143,5 @@ SYM_FUNC_START(restore_registers) /* tell the hibernation core that we've just restored the memory */ movq %rax, in_suspend(%rip) =20 - ret + RET SYM_FUNC_END(restore_registers) --- a/arch/x86/um/checksum_32.S +++ b/arch/x86/um/checksum_32.S @@ -110,7 +110,7 @@ csum_partial: 7:=09 popl %ebx popl %esi - ret + RET =20 #else =20 @@ -208,7 +208,7 @@ csum_partial: 80:=20 popl %ebx popl %esi - ret + RET =09 #endif EXPORT_SYMBOL(csum_partial) --- a/arch/x86/um/setjmp_32.S +++ b/arch/x86/um/setjmp_32.S @@ -34,7 +34,7 @@ kernel_setjmp: movl %esi,12(%edx) movl %edi,16(%edx) movl %ecx,20(%edx) # Return address - ret + RET =20 .size kernel_setjmp,.-kernel_setjmp =20 --- a/arch/x86/um/setjmp_64.S +++ b/arch/x86/um/setjmp_64.S @@ -33,7 +33,7 @@ kernel_setjmp: movq %r14,40(%rdi) movq %r15,48(%rdi) movq %rsi,56(%rdi) # Return address - ret + RET =20 .size kernel_setjmp,.-kernel_setjmp =20 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -45,7 +45,7 @@ SYM_FUNC_START(xen_irq_enable_direct) call check_events 1: FRAME_END - ret + RET SYM_FUNC_END(xen_irq_enable_direct) =20 =20 @@ -55,7 +55,7 @@ SYM_FUNC_END(xen_irq_enable_direct) */ SYM_FUNC_START(xen_irq_disable_direct) movb $1, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask - ret + RET SYM_FUNC_END(xen_irq_disable_direct) =20 /* @@ -71,7 +71,7 @@ SYM_FUNC_START(xen_save_fl_direct) testb $0xff, PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_mask setz %ah addb %ah, %ah - ret + RET SYM_FUNC_END(xen_save_fl_direct) =20 =20 @@ -98,7 +98,7 @@ SYM_FUNC_START(xen_restore_fl_direct) call check_events 1: FRAME_END - ret + RET SYM_FUNC_END(xen_restore_fl_direct) =20 =20 @@ -128,7 +128,7 @@ SYM_FUNC_START(check_events) pop %rcx pop %rax FRAME_END - ret + RET SYM_FUNC_END(check_events) =20 SYM_FUNC_START(xen_read_cr2) @@ -136,14 +136,14 @@ SYM_FUNC_START(xen_read_cr2) _ASM_MOV PER_CPU_VAR(xen_vcpu), %_ASM_AX _ASM_MOV XEN_vcpu_info_arch_cr2(%_ASM_AX), %_ASM_AX FRAME_END - ret + RET SYM_FUNC_END(xen_read_cr2); =20 SYM_FUNC_START(xen_read_cr2_direct) FRAME_BEGIN _ASM_MOV PER_CPU_VAR(xen_vcpu_info) + XEN_vcpu_info_arch_cr2, %_ASM_AX FRAME_END - ret + RET SYM_FUNC_END(xen_read_cr2_direct); =20 .macro xen_pv_trap name --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -70,7 +70,7 @@ SYM_CODE_START(hypercall_page) .rept (PAGE_SIZE / 32) UNWIND_HINT_FUNC .skip 31, 0x90 - ret + RET .endr =20 #define HYPERCALL(n) \ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5179CCA483 for ; Sat, 23 Jul 2022 10:04:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238221AbiGWKEG (ORCPT ); Sat, 23 Jul 2022 06:04:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238082AbiGWKDl (ORCPT ); Sat, 23 Jul 2022 06:03:41 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA5158C8EB; Sat, 23 Jul 2022 02:59:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 49D94B82C1A; Sat, 23 Jul 2022 09:59:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F6CEC341C7; Sat, 23 Jul 2022 09:59:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570373; bh=ccttcUejxIAp0iXxK7DSBQA0Uf0qGZNTazYuhCCZ8cM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iRilNCXb74SxxyNu47iQDYf1B1/nJuQC+FNUoQUvEmrR5nXL3S/dAhHbDNEQaD04w MFot4ftY7e5Vsd8EWf9VugLbGDy3cqL58iv4m3YUEdoWj9K/s77nTWLQqp10ILl4MM GZlfUJaIW+y9MIoDmYKt8k5OgudR2ygw4xsOEaDY= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 063/148] x86: Prepare inline-asm for straight-line-speculation Date: Sat, 23 Jul 2022 11:54:35 +0200 Message-Id: <20220723095241.926858016@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit b17c2baa305cccbd16bafa289fd743cc2db77966 upstream. Replace all ret/retq instructions with ASM_RET in preparation of making it more than a single instruction. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/20211204134907.964635458@infradead.org Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/linkage.h | 4 ++++ arch/x86/include/asm/paravirt.h | 2 +- arch/x86/include/asm/qspinlock_paravirt.h | 4 ++-- arch/x86/kernel/alternative.c | 2 +- arch/x86/kernel/kprobes/core.c | 2 +- arch/x86/kernel/paravirt.c | 2 +- arch/x86/kvm/emulate.c | 4 ++-- arch/x86/lib/error-inject.c | 3 ++- samples/ftrace/ftrace-direct-modify.c | 4 ++-- samples/ftrace/ftrace-direct-too.c | 2 +- samples/ftrace/ftrace-direct.c | 2 +- 11 files changed, 18 insertions(+), 13 deletions(-) --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -18,6 +18,10 @@ #define __ALIGN_STR __stringify(__ALIGN) #endif =20 +#else /* __ASSEMBLY__ */ + +#define ASM_RET "ret\n\t" + #endif /* __ASSEMBLY__ */ =20 #endif /* _ASM_X86_LINKAGE_H */ --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -630,7 +630,7 @@ bool __raw_callee_save___native_vcpu_is_ "call " #func ";" \ PV_RESTORE_ALL_CALLER_REGS \ FRAME_END \ - "ret;" \ + ASM_RET \ ".size " PV_THUNK_NAME(func) ", .-" PV_THUNK_NAME(func) ";" \ ".popsection") =20 --- a/arch/x86/include/asm/qspinlock_paravirt.h +++ b/arch/x86/include/asm/qspinlock_paravirt.h @@ -48,7 +48,7 @@ asm (".pushsection .text;" "jne .slowpath;" "pop %rdx;" FRAME_END - "ret;" + ASM_RET ".slowpath: " "push %rsi;" "movzbl %al,%esi;" @@ -56,7 +56,7 @@ asm (".pushsection .text;" "pop %rsi;" "pop %rdx;" FRAME_END - "ret;" + ASM_RET ".size " PV_UNLOCK ", .-" PV_UNLOCK ";" ".popsection"); =20 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -869,7 +869,7 @@ asm ( " .type int3_magic, @function\n" "int3_magic:\n" " movl $1, (%" _ASM_ARG1 ")\n" -" ret\n" + ASM_RET " .size int3_magic, .-int3_magic\n" " .popsection\n" ); --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -768,7 +768,7 @@ asm( RESTORE_REGS_STRING " popfl\n" #endif - " ret\n" + ASM_RET ".size kretprobe_trampoline, .-kretprobe_trampoline\n" ); NOKPROBE_SYMBOL(kretprobe_trampoline); --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -40,7 +40,7 @@ extern void _paravirt_nop(void); asm (".pushsection .entry.text, \"ax\"\n" ".global _paravirt_nop\n" "_paravirt_nop:\n\t" - "ret\n\t" + ASM_RET ".size _paravirt_nop, . - _paravirt_nop\n\t" ".type _paravirt_nop, @function\n\t" ".popsection"); --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -316,7 +316,7 @@ static int fastop(struct x86_emulate_ctx __FOP_FUNC(#name) =20 #define __FOP_RET(name) \ - "ret \n\t" \ + ASM_RET \ ".size " name ", .-" name "\n\t" =20 #define FOP_RET(name) \ @@ -437,7 +437,7 @@ static int fastop(struct x86_emulate_ctx =20 asm(".pushsection .fixup, \"ax\"\n" ".global kvm_fastop_exception \n" - "kvm_fastop_exception: xor %esi, %esi; ret\n" + "kvm_fastop_exception: xor %esi, %esi; " ASM_RET ".popsection"); =20 FOP_START(setcc) --- a/arch/x86/lib/error-inject.c +++ b/arch/x86/lib/error-inject.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 =20 +#include #include #include =20 @@ -10,7 +11,7 @@ asm( ".type just_return_func, @function\n" ".globl just_return_func\n" "just_return_func:\n" - " ret\n" + ASM_RET ".size just_return_func, .-just_return_func\n" ); =20 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -31,7 +31,7 @@ asm ( " call my_direct_func1\n" " leave\n" " .size my_tramp1, .-my_tramp1\n" -" ret\n" + ASM_RET " .type my_tramp2, @function\n" " .globl my_tramp2\n" " my_tramp2:" @@ -39,7 +39,7 @@ asm ( " movq %rsp, %rbp\n" " call my_direct_func2\n" " leave\n" -" ret\n" + ASM_RET " .size my_tramp2, .-my_tramp2\n" " .popsection\n" ); --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -31,7 +31,7 @@ asm ( " popq %rsi\n" " popq %rdi\n" " leave\n" -" ret\n" + ASM_RET " .size my_tramp, .-my_tramp\n" " .popsection\n" ); --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -24,7 +24,7 @@ asm ( " call my_direct_func\n" " popq %rdi\n" " leave\n" -" ret\n" + ASM_RET " .size my_tramp, .-my_tramp\n" " .popsection\n" ); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AE58C433EF for ; Sat, 23 Jul 2022 10:04:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238266AbiGWKEs (ORCPT ); Sat, 23 Jul 2022 06:04:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238400AbiGWKEV (ORCPT ); Sat, 23 Jul 2022 06:04:21 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C318798226; Sat, 23 Jul 2022 02:59:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 21102B82C23; Sat, 23 Jul 2022 09:59:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7CB43C341C0; Sat, 23 Jul 2022 09:59:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570375; bh=jXkipwUJuN5Bshbw5oq+n7KCie4OGQQnZ71o0LvNP0o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QidTTwuaWvFVwebTQ+XjKGywWd3WhpF3bYKdI98lzHFHd4eYL16j8Qln6Fw1Hkxjz lNaG5GUBOPtr+8nDW1xKfmp2DgptCZfjfhI8oLi7KL2gpXuZfGQm8MbCRT5dkEqiCi KPdIMkbDE930ROoN9H002f3SZscfD/ZeV32nPhLA= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 064/148] x86/alternative: Relax text_poke_bp() constraint Date: Sat, 23 Jul 2022 11:54:36 +0200 Message-Id: <20220723095242.224281086@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 26c44b776dba4ac692a0bf5a3836feb8a63fea6b upstream. Currently, text_poke_bp() is very strict to only allow patching a single instruction; however with straight-line-speculation it will be required to patch: ret; int3, which is two instructions. As such, relax the constraints a little to allow int3 padding for all instructions that do not imply the execution of the next instruction, ie: RET, JMP.d8 and JMP.d32. While there, rename the text_poke_loc::rel32 field to ::disp. Note: this fills up the text_poke_loc structure which is now a round 16 bytes big. [ bp: Put comments ontop instead of on the side. ] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/20211204134908.082342723@infradead.org Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/alternative.c | 49 +++++++++++++++++++++++++++++--------= ----- 1 file changed, 34 insertions(+), 15 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1243,10 +1243,13 @@ void text_poke_sync(void) } =20 struct text_poke_loc { - s32 rel_addr; /* addr :=3D _stext + rel_addr */ - s32 rel32; + /* addr :=3D _stext + rel_addr */ + s32 rel_addr; + s32 disp; + u8 len; u8 opcode; const u8 text[POKE_MAX_OPCODE_SIZE]; + /* see text_poke_bp_batch() */ u8 old; }; =20 @@ -1261,7 +1264,8 @@ static struct bp_patching_desc *bp_desc; static __always_inline struct bp_patching_desc *try_get_desc(struct bp_patching_desc **descp) { - struct bp_patching_desc *desc =3D __READ_ONCE(*descp); /* rcu_dereference= */ + /* rcu_dereference */ + struct bp_patching_desc *desc =3D __READ_ONCE(*descp); =20 if (!desc || !arch_atomic_inc_not_zero(&desc->refs)) return NULL; @@ -1295,7 +1299,7 @@ noinstr int poke_int3_handler(struct pt_ { struct bp_patching_desc *desc; struct text_poke_loc *tp; - int len, ret =3D 0; + int ret =3D 0; void *ip; =20 if (user_mode(regs)) @@ -1335,8 +1339,7 @@ noinstr int poke_int3_handler(struct pt_ goto out_put; } =20 - len =3D text_opcode_size(tp->opcode); - ip +=3D len; + ip +=3D tp->len; =20 switch (tp->opcode) { case INT3_INSN_OPCODE: @@ -1351,12 +1354,12 @@ noinstr int poke_int3_handler(struct pt_ break; =20 case CALL_INSN_OPCODE: - int3_emulate_call(regs, (long)ip + tp->rel32); + int3_emulate_call(regs, (long)ip + tp->disp); break; =20 case JMP32_INSN_OPCODE: case JMP8_INSN_OPCODE: - int3_emulate_jmp(regs, (long)ip + tp->rel32); + int3_emulate_jmp(regs, (long)ip + tp->disp); break; =20 default: @@ -1431,7 +1434,7 @@ static void text_poke_bp_batch(struct te */ for (do_sync =3D 0, i =3D 0; i < nr_entries; i++) { u8 old[POKE_MAX_OPCODE_SIZE] =3D { tp[i].old, }; - int len =3D text_opcode_size(tp[i].opcode); + int len =3D tp[i].len; =20 if (len - INT3_INSN_SIZE > 0) { memcpy(old + INT3_INSN_SIZE, @@ -1508,21 +1511,37 @@ static void text_poke_loc_init(struct te const void *opcode, size_t len, const void *emulate) { struct insn insn; - int ret; + int ret, i; =20 memcpy((void *)tp->text, opcode, len); if (!emulate) emulate =3D opcode; =20 ret =3D insn_decode_kernel(&insn, emulate); - BUG_ON(ret < 0); - BUG_ON(len !=3D insn.length); =20 tp->rel_addr =3D addr - (void *)_stext; + tp->len =3D len; tp->opcode =3D insn.opcode.bytes[0]; =20 switch (tp->opcode) { + case RET_INSN_OPCODE: + case JMP32_INSN_OPCODE: + case JMP8_INSN_OPCODE: + /* + * Control flow instructions without implied execution of the + * next instruction can be padded with INT3. + */ + for (i =3D insn.length; i < len; i++) + BUG_ON(tp->text[i] !=3D INT3_INSN_OPCODE); + break; + + default: + BUG_ON(len !=3D insn.length); + }; + + + switch (tp->opcode) { case INT3_INSN_OPCODE: case RET_INSN_OPCODE: break; @@ -1530,7 +1549,7 @@ static void text_poke_loc_init(struct te case CALL_INSN_OPCODE: case JMP32_INSN_OPCODE: case JMP8_INSN_OPCODE: - tp->rel32 =3D insn.immediate.value; + tp->disp =3D insn.immediate.value; break; =20 default: /* assume NOP */ @@ -1538,13 +1557,13 @@ static void text_poke_loc_init(struct te case 2: /* NOP2 -- emulate as JMP8+0 */ BUG_ON(memcmp(emulate, ideal_nops[len], len)); tp->opcode =3D JMP8_INSN_OPCODE; - tp->rel32 =3D 0; + tp->disp =3D 0; break; =20 case 5: /* NOP5 -- emulate as JMP32+0 */ BUG_ON(memcmp(emulate, ideal_nops[NOP_ATOMIC5], len)); tp->opcode =3D JMP32_INSN_OPCODE; - tp->rel32 =3D 0; + tp->disp =3D 0; break; =20 default: /* unknown instruction */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 276B3C43334 for ; Sat, 23 Jul 2022 10:04:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238299AbiGWKEw (ORCPT ); Sat, 23 Jul 2022 06:04:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238424AbiGWKEW (ORCPT ); Sat, 23 Jul 2022 06:04:22 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2F438F50F; Sat, 23 Jul 2022 02:59:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3C55E611BF; Sat, 23 Jul 2022 09:59:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4B29FC341C7; Sat, 23 Jul 2022 09:59:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570378; bh=eFvAyJVDibAY2Sr7A/p+MgTJbLpaK5JM6AIuZ6Yji2E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hLdSODmnPLqB2qDcbaPNWuVwQi/NLsqnNblfwcJpM6QjEazrS1jCGwmuYnAKeIljY DAOhODYEjV3iDGhyALuXL1MzwiSi2/n/K7rUt18Xyc58JWTSDH44rWVbe4YmSh/iTM kA16uJHswUVSS0O4p+iwsX7/tZIA4/Z3JgBIKcEs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 065/148] objtool: Add straight-line-speculation validation Date: Sat, 23 Jul 2022 11:54:37 +0200 Message-Id: <20220723095242.497080327@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 1cc1e4c8aab4213bd4e6353dec2620476a233d6d upstream. Teach objtool to validate the straight-line-speculation constraints: - speculation trap after indirect calls - speculation trap after RET Notable: when an instruction is annotated RETPOLINE_SAFE, indicating speculation isn't a problem, also don't care about sls for that instruction. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/20211204134908.023037659@infradead.org Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman [bwh: Backported to 5.10: adjust filenames, context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch.h | 1 + tools/objtool/arch/x86/decode.c | 13 +++++++++---- tools/objtool/builtin-check.c | 4 +++- tools/objtool/builtin.h | 3 ++- tools/objtool/check.c | 14 ++++++++++++++ 5 files changed, 29 insertions(+), 6 deletions(-) --- a/tools/objtool/arch.h +++ b/tools/objtool/arch.h @@ -26,6 +26,7 @@ enum insn_type { INSN_CLAC, INSN_STD, INSN_CLD, + INSN_TRAP, INSN_OTHER, }; =20 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -456,6 +456,11 @@ int arch_decode_instruction(const struct =20 break; =20 + case 0xcc: + /* int3 */ + *type =3D INSN_TRAP; + break; + case 0xe3: /* jecxz/jrcxz */ *type =3D INSN_JUMP_CONDITIONAL; @@ -592,10 +597,10 @@ const char *arch_ret_insn(int len) { static const char ret[5][5] =3D { { BYTE_RET }, - { BYTE_RET, 0x90 }, - { BYTE_RET, 0x66, 0x90 }, - { BYTE_RET, 0x0f, 0x1f, 0x00 }, - { BYTE_RET, 0x0f, 0x1f, 0x40, 0x00 }, + { BYTE_RET, 0xcc }, + { BYTE_RET, 0xcc, 0x90 }, + { BYTE_RET, 0xcc, 0x66, 0x90 }, + { BYTE_RET, 0xcc, 0x0f, 0x1f, 0x00 }, }; =20 if (len < 1 || len > 5) { --- a/tools/objtool/builtin-check.c +++ b/tools/objtool/builtin-check.c @@ -18,7 +18,8 @@ #include "builtin.h" #include "objtool.h" =20 -bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, stats, = validate_dup, vmlinux; +bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, stats, + validate_dup, vmlinux, sls; =20 static const char * const check_usage[] =3D { "objtool check [] file.o", @@ -35,6 +36,7 @@ const struct option check_options[] =3D { OPT_BOOLEAN('s', "stats", &stats, "print statistics"), OPT_BOOLEAN('d', "duplicate", &validate_dup, "duplicate validation for vm= linux.o"), OPT_BOOLEAN('l', "vmlinux", &vmlinux, "vmlinux.o validation"), + OPT_BOOLEAN('S', "sls", &sls, "validate straight-line-speculation"), OPT_END(), }; =20 --- a/tools/objtool/builtin.h +++ b/tools/objtool/builtin.h @@ -8,7 +8,8 @@ #include =20 extern const struct option check_options[]; -extern bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, = stats, validate_dup, vmlinux; +extern bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, = stats, + validate_dup, vmlinux, sls; =20 extern int cmd_check(int argc, const char **argv); extern int cmd_orc(int argc, const char **argv); --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2775,6 +2775,12 @@ static int validate_branch(struct objtoo switch (insn->type) { =20 case INSN_RETURN: + if (next_insn && next_insn->type =3D=3D INSN_TRAP) { + next_insn->ignore =3D true; + } else if (sls && !insn->retpoline_safe) { + WARN_FUNC("missing int3 after ret", + insn->sec, insn->offset); + } return validate_return(func, insn, &state); =20 case INSN_CALL: @@ -2818,6 +2824,14 @@ static int validate_branch(struct objtoo break; =20 case INSN_JUMP_DYNAMIC: + if (next_insn && next_insn->type =3D=3D INSN_TRAP) { + next_insn->ignore =3D true; + } else if (sls && !insn->retpoline_safe) { + WARN_FUNC("missing int3 after indirect jump", + insn->sec, insn->offset); + } + + /* fallthrough */ case INSN_JUMP_DYNAMIC_CONDITIONAL: if (is_sibling_call(insn)) { ret =3D validate_sibling_call(insn, &state); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D89D9C43334 for ; Sat, 23 Jul 2022 10:04:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238352AbiGWKE4 (ORCPT ); Sat, 23 Jul 2022 06:04:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238451AbiGWKEY (ORCPT ); Sat, 23 Jul 2022 06:04:24 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 968609B1AB; Sat, 23 Jul 2022 03:00:00 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C7885B82C1D; Sat, 23 Jul 2022 09:59:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2FA2CC341C0; Sat, 23 Jul 2022 09:59:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570381; bh=k9IrByeFv4ANj8rXmHXEEaxduV+WzzfaR+gGXS2EnL4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NJbV2T3sP2ZXoNnMN9IkL3jRlm4by6ZC/m9ohsXeQ9wtv5o8pHulHgRUj+svieW9S bxE/t4YQwAqLX2W1yN53Au2q9UDIgdG+BUeH/n4lqJZbX05spoLneEdCBUr8Lx/A2Y bkiG/Drk4Y/8SRHgHl4g5FvOjSrK22u4Iqwm4V+o= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 066/148] x86: Add straight-line-speculation mitigation Date: Sat, 23 Jul 2022 11:54:38 +0200 Message-Id: <20220723095242.750401927@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit e463a09af2f0677b9485a7e8e4e70b396b2ffb6f upstream. Make use of an upcoming GCC feature to mitigate straight-line-speculation for x86: https://gcc.gnu.org/g:53a643f8568067d7700a9f2facc8ba39974973d3 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D102952 https://bugs.llvm.org/show_bug.cgi?id=3D52323 It's built tested on x86_64-allyesconfig using GCC-12 and GCC-11. Maintenance overhead of this should be fairly low due to objtool validation. Size overhead of all these additional int3 instructions comes to: text data bss dec hex filename 22267751 6933356 2011368 31212475 1dc43bb defconfig-build/vmlinux 22804126 6933356 1470696 31208178 1dc32f2 defconfig-build/vmlinux.sls Or roughly 2.4% additional text. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/20211204134908.140103474@infradead.org Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman [bwh: Backported to 5.10: - In scripts/Makefile.build, add the objtool option with an ifdef block, same as for other options - Adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/Kconfig | 12 ++++++++++++ arch/x86/Makefile | 6 +++++- arch/x86/include/asm/linkage.h | 10 ++++++++++ arch/x86/include/asm/static_call.h | 2 +- arch/x86/kernel/ftrace.c | 2 +- arch/x86/kernel/static_call.c | 5 +++-- arch/x86/lib/memmove_64.S | 2 +- arch/x86/lib/retpoline.S | 2 +- scripts/Makefile.build | 3 +++ scripts/link-vmlinux.sh | 3 +++ 10 files changed, 40 insertions(+), 7 deletions(-) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -462,6 +462,18 @@ config RETPOLINE branches. Requires a compiler with -mindirect-branch=3Dthunk-extern support for full protection. The kernel may run slower. =20 +config CC_HAS_SLS + def_bool $(cc-option,-mharden-sls=3Dall) + +config SLS + bool "Mitigate Straight-Line-Speculation" + depends on CC_HAS_SLS && X86_64 + default n + help + Compile the kernel with straight-line-speculation options to guard + against straight line speculation. The kernel image might be slightly + larger. + config X86_CPU_RESCTRL bool "x86 CPU resource control support" depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD) --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -196,7 +196,11 @@ ifdef CONFIG_RETPOLINE endif endif =20 -KBUILD_LDFLAGS :=3D -m elf_$(UTS_MACHINE) +ifdef CONFIG_SLS + KBUILD_CFLAGS +=3D -mharden-sls=3Dall +endif + +KBUILD_LDFLAGS +=3D -m elf_$(UTS_MACHINE) =20 ifdef CONFIG_X86_NEED_RELOCS LDFLAGS_vmlinux :=3D --emit-relocs --discard-none --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -18,9 +18,19 @@ #define __ALIGN_STR __stringify(__ALIGN) #endif =20 +#ifdef CONFIG_SLS +#define RET ret; int3 +#else +#define RET ret +#endif + #else /* __ASSEMBLY__ */ =20 +#ifdef CONFIG_SLS +#define ASM_RET "ret; int3\n\t" +#else #define ASM_RET "ret\n\t" +#endif =20 #endif /* __ASSEMBLY__ */ =20 --- a/arch/x86/include/asm/static_call.h +++ b/arch/x86/include/asm/static_call.h @@ -35,7 +35,7 @@ __ARCH_DEFINE_STATIC_CALL_TRAMP(name, ".byte 0xe9; .long " #func " - (. += 4)") =20 #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ - __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "ret; nop; nop; nop; nop") + __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "ret; int3; nop; nop; nop") =20 =20 #define ARCH_ADD_TRAMP_KEY(name) \ --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -308,7 +308,7 @@ union ftrace_op_code_union { } __attribute__((packed)); }; =20 -#define RET_SIZE 1 +#define RET_SIZE 1 + IS_ENABLED(CONFIG_SLS) =20 static unsigned long create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -11,6 +11,8 @@ enum insn_type { RET =3D 3, /* tramp / site cond-tail-call */ }; =20 +static const u8 retinsn[] =3D { RET_INSN_OPCODE, 0xcc, 0xcc, 0xcc, 0xcc }; + static void __ref __static_call_transform(void *insn, enum insn_type type,= void *func) { int size =3D CALL_INSN_SIZE; @@ -30,8 +32,7 @@ static void __ref __static_call_transfor break; =20 case RET: - code =3D text_gen_insn(RET_INSN_OPCODE, insn, func); - size =3D RET_INSN_SIZE; + code =3D &retinsn; break; } =20 --- a/arch/x86/lib/memmove_64.S +++ b/arch/x86/lib/memmove_64.S @@ -40,7 +40,7 @@ SYM_FUNC_START(__memmove) /* FSRM implies ERMS =3D> no length checks, do the copy directly */ .Lmemmove_begin_forward: ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM - ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; RET", X86_FEATURE_ERMS + ALTERNATIVE "", __stringify(movq %rdx, %rcx; rep movsb; RET), X86_FEATURE= _ERMS =20 /* * movsq instruction have many startup latency --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -34,7 +34,7 @@ SYM_INNER_LABEL(__x86_indirect_thunk_\re =20 ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ __stringify(RETPOLINE \reg), X86_FEATURE_RETPOLINE, \ - __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), X86_FEAT= URE_RETPOLINE_LFENCE + __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg; int3), X8= 6_FEATURE_RETPOLINE_LFENCE =20 .endm =20 --- a/scripts/Makefile.build +++ b/scripts/Makefile.build @@ -230,6 +230,9 @@ endif ifdef CONFIG_X86_SMAP objtool_args +=3D --uaccess endif +ifdef CONFIG_SLS + objtool_args +=3D --sls +endif =20 # 'OBJECT_FILES_NON_STANDARD :=3D y': skip objtool checking for a directory # 'OBJECT_FILES_NON_STANDARD_foo.o :=3D 'y': skip objtool checking for a f= ile --- a/scripts/link-vmlinux.sh +++ b/scripts/link-vmlinux.sh @@ -77,6 +77,9 @@ objtool_link() if [ -n "${CONFIG_X86_SMAP}" ]; then objtoolopt=3D"${objtoolopt} --uaccess" fi + if [ -n "${CONFIG_SLS}" ]; then + objtoolopt=3D"${objtoolopt} --sls" + fi info OBJTOOL ${1} tools/objtool/objtool ${objtoolopt} ${1} fi From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB823C433EF for ; Sat, 23 Jul 2022 10:04:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238250AbiGWKEp (ORCPT ); Sat, 23 Jul 2022 06:04:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238263AbiGWKD5 (ORCPT ); Sat, 23 Jul 2022 06:03:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE3FB90DA6; Sat, 23 Jul 2022 02:59:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EC9F56116A; Sat, 23 Jul 2022 09:59:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0547DC341C0; Sat, 23 Jul 2022 09:59:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570384; bh=eyipu8EB/64Q0rbW76zZlxYKVV59IRLRSGQ+HpDrRTQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u1VvcImO06aIjOLhtfvi7oR2M4KlHN5Oa5Yj7yPiwyqeXxHvcfQw891m3+v4wpfxf rxaDttjjxSVu0H4p66VASAoGXLWHRG6lxaRRYr8uUuxtH5srru4EYbYi0WalUllJeI YAdPyBU3s7cqytMmcEMnpa9OjYpcEJoMmFX698DE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Peter Zijlstra , Arnaldo Carvalho de Melo , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 067/148] tools arch: Update arch/x86/lib/mem{cpy,set}_64.S copies used in perf bench mem memcpy Date: Sat, 23 Jul 2022 11:54:39 +0200 Message-Id: <20220723095243.011444614@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Arnaldo Carvalho de Melo commit 35cb8c713a496e8c114eed5e2a5a30b359876df2 upstream. To bring in the change made in this cset: f94909ceb1ed4bfd ("x86: Prepare asm files for straight-line-speculation") It silences these perf tools build warnings, no change in the tools: Warning: Kernel ABI header at 'tools/arch/x86/lib/memcpy_64.S' differs fr= om latest version at 'arch/x86/lib/memcpy_64.S' diff -u tools/arch/x86/lib/memcpy_64.S arch/x86/lib/memcpy_64.S Warning: Kernel ABI header at 'tools/arch/x86/lib/memset_64.S' differs fr= om latest version at 'arch/x86/lib/memset_64.S' diff -u tools/arch/x86/lib/memset_64.S arch/x86/lib/memset_64.S The code generated was checked before and after using 'objdump -d /tmp/buil= d/perf/bench/mem-memcpy-x86-64-asm.o', no changes. Cc: Borislav Petkov Cc: Peter Zijlstra Signed-off-by: Arnaldo Carvalho de Melo Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/arch/x86/lib/memcpy_64.S | 12 ++++++------ tools/arch/x86/lib/memset_64.S | 6 +++--- 2 files changed, 9 insertions(+), 9 deletions(-) --- a/tools/arch/x86/lib/memcpy_64.S +++ b/tools/arch/x86/lib/memcpy_64.S @@ -39,7 +39,7 @@ SYM_FUNC_START_WEAK(memcpy) rep movsq movl %edx, %ecx rep movsb - ret + RET SYM_FUNC_END(memcpy) SYM_FUNC_END_ALIAS(__memcpy) EXPORT_SYMBOL(memcpy) @@ -53,7 +53,7 @@ SYM_FUNC_START_LOCAL(memcpy_erms) movq %rdi, %rax movq %rdx, %rcx rep movsb - ret + RET SYM_FUNC_END(memcpy_erms) =20 SYM_FUNC_START_LOCAL(memcpy_orig) @@ -137,7 +137,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movq %r9, 1*8(%rdi) movq %r10, -2*8(%rdi, %rdx) movq %r11, -1*8(%rdi, %rdx) - retq + RET .p2align 4 .Lless_16bytes: cmpl $8, %edx @@ -149,7 +149,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movq -1*8(%rsi, %rdx), %r9 movq %r8, 0*8(%rdi) movq %r9, -1*8(%rdi, %rdx) - retq + RET .p2align 4 .Lless_8bytes: cmpl $4, %edx @@ -162,7 +162,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movl -4(%rsi, %rdx), %r8d movl %ecx, (%rdi) movl %r8d, -4(%rdi, %rdx) - retq + RET .p2align 4 .Lless_3bytes: subl $1, %edx @@ -180,7 +180,7 @@ SYM_FUNC_START_LOCAL(memcpy_orig) movb %cl, (%rdi) =20 .Lend: - retq + RET SYM_FUNC_END(memcpy_orig) =20 .popsection --- a/tools/arch/x86/lib/memset_64.S +++ b/tools/arch/x86/lib/memset_64.S @@ -40,7 +40,7 @@ SYM_FUNC_START(__memset) movl %edx,%ecx rep stosb movq %r9,%rax - ret + RET SYM_FUNC_END(__memset) SYM_FUNC_END_ALIAS(memset) EXPORT_SYMBOL(memset) @@ -63,7 +63,7 @@ SYM_FUNC_START_LOCAL(memset_erms) movq %rdx,%rcx rep stosb movq %r9,%rax - ret + RET SYM_FUNC_END(memset_erms) =20 SYM_FUNC_START_LOCAL(memset_orig) @@ -125,7 +125,7 @@ SYM_FUNC_START_LOCAL(memset_orig) =20 .Lende: movq %r10,%rax - ret + RET =20 .Lbad_alignment: cmpq $7,%rdx From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EE39C43334 for ; Sat, 23 Jul 2022 10:04:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238194AbiGWKEl (ORCPT ); Sat, 23 Jul 2022 06:04:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238240AbiGWKEB (ORCPT ); Sat, 23 Jul 2022 06:04:01 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 350709368D; Sat, 23 Jul 2022 02:59:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A7B65B82C1B; Sat, 23 Jul 2022 09:59:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DBB24C341C0; Sat, 23 Jul 2022 09:59:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570387; bh=NTrzfJvQ19VyAP61TafzQNav55FmjRegdN7rwksTswA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U4SJ6ZB7pfTJHLGWUkDWwIVKCb3oz6+r+jzqw2wZvQkpPl7PCBcRvfZWk4+uQNxE7 l5aIoQPW8CjWVEyI78X10b/UMj5cn10rP4fzZzpLKoW/SdIIoRHVWSCxPDVkneX06l 5C15pTWmo+jU60rYq39RKGe7sr1GA5Dgsyx1K06k= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Jamie Heilman , Borislav Petkov , "Peter Zijlstra (Intel)" , Paolo Bonzini , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 068/148] kvm/emulate: Fix SETcc emulation function offsets with SLS Date: Sat, 23 Jul 2022 11:54:40 +0200 Message-Id: <20220723095243.324774965@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit fe83f5eae432ccc8e90082d6ed506d5233547473 upstream. The commit in Fixes started adding INT3 after RETs as a mitigation against straight-line speculation. The fastop SETcc implementation in kvm's insn emulator uses macro magic to generate all possible SETcc functions and to jump to them when emulating the respective instruction. However, it hardcodes the size and alignment of those functions to 4: a three-byte SETcc insn and a single-byte RET. BUT, with SLS, there's an INT3 that gets slapped after the RET, which brings the whole scheme out of alignment: 15: 0f 90 c0 seto %al 18: c3 ret 19: cc int3 1a: 0f 1f 00 nopl (%rax) 1d: 0f 91 c0 setno %al 20: c3 ret 21: cc int3 22: 0f 1f 00 nopl (%rax) 25: 0f 92 c0 setb %al 28: c3 ret 29: cc int3 and this explodes like this: int3: 0000 [#1] PREEMPT SMP PTI CPU: 0 PID: 2435 Comm: qemu-system-x86 Not tainted 5.17.0-rc8-sls #1 Hardware name: Dell Inc. Precision WorkStation T3400 /0TP412, BIOS A14 0= 4/30/2012 RIP: 0010:setc+0x5/0x8 [kvm] Code: 00 00 0f 1f 00 0f b6 05 43 24 06 00 c3 cc 0f 1f 80 00 00 00 00 0f 9= 0 c0 c3 cc 0f \ 1f 00 0f 91 c0 c3 cc 0f 1f 00 0f 92 c0 c3 cc <0f> 1f 00 0f 93 c0 c3 cc 0= f 1f 00 \ 0f 94 c0 c3 cc 0f 1f 00 0f 95 c0 Call Trace: ? x86_emulate_insn [kvm] ? x86_emulate_instruction [kvm] ? vmx_handle_exit [kvm_intel] ? kvm_arch_vcpu_ioctl_run [kvm] ? kvm_vcpu_ioctl [kvm] ? __x64_sys_ioctl ? do_syscall_64 ? entry_SYSCALL_64_after_hwframe Raise the alignment value when SLS is enabled and use a macro for that instead of hard-coding naked numbers. Fixes: e463a09af2f0 ("x86: Add straight-line-speculation mitigation") Reported-by: Jamie Heilman Signed-off-by: Borislav Petkov Acked-by: Peter Zijlstra (Intel) Tested-by: Jamie Heilman Link: https://lore.kernel.org/r/YjGzJwjrvxg5YZ0Z@audible.transient.net [Add a comment and a bit of safety checking, since this is going to be chan= ged again for IBT support. - Paolo] Signed-off-by: Paolo Bonzini Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/emulate.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -428,8 +428,23 @@ static int fastop(struct x86_emulate_ctx FOP_END =20 /* Special case for SETcc - 1 instruction per cc */ + +/* + * Depending on .config the SETcc functions look like: + * + * SETcc %al [3 bytes] + * RET [1 byte] + * INT3 [1 byte; CONFIG_SLS] + * + * Which gives possible sizes 4 or 5. When rounded up to the + * next power-of-two alignment they become 4 or 8. + */ +#define SETCC_LENGTH (4 + IS_ENABLED(CONFIG_SLS)) +#define SETCC_ALIGN (4 << IS_ENABLED(CONFIG_SLS)) +static_assert(SETCC_LENGTH <=3D SETCC_ALIGN); + #define FOP_SETCC(op) \ - ".align 4 \n\t" \ + ".align " __stringify(SETCC_ALIGN) " \n\t" \ ".type " #op ", @function \n\t" \ #op ": \n\t" \ #op " %al \n\t" \ @@ -1055,7 +1070,7 @@ static int em_bsr_c(struct x86_emulate_c static __always_inline u8 test_cc(unsigned int condition, unsigned long fl= ags) { u8 rc; - void (*fop)(void) =3D (void *)em_setcc + 4 * (condition & 0xf); + void (*fop)(void) =3D (void *)em_setcc + SETCC_ALIGN * (condition & 0xf); =20 flags =3D (flags & EFLAGS_MASK) | X86_EFLAGS_IF; asm("push %[flags]; popf; " CALL_NOSPEC From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C6FAC433EF for ; Sat, 23 Jul 2022 10:05:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233691AbiGWKFA (ORCPT ); Sat, 23 Jul 2022 06:05:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238467AbiGWKEZ (ORCPT ); Sat, 23 Jul 2022 06:04:25 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFBF49B9F1; Sat, 23 Jul 2022 03:00:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B205961204; Sat, 23 Jul 2022 09:59:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0475C341C0; Sat, 23 Jul 2022 09:59:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570390; bh=a3uaGNTJRCN1AZgQunj5sehXg5YFviJZKOEKcHUEBxY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TygPaWoA+fBLfJ0qPdyl6WYxlRAXM7gLTAp9/Z3fADwWN5X64AXo/ssJWS0h4lSS4 LG21cS4Zj9Zs9Mgi1PUCcPORvbr6LRaQ4phOHFj1ReaQFNfbS6+aMPb/ItdngaJHks mBZfXHQAWLzHjy0wEeAjHujq4dX4iceZEc3df81c= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 069/148] objtool: Default ignore INT3 for unreachable Date: Sat, 23 Jul 2022 11:54:41 +0200 Message-Id: <20220723095243.581466839@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 1ffbe4e935f9b7308615c75be990aec07464d1e7 upstream. Ignore all INT3 instructions for unreachable code warnings, similar to NOP. This allows using INT3 for various paddings instead of NOPs. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Josh Poimboeuf Link: https://lore.kernel.org/r/20220308154317.343312938@infradead.org Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2775,9 +2775,8 @@ static int validate_branch(struct objtoo switch (insn->type) { =20 case INSN_RETURN: - if (next_insn && next_insn->type =3D=3D INSN_TRAP) { - next_insn->ignore =3D true; - } else if (sls && !insn->retpoline_safe) { + if (sls && !insn->retpoline_safe && + next_insn && next_insn->type !=3D INSN_TRAP) { WARN_FUNC("missing int3 after ret", insn->sec, insn->offset); } @@ -2824,9 +2823,8 @@ static int validate_branch(struct objtoo break; =20 case INSN_JUMP_DYNAMIC: - if (next_insn && next_insn->type =3D=3D INSN_TRAP) { - next_insn->ignore =3D true; - } else if (sls && !insn->retpoline_safe) { + if (sls && !insn->retpoline_safe && + next_insn && next_insn->type !=3D INSN_TRAP) { WARN_FUNC("missing int3 after indirect jump", insn->sec, insn->offset); } @@ -2997,7 +2995,7 @@ static bool ignore_unreachable_insn(stru int i; struct instruction *prev_insn; =20 - if (insn->ignore || insn->type =3D=3D INSN_NOP) + if (insn->ignore || insn->type =3D=3D INSN_NOP || insn->type =3D=3D INSN_= TRAP) return true; =20 /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0403FC433EF for ; Sat, 23 Jul 2022 10:03:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238230AbiGWKDn (ORCPT ); Sat, 23 Jul 2022 06:03:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237922AbiGWKDR (ORCPT ); Sat, 23 Jul 2022 06:03:17 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 972FF87222; Sat, 23 Jul 2022 02:59:24 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 99DE86116A; Sat, 23 Jul 2022 09:59:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A1EB5C341C0; Sat, 23 Jul 2022 09:59:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570361; bh=DYakpdbPRHY7KHTWH+Lrd6E6Ay+oTi6Mb+KyBy/h+A8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hSlykiNOF6NfbYtd5zTGivPsYnLmfvBnIKWvmDJytSMozIv0FhMiL6nD9DiSni/Hu fkkC+WH9pHbqVz30RmcNrb4dH7KFXPBglEARTSE3BEFq3I7v7F4fbkXi5Oq1NjXjiY 49lC72Ofy9t3nUD/4WA59Naaf3Lt10q58Hnc6MOw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Herbert Xu , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 070/148] crypto: x86/poly1305 - Fixup SLS Date: Sat, 23 Jul 2022 11:54:42 +0200 Message-Id: <20220723095243.840940616@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 7ed7aa4de9421229be6d331ed52d5cd09c99f409 upstream. Due to being a perl generated asm file, it got missed by the mass convertion script. arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_in= it_x86_64()+0x3a: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_x86_64()+0xf2: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_em= it_x86_64()+0x37: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: __poly1305_= block()+0x6d: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: __poly1305_= init_avx()+0x1e8: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_avx()+0x18a: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_avx()+0xaf8: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_em= it_avx()+0x99: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_avx2()+0x18a: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_avx2()+0x776: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_avx512()+0x18a: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_avx512()+0x796: missing int3 after ret arch/x86/crypto/poly1305-x86_64-cryptogams.o: warning: objtool: poly1305_bl= ocks_avx512()+0x10bd: missing int3 after ret Fixes: f94909ceb1ed ("x86: Prepare asm files for straight-line-speculation") Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Herbert Xu Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/crypto/poly1305-x86_64-cryptogams.pl | 38 +++++++++++++--------= ----- 1 file changed, 19 insertions(+), 19 deletions(-) --- a/arch/x86/crypto/poly1305-x86_64-cryptogams.pl +++ b/arch/x86/crypto/poly1305-x86_64-cryptogams.pl @@ -297,7 +297,7 @@ ___ $code.=3D<<___; mov \$1,%eax .Lno_key: - ret + RET ___ &end_function("poly1305_init_x86_64"); =20 @@ -373,7 +373,7 @@ $code.=3D<<___; .cfi_adjust_cfa_offset -48 .Lno_data: .Lblocks_epilogue: - ret + RET .cfi_endproc ___ &end_function("poly1305_blocks_x86_64"); @@ -399,7 +399,7 @@ $code.=3D<<___; mov %rax,0($mac) # write result mov %rcx,8($mac) =20 - ret + RET ___ &end_function("poly1305_emit_x86_64"); if ($avx) { @@ -429,7 +429,7 @@ ___ &poly1305_iteration(); $code.=3D<<___; pop $ctx - ret + RET .size __poly1305_block,.-__poly1305_block =20 .type __poly1305_init_avx,\@abi-omnipotent @@ -594,7 +594,7 @@ __poly1305_init_avx: =20 lea -48-64($ctx),$ctx # size [de-]optimization pop %rbp - ret + RET .size __poly1305_init_avx,.-__poly1305_init_avx ___ =20 @@ -747,7 +747,7 @@ $code.=3D<<___; .cfi_restore %rbp .Lno_data_avx: .Lblocks_avx_epilogue: - ret + RET .cfi_endproc =20 .align 32 @@ -1452,7 +1452,7 @@ $code.=3D<<___ if (!$win64); ___ $code.=3D<<___; vzeroupper - ret + RET .cfi_endproc ___ &end_function("poly1305_blocks_avx"); @@ -1508,7 +1508,7 @@ $code.=3D<<___; mov %rax,0($mac) # write result mov %rcx,8($mac) =20 - ret + RET ___ &end_function("poly1305_emit_avx"); =20 @@ -1675,7 +1675,7 @@ $code.=3D<<___; .cfi_restore %rbp .Lno_data_avx2$suffix: .Lblocks_avx2_epilogue$suffix: - ret + RET .cfi_endproc =20 .align 32 @@ -2201,7 +2201,7 @@ $code.=3D<<___ if (!$win64); ___ $code.=3D<<___; vzeroupper - ret + RET .cfi_endproc ___ if($avx > 2 && $avx512) { @@ -2792,7 +2792,7 @@ $code.=3D<<___ if (!$win64); .cfi_def_cfa_register %rsp ___ $code.=3D<<___; - ret + RET .cfi_endproc ___ =20 @@ -2893,7 +2893,7 @@ $code.=3D<<___ if ($flavour =3D~ /elf32/); ___ $code.=3D<<___; mov \$1,%eax - ret + RET .size poly1305_init_base2_44,.-poly1305_init_base2_44 ___ { @@ -3010,7 +3010,7 @@ poly1305_blocks_vpmadd52: jnz .Lblocks_vpmadd52_4x =20 .Lno_data_vpmadd52: - ret + RET .size poly1305_blocks_vpmadd52,.-poly1305_blocks_vpmadd52 ___ } @@ -3451,7 +3451,7 @@ poly1305_blocks_vpmadd52_4x: vzeroall =20 .Lno_data_vpmadd52_4x: - ret + RET .size poly1305_blocks_vpmadd52_4x,.-poly1305_blocks_vpmadd52_4x ___ } @@ -3824,7 +3824,7 @@ $code.=3D<<___; vzeroall =20 .Lno_data_vpmadd52_8x: - ret + RET .size poly1305_blocks_vpmadd52_8x,.-poly1305_blocks_vpmadd52_8x ___ } @@ -3861,7 +3861,7 @@ poly1305_emit_base2_44: mov %rax,0($mac) # write result mov %rcx,8($mac) =20 - ret + RET .size poly1305_emit_base2_44,.-poly1305_emit_base2_44 ___ } } } @@ -3916,7 +3916,7 @@ xor128_encrypt_n_pad: =20 .Ldone_enc: mov $otp,%rax - ret + RET .size xor128_encrypt_n_pad,.-xor128_encrypt_n_pad =20 .globl xor128_decrypt_n_pad @@ -3967,7 +3967,7 @@ xor128_decrypt_n_pad: =20 .Ldone_dec: mov $otp,%rax - ret + RET .size xor128_decrypt_n_pad,.-xor128_decrypt_n_pad ___ } @@ -4109,7 +4109,7 @@ avx_handler: pop %rbx pop %rdi pop %rsi - ret + RET .size avx_handler,.-avx_handler =20 .section .pdata From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DD68C43334 for ; Sat, 23 Jul 2022 10:03:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238111AbiGWKDw (ORCPT ); Sat, 23 Jul 2022 06:03:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238105AbiGWKDY (ORCPT ); Sat, 23 Jul 2022 06:03:24 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB5CC88F0A; Sat, 23 Jul 2022 02:59:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 619FDB82C1B; Sat, 23 Jul 2022 09:59:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81151C341C0; Sat, 23 Jul 2022 09:59:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570364; bh=v2jQRb/cwYZnNJNo6P3W1xJ1OpzzWBGK68XPPMII0hA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S2JjE2osv5Vzw8YU+AAwf84S+N/wjxQxQJD7lznZ8SjmQ0kxIJ5mR7GhmFbdp8/vy 3C52MhcSO25N91GaqUbJdDOz/JDtG4jaSOXRHbcih1md0ga5ggKxfd/1MWxmN0B/41 NVUNhql//TGL1bCmThYbvckmV80nUpHiJIF5gtqs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Sasha Levin , Ben Hutchings Subject: [PATCH 5.10 071/148] objtool: Fix SLS validation for kcov tail-call replacement Date: Sat, 23 Jul 2022 11:54:43 +0200 Message-Id: <20220723095244.084751710@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 7a53f408902d913cd541b4f8ad7dbcd4961f5b82 upstream. Since not all compilers have a function attribute to disable KCOV instrumentation, objtool can rewrite KCOV instrumentation in noinstr functions as per commit: f56dae88a81f ("objtool: Handle __sanitize_cov*() tail calls") However, this has subtle interaction with the SLS validation from commit: 1cc1e4c8aab4 ("objtool: Add straight-line-speculation validation") In that when a tail-call instrucion is replaced with a RET an additional INT3 instruction is also written, but is not represented in the decoded instruction stream. This then leads to false positive missing INT3 objtool warnings in noinstr code. Instead of adding additional struct instruction objects, mark the RET instruction with retpoline_safe to suppress the warning (since we know there really is an INT3). Fixes: 1cc1e4c8aab4 ("objtool: Add straight-line-speculation validation") Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20220323230712.GA8939@worktop.programming.k= icks-ass.net Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 11 +++++++++++ 1 file changed, 11 insertions(+) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -961,6 +961,17 @@ static void annotate_call_site(struct ob : arch_nop_insn(insn->len)); =20 insn->type =3D sibling ? INSN_RETURN : INSN_NOP; + + if (sibling) { + /* + * We've replaced the tail-call JMP insn by two new + * insn: RET; INT3, except we only have a single struct + * insn here. Mark it retpoline_safe to avoid the SLS + * warning, instead of adding another insn. + */ + insn->retpoline_safe =3D true; + } + return; } } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8ED8C43334 for ; Sat, 23 Jul 2022 10:04:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238032AbiGWKEg (ORCPT ); Sat, 23 Jul 2022 06:04:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238161AbiGWKD4 (ORCPT ); Sat, 23 Jul 2022 06:03:56 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 323608F530; Sat, 23 Jul 2022 02:59:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3459DB82C1F; Sat, 23 Jul 2022 09:59:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 95ACBC341C0; Sat, 23 Jul 2022 09:59:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570367; bh=efQhmefNJjZgawAawcRoXoGa/B0C/xqSoQnQCU4vV4I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E0o8ov/u46RcHQ2U2htLBO3nEpXSonJxxS+BdF5PC1rPFePYhEtoJ4niLS+g1LjAZ N1/qSoZXyzWoFZzukKBM31Zdb/IPgaFReUVMpFtZ6xn2F8JXy5kh+Fs8Cpzlj49mR2 21rSxoSWv2RUzzMrttSLwHuJ3e3Z8PuhNbr2ElTM= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 072/148] objtool: Fix code relocs vs weak symbols Date: Sat, 23 Jul 2022 11:54:44 +0200 Message-Id: <20220723095244.360855208@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 4abff6d48dbcea8200c7ea35ba70c242d128ebf3 upstream. Occasionally objtool driven code patching (think .static_call_sites .retpoline_sites etc..) goes sideways and it tries to patch an instruction that doesn't match. Much head-scatching and cursing later the problem is as outlined below and affects every section that objtool generates for us, very much including the ORC data. The below uses .static_call_sites because it's convenient for demonstration purposes, but as mentioned the ORC sections, .retpoline_sites and __mount_loc are all similarly affected. Consider: foo-weak.c: extern void __SCT__foo(void); __attribute__((weak)) void foo(void) { return __SCT__foo(); } foo.c: extern void __SCT__foo(void); extern void my_foo(void); void foo(void) { my_foo(); return __SCT__foo(); } These generate the obvious code (gcc -O2 -fcf-protection=3Dnone -fno-asynchronous-unwind-tables -c foo*.c): foo-weak.o: 0000000000000000 : 0: e9 00 00 00 00 jmpq 5 1: R_X86_64_PLT32 = __SCT__foo-0x4 foo.o: 0000000000000000 : 0: 48 83 ec 08 sub $0x8,%rsp 4: e8 00 00 00 00 callq 9 5: R_X86_64_PLT32 = my_foo-0x4 9: 48 83 c4 08 add $0x8,%rsp d: e9 00 00 00 00 jmpq 12 e: R_X86_64_PLT32 = __SCT__foo-0x4 Now, when we link these two files together, you get something like (ld -r -o foos.o foo-weak.o foo.o): foos.o: 0000000000000000 : 0: e9 00 00 00 00 jmpq 5 1: R_X86_64_PLT32 = __SCT__foo-0x4 5: 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:0x0(%rax,%rax,1) f: 90 nop 0000000000000010 : 10: 48 83 ec 08 sub $0x8,%rsp 14: e8 00 00 00 00 callq 19 15: R_X86_64_PLT32 = my_foo-0x4 19: 48 83 c4 08 add $0x8,%rsp 1d: e9 00 00 00 00 jmpq 22 1e: R_X86_64_PLT32 = __SCT__foo-0x4 Noting that ld preserves the weak function text, but strips the symbol off of it (hence objdump doing that funny negative offset thing). This does lead to 'interesting' unused code issues with objtool when ran on linked objects, but that seems to be working (fingers crossed). So far so good.. Now lets consider the objtool static_call output section (readelf output, old binutils): foo-weak.o: Relocation section '.rela.static_call_sites' at offset 0x2c8 contains 1 ent= ry: Offset Info Type Symbol's Value = Symbol's Name + Addend 0000000000000000 0000000200000002 R_X86_64_PC32 0000000000000000 = .text + 0 0000000000000004 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 foo.o: Relocation section '.rela.static_call_sites' at offset 0x310 contains 2 ent= ries: Offset Info Type Symbol's Value = Symbol's Name + Addend 0000000000000000 0000000200000002 R_X86_64_PC32 0000000000000000 = .text + d 0000000000000004 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 foos.o: Relocation section '.rela.static_call_sites' at offset 0x430 contains 4 ent= ries: Offset Info Type Symbol's Value = Symbol's Name + Addend 0000000000000000 0000000100000002 R_X86_64_PC32 0000000000000000 = .text + 0 0000000000000004 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 0000000000000008 0000000100000002 R_X86_64_PC32 0000000000000000 = .text + 1d 000000000000000c 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 So we have two patch sites, one in the dead code of the weak foo and one in the real foo. All is well. *HOWEVER*, when the toolchain strips unused section symbols it generates things like this (using new enough binutils): foo-weak.o: Relocation section '.rela.static_call_sites' at offset 0x2c8 contains 1 ent= ry: Offset Info Type Symbol's Value = Symbol's Name + Addend 0000000000000000 0000000200000002 R_X86_64_PC32 0000000000000000 = foo + 0 0000000000000004 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 foo.o: Relocation section '.rela.static_call_sites' at offset 0x310 contains 2 ent= ries: Offset Info Type Symbol's Value = Symbol's Name + Addend 0000000000000000 0000000200000002 R_X86_64_PC32 0000000000000000 = foo + d 0000000000000004 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 foos.o: Relocation section '.rela.static_call_sites' at offset 0x430 contains 4 ent= ries: Offset Info Type Symbol's Value = Symbol's Name + Addend 0000000000000000 0000000100000002 R_X86_64_PC32 0000000000000000 = foo + 0 0000000000000004 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 0000000000000008 0000000100000002 R_X86_64_PC32 0000000000000000 = foo + d 000000000000000c 0000000d00000002 R_X86_64_PC32 0000000000000000 = __SCT__foo + 1 And now we can see how that foos.o .static_call_sites goes side-ways, we now have _two_ patch sites in foo. One for the weak symbol at foo+0 (which is no longer a static_call site!) and one at foo+d which is in fact the right location. This seems to happen when objtool cannot find a section symbol, in which case it falls back to any other symbol to key off of, however in this case that goes terribly wrong! As such, teach objtool to create a section symbol when there isn't one. Fixes: 44f6a7c0755d ("objtool: Fix seg fault with Clang non-section symbols= ") Signed-off-by: Peter Zijlstra (Intel) Acked-by: Josh Poimboeuf Link: https://lkml.kernel.org/r/20220419203807.655552918@infradead.org Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/elf.c | 187 +++++++++++++++++++++++++++++++++++++++++++++--= ----- 1 file changed, 165 insertions(+), 22 deletions(-) --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -537,37 +537,180 @@ int elf_add_reloc(struct elf *elf, struc return 0; } =20 -int elf_add_reloc_to_insn(struct elf *elf, struct section *sec, - unsigned long offset, unsigned int type, - struct section *insn_sec, unsigned long insn_off) +/* + * Ensure that any reloc section containing references to @sym is marked + * changed such that it will get re-generated in elf_rebuild_reloc_section= s() + * with the new symbol index. + */ +static void elf_dirty_reloc_sym(struct elf *elf, struct symbol *sym) +{ + struct section *sec; + + list_for_each_entry(sec, &elf->sections, list) { + struct reloc *reloc; + + if (sec->changed) + continue; + + list_for_each_entry(reloc, &sec->reloc_list, list) { + if (reloc->sym =3D=3D sym) { + sec->changed =3D true; + break; + } + } + } +} + +/* + * Move the first global symbol, as per sh_info, into a new, higher symbol + * index. This fees up the shndx for a new local symbol. + */ +static int elf_move_global_symbol(struct elf *elf, struct section *symtab, + struct section *symtab_shndx) { + Elf_Data *data, *shndx_data =3D NULL; + Elf32_Word first_non_local; struct symbol *sym; - int addend; + Elf_Scn *s; =20 - if (insn_sec->sym) { - sym =3D insn_sec->sym; - addend =3D insn_off; + first_non_local =3D symtab->sh.sh_info; =20 - } else { - /* - * The Clang assembler strips section symbols, so we have to - * reference the function symbol instead: - */ - sym =3D find_symbol_containing(insn_sec, insn_off); - if (!sym) { - /* - * Hack alert. This happens when we need to reference - * the NOP pad insn immediately after the function. - */ - sym =3D find_symbol_containing(insn_sec, insn_off - 1); + sym =3D find_symbol_by_index(elf, first_non_local); + if (!sym) { + WARN("no non-local symbols !?"); + return first_non_local; + } + + s =3D elf_getscn(elf->elf, symtab->idx); + if (!s) { + WARN_ELF("elf_getscn"); + return -1; + } + + data =3D elf_newdata(s); + if (!data) { + WARN_ELF("elf_newdata"); + return -1; + } + + data->d_buf =3D &sym->sym; + data->d_size =3D sizeof(sym->sym); + data->d_align =3D 1; + data->d_type =3D ELF_T_SYM; + + sym->idx =3D symtab->sh.sh_size / sizeof(sym->sym); + elf_dirty_reloc_sym(elf, sym); + + symtab->sh.sh_info +=3D 1; + symtab->sh.sh_size +=3D data->d_size; + symtab->changed =3D true; + + if (symtab_shndx) { + s =3D elf_getscn(elf->elf, symtab_shndx->idx); + if (!s) { + WARN_ELF("elf_getscn"); + return -1; } =20 - if (!sym) { - WARN("can't find symbol containing %s+0x%lx", insn_sec->name, insn_off); + shndx_data =3D elf_newdata(s); + if (!shndx_data) { + WARN_ELF("elf_newshndx_data"); return -1; } =20 - addend =3D insn_off - sym->offset; + shndx_data->d_buf =3D &sym->sec->idx; + shndx_data->d_size =3D sizeof(Elf32_Word); + shndx_data->d_align =3D 4; + shndx_data->d_type =3D ELF_T_WORD; + + symtab_shndx->sh.sh_size +=3D 4; + symtab_shndx->changed =3D true; + } + + return first_non_local; +} + +static struct symbol * +elf_create_section_symbol(struct elf *elf, struct section *sec) +{ + struct section *symtab, *symtab_shndx; + Elf_Data *shndx_data =3D NULL; + struct symbol *sym; + Elf32_Word shndx; + + symtab =3D find_section_by_name(elf, ".symtab"); + if (symtab) { + symtab_shndx =3D find_section_by_name(elf, ".symtab_shndx"); + if (symtab_shndx) + shndx_data =3D symtab_shndx->data; + } else { + WARN("no .symtab"); + return NULL; + } + + sym =3D malloc(sizeof(*sym)); + if (!sym) { + perror("malloc"); + return NULL; + } + memset(sym, 0, sizeof(*sym)); + + sym->idx =3D elf_move_global_symbol(elf, symtab, symtab_shndx); + if (sym->idx < 0) { + WARN("elf_move_global_symbol"); + return NULL; + } + + sym->name =3D sec->name; + sym->sec =3D sec; + + // st_name 0 + sym->sym.st_info =3D GELF_ST_INFO(STB_LOCAL, STT_SECTION); + // st_other 0 + // st_value 0 + // st_size 0 + shndx =3D sec->idx; + if (shndx >=3D SHN_UNDEF && shndx < SHN_LORESERVE) { + sym->sym.st_shndx =3D shndx; + if (!shndx_data) + shndx =3D 0; + } else { + sym->sym.st_shndx =3D SHN_XINDEX; + if (!shndx_data) { + WARN("no .symtab_shndx"); + return NULL; + } + } + + if (!gelf_update_symshndx(symtab->data, shndx_data, sym->idx, &sym->sym, = shndx)) { + WARN_ELF("gelf_update_symshndx"); + return NULL; + } + + elf_add_symbol(elf, sym); + + return sym; +} + +int elf_add_reloc_to_insn(struct elf *elf, struct section *sec, + unsigned long offset, unsigned int type, + struct section *insn_sec, unsigned long insn_off) +{ + struct symbol *sym =3D insn_sec->sym; + int addend =3D insn_off; + + if (!sym) { + /* + * Due to how weak functions work, we must use section based + * relocations. Symbol based relocations would result in the + * weak and non-weak function annotations being overlaid on the + * non-weak function after linking. + */ + sym =3D elf_create_section_symbol(elf, insn_sec); + if (!sym) + return -1; + + insn_sec->sym =3D sym; } =20 return elf_add_reloc(elf, sec, offset, type, sym, addend); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49317C43334 for ; Sat, 23 Jul 2022 10:06:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238596AbiGWKGn (ORCPT ); Sat, 23 Jul 2022 06:06:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238438AbiGWKGX (ORCPT ); Sat, 23 Jul 2022 06:06:23 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6320EBE9D5; Sat, 23 Jul 2022 03:00:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3153CB82B92; Sat, 23 Jul 2022 10:00:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28A6CC341CB; Sat, 23 Jul 2022 10:00:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570427; bh=mhfFvQtBEybVbqarS8ueWbYRtpojy7GFRiJPB7rBpNU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=17k6NY21qXSisx3yneCWsyd3Ws7rvFYUsy05vRCRq0ZrqaLtY8vroGo/GD7e3WIFz 1miaWNV5meExgozjiqhzwI8hNimh3HTYTAdjgkXbWfHyzAy70DngEGU2bsn22rcPV7 8YlIZ8a705n/l9n26h3ROYT3pRjLjOUtbh0CqpbE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 073/148] objtool: Fix type of reloc::addend Date: Sat, 23 Jul 2022 11:54:45 +0200 Message-Id: <20220723095244.628537805@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit c087c6e7b551b7f208c0b852304f044954cf2bb3 upstream. Elf{32,64}_Rela::r_addend is of type: Elf{32,64}_Sword, that means that our reloc::addend needs to be long or face tuncation issues when we do elf_rebuild_reloc_section(): - 107: 48 b8 00 00 00 00 00 00 00 00 movabs $0x0,%rax 109: R_X8= 6_64_64 level4_kernel_pgt+0x80000067 + 107: 48 b8 00 00 00 00 00 00 00 00 movabs $0x0,%rax 109: R_X8= 6_64_64 level4_kernel_pgt-0x7fffff99 Fixes: 627fce14809b ("objtool: Add ORC unwind table generation") Signed-off-by: Peter Zijlstra (Intel) Acked-by: Josh Poimboeuf Link: https://lkml.kernel.org/r/20220419203807.596871927@infradead.org Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 8 ++++---- tools/objtool/elf.c | 2 +- tools/objtool/elf.h | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -467,12 +467,12 @@ static int add_dead_ends(struct objtool_ else if (reloc->addend =3D=3D reloc->sym->sec->len) { insn =3D find_last_insn(file, reloc->sym->sec); if (!insn) { - WARN("can't find unreachable insn at %s+0x%x", + WARN("can't find unreachable insn at %s+0x%lx", reloc->sym->sec->name, reloc->addend); return -1; } } else { - WARN("can't find unreachable insn at %s+0x%x", + WARN("can't find unreachable insn at %s+0x%lx", reloc->sym->sec->name, reloc->addend); return -1; } @@ -502,12 +502,12 @@ reachable: else if (reloc->addend =3D=3D reloc->sym->sec->len) { insn =3D find_last_insn(file, reloc->sym->sec); if (!insn) { - WARN("can't find reachable insn at %s+0x%x", + WARN("can't find reachable insn at %s+0x%lx", reloc->sym->sec->name, reloc->addend); return -1; } } else { - WARN("can't find reachable insn at %s+0x%x", + WARN("can't find reachable insn at %s+0x%lx", reloc->sym->sec->name, reloc->addend); return -1; } --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -509,7 +509,7 @@ static struct section *elf_create_reloc_ int reltype); =20 int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, - unsigned int type, struct symbol *sym, int addend) + unsigned int type, struct symbol *sym, long addend) { struct reloc *reloc; =20 --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -73,7 +73,7 @@ struct reloc { struct symbol *sym; unsigned long offset; unsigned int type; - int addend; + long addend; int idx; bool jump_table_start; }; @@ -127,7 +127,7 @@ struct elf *elf_open_read(const char *na struct section *elf_create_section(struct elf *elf, const char *name, unsi= gned int sh_flags, size_t entsize, int nr); =20 int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, - unsigned int type, struct symbol *sym, int addend); + unsigned int type, struct symbol *sym, long addend); int elf_add_reloc_to_insn(struct elf *elf, struct section *sec, unsigned long offset, unsigned int type, struct section *insn_sec, unsigned long insn_off); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6882BC43334 for ; Sat, 23 Jul 2022 10:05:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238393AbiGWKFE (ORCPT ); Sat, 23 Jul 2022 06:05:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238537AbiGWKE3 (ORCPT ); Sat, 23 Jul 2022 06:04:29 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6C6EBA2; Sat, 23 Jul 2022 03:00:05 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 702D660C81; Sat, 23 Jul 2022 09:59:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7D80EC385A5; Sat, 23 Jul 2022 09:59:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570395; bh=yvRta9CPHVpUzVxgqMCZSu+Lo934yTJmq1xlRyl5HUU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GqQYqFg6euansEutD6wJrBbnWrMfzH64dQD/mk+V/Mw8soDVPGHNgFNGANff+jzzX +LiiQzzAxDfRq1kj8elU9jXHt/zGzUlGzcrsD1cjx984hbxsWYIxezcL1gQYZQpi27 1O+3IaBrv/FzBQnxNAqcuogZ7DTSTYvR0128Mrco= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Nathan Chancellor , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 074/148] objtool: Fix symbol creation Date: Sat, 23 Jul 2022 11:54:46 +0200 Message-Id: <20220723095244.898814286@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit ead165fa1042247b033afad7be4be9b815d04ade upstream. Nathan reported objtool failing with the following messages: warning: objtool: no non-local symbols !? warning: objtool: gelf_update_symshndx: invalid section index The problem is due to commit 4abff6d48dbc ("objtool: Fix code relocs vs weak symbols") failing to consider the case where an object would have no non-local symbols. The problem that commit tries to address is adding a STB_LOCAL symbol to the symbol table in light of the ELF spec's requirement that: In each symbol table, all symbols with STB_LOCAL binding preced the weak and global symbols. As ``Sections'' above describes, a symbol table section's sh_info section header member holds the symbol table index for the first non-local symbol. The approach taken is to find this first non-local symbol, move that to the end and then re-use the freed spot to insert a new local symbol and increment sh_info. Except it never considered the case of object files without global symbols and got a whole bunch of details wrong -- so many in fact that it is a wonder it ever worked :/ Specifically: - It failed to re-hash the symbol on the new index, so a subsequent find_symbol_by_index() would not find it at the new location and a query for the old location would now return a non-deterministic choice between the old and new symbol. - It failed to appreciate that the GElf wrappers are not a valid disk format (it works because GElf is basically Elf64 and we only support x86_64 atm.) - It failed to fully appreciate how horrible the libelf API really is and got the gelf_update_symshndx() call pretty much completely wrong; with the direct consequence that if inserting a second STB_LOCAL symbol would require moving the same STB_GLOBAL symbol again it would completely come unstuck. Write a new elf_update_symbol() function that wraps all the magic required to update or create a new symbol at a given index. Specifically, gelf_update_sym*() require an @ndx argument that is relative to the @data argument; this means you have to manually iterate the section data descriptor list and update @ndx. Fixes: 4abff6d48dbc ("objtool: Fix code relocs vs weak symbols") Reported-by: Nathan Chancellor Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Acked-by: Josh Poimboeuf Tested-by: Nathan Chancellor Cc: Link: https://lkml.kernel.org/r/YoPCTEYjoPqE4ZxB@hirez.programming.kicks-as= s.net Signed-off-by: Greg Kroah-Hartman [bwh: Backported to 5.10: elf_hash_add() takes a hash table pointer, not just a name] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/elf.c | 196 +++++++++++++++++++++++++++++++++--------------= ----- 1 file changed, 128 insertions(+), 68 deletions(-) --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -346,6 +346,8 @@ static void elf_add_symbol(struct elf *e struct list_head *entry; struct rb_node *pnode; =20 + sym->alias =3D sym; + sym->type =3D GELF_ST_TYPE(sym->sym.st_info); sym->bind =3D GELF_ST_BIND(sym->sym.st_info); =20 @@ -401,7 +403,6 @@ static int read_symbols(struct elf *elf) return -1; } memset(sym, 0, sizeof(*sym)); - sym->alias =3D sym; =20 sym->idx =3D i; =20 @@ -562,24 +563,21 @@ static void elf_dirty_reloc_sym(struct e } =20 /* - * Move the first global symbol, as per sh_info, into a new, higher symbol - * index. This fees up the shndx for a new local symbol. + * The libelf API is terrible; gelf_update_sym*() takes a data block relat= ive + * index value, *NOT* the symbol index. As such, iterate the data blocks a= nd + * adjust index until it fits. + * + * If no data block is found, allow adding a new data block provided the i= ndex + * is only one past the end. */ -static int elf_move_global_symbol(struct elf *elf, struct section *symtab, - struct section *symtab_shndx) +static int elf_update_symbol(struct elf *elf, struct section *symtab, + struct section *symtab_shndx, struct symbol *sym) { - Elf_Data *data, *shndx_data =3D NULL; - Elf32_Word first_non_local; - struct symbol *sym; - Elf_Scn *s; - - first_non_local =3D symtab->sh.sh_info; - - sym =3D find_symbol_by_index(elf, first_non_local); - if (!sym) { - WARN("no non-local symbols !?"); - return first_non_local; - } + Elf32_Word shndx =3D sym->sec ? sym->sec->idx : SHN_UNDEF; + Elf_Data *symtab_data =3D NULL, *shndx_data =3D NULL; + Elf64_Xword entsize =3D symtab->sh.sh_entsize; + int max_idx, idx =3D sym->idx; + Elf_Scn *s, *t =3D NULL; =20 s =3D elf_getscn(elf->elf, symtab->idx); if (!s) { @@ -587,79 +585,124 @@ static int elf_move_global_symbol(struct return -1; } =20 - data =3D elf_newdata(s); - if (!data) { - WARN_ELF("elf_newdata"); - return -1; + if (symtab_shndx) { + t =3D elf_getscn(elf->elf, symtab_shndx->idx); + if (!t) { + WARN_ELF("elf_getscn"); + return -1; + } } =20 - data->d_buf =3D &sym->sym; - data->d_size =3D sizeof(sym->sym); - data->d_align =3D 1; - data->d_type =3D ELF_T_SYM; + for (;;) { + /* get next data descriptor for the relevant sections */ + symtab_data =3D elf_getdata(s, symtab_data); + if (t) + shndx_data =3D elf_getdata(t, shndx_data); + + /* end-of-list */ + if (!symtab_data) { + void *buf; + + if (idx) { + /* we don't do holes in symbol tables */ + WARN("index out of range"); + return -1; + } =20 - sym->idx =3D symtab->sh.sh_size / sizeof(sym->sym); - elf_dirty_reloc_sym(elf, sym); + /* if @idx =3D=3D 0, it's the next contiguous entry, create it */ + symtab_data =3D elf_newdata(s); + if (t) + shndx_data =3D elf_newdata(t); + + buf =3D calloc(1, entsize); + if (!buf) { + WARN("malloc"); + return -1; + } =20 - symtab->sh.sh_info +=3D 1; - symtab->sh.sh_size +=3D data->d_size; - symtab->changed =3D true; + symtab_data->d_buf =3D buf; + symtab_data->d_size =3D entsize; + symtab_data->d_align =3D 1; + symtab_data->d_type =3D ELF_T_SYM; + + symtab->sh.sh_size +=3D entsize; + symtab->changed =3D true; + + if (t) { + shndx_data->d_buf =3D &sym->sec->idx; + shndx_data->d_size =3D sizeof(Elf32_Word); + shndx_data->d_align =3D sizeof(Elf32_Word); + shndx_data->d_type =3D ELF_T_WORD; =20 - if (symtab_shndx) { - s =3D elf_getscn(elf->elf, symtab_shndx->idx); - if (!s) { - WARN_ELF("elf_getscn"); + symtab_shndx->sh.sh_size +=3D sizeof(Elf32_Word); + symtab_shndx->changed =3D true; + } + + break; + } + + /* empty blocks should not happen */ + if (!symtab_data->d_size) { + WARN("zero size data"); return -1; } =20 - shndx_data =3D elf_newdata(s); + /* is this the right block? */ + max_idx =3D symtab_data->d_size / entsize; + if (idx < max_idx) + break; + + /* adjust index and try again */ + idx -=3D max_idx; + } + + /* something went side-ways */ + if (idx < 0) { + WARN("negative index"); + return -1; + } + + /* setup extended section index magic and write the symbol */ + if (shndx >=3D SHN_UNDEF && shndx < SHN_LORESERVE) { + sym->sym.st_shndx =3D shndx; + if (!shndx_data) + shndx =3D 0; + } else { + sym->sym.st_shndx =3D SHN_XINDEX; if (!shndx_data) { - WARN_ELF("elf_newshndx_data"); + WARN("no .symtab_shndx"); return -1; } + } =20 - shndx_data->d_buf =3D &sym->sec->idx; - shndx_data->d_size =3D sizeof(Elf32_Word); - shndx_data->d_align =3D 4; - shndx_data->d_type =3D ELF_T_WORD; - - symtab_shndx->sh.sh_size +=3D 4; - symtab_shndx->changed =3D true; + if (!gelf_update_symshndx(symtab_data, shndx_data, idx, &sym->sym, shndx)= ) { + WARN_ELF("gelf_update_symshndx"); + return -1; } =20 - return first_non_local; + return 0; } =20 static struct symbol * elf_create_section_symbol(struct elf *elf, struct section *sec) { struct section *symtab, *symtab_shndx; - Elf_Data *shndx_data =3D NULL; - struct symbol *sym; - Elf32_Word shndx; + Elf32_Word first_non_local, new_idx; + struct symbol *sym, *old; =20 symtab =3D find_section_by_name(elf, ".symtab"); if (symtab) { symtab_shndx =3D find_section_by_name(elf, ".symtab_shndx"); - if (symtab_shndx) - shndx_data =3D symtab_shndx->data; } else { WARN("no .symtab"); return NULL; } =20 - sym =3D malloc(sizeof(*sym)); + sym =3D calloc(1, sizeof(*sym)); if (!sym) { perror("malloc"); return NULL; } - memset(sym, 0, sizeof(*sym)); - - sym->idx =3D elf_move_global_symbol(elf, symtab, symtab_shndx); - if (sym->idx < 0) { - WARN("elf_move_global_symbol"); - return NULL; - } =20 sym->name =3D sec->name; sym->sec =3D sec; @@ -669,24 +712,41 @@ elf_create_section_symbol(struct elf *el // st_other 0 // st_value 0 // st_size 0 - shndx =3D sec->idx; - if (shndx >=3D SHN_UNDEF && shndx < SHN_LORESERVE) { - sym->sym.st_shndx =3D shndx; - if (!shndx_data) - shndx =3D 0; - } else { - sym->sym.st_shndx =3D SHN_XINDEX; - if (!shndx_data) { - WARN("no .symtab_shndx"); + + /* + * Move the first global symbol, as per sh_info, into a new, higher + * symbol index. This fees up a spot for a new local symbol. + */ + first_non_local =3D symtab->sh.sh_info; + new_idx =3D symtab->sh.sh_size / symtab->sh.sh_entsize; + old =3D find_symbol_by_index(elf, first_non_local); + if (old) { + old->idx =3D new_idx; + + hlist_del(&old->hash); + elf_hash_add(elf->symbol_hash, &old->hash, old->idx); + + elf_dirty_reloc_sym(elf, old); + + if (elf_update_symbol(elf, symtab, symtab_shndx, old)) { + WARN("elf_update_symbol move"); return NULL; } + + new_idx =3D first_non_local; } =20 - if (!gelf_update_symshndx(symtab->data, shndx_data, sym->idx, &sym->sym, = shndx)) { - WARN_ELF("gelf_update_symshndx"); + sym->idx =3D new_idx; + if (elf_update_symbol(elf, symtab, symtab_shndx, sym)) { + WARN("elf_update_symbol"); return NULL; } =20 + /* + * Either way, we added a LOCAL symbol. + */ + symtab->sh.sh_info +=3D 1; + elf_add_symbol(elf, sym); =20 return sym; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA835C43334 for ; Sat, 23 Jul 2022 10:05:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238354AbiGWKFn (ORCPT ); Sat, 23 Jul 2022 06:05:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238222AbiGWKEn (ORCPT ); Sat, 23 Jul 2022 06:04:43 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37F83165A9; Sat, 23 Jul 2022 03:00:16 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C20F7B82B92; Sat, 23 Jul 2022 10:00:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 36FDDC341C0; Sat, 23 Jul 2022 10:00:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570404; bh=LqAXempH+uBg7nKHHZs2ygdeOzpx6l017Z5ZBhQO7y4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=zxfntsBvAfJo8Or85f+KyghrZSFIZRM3lOGrfQLM65TkjpaWuYq9u/JVVgaAaS4iO aIMP0EKK2p1mfQwdtBVXTw4+QAuu1SD2qCgc2iYYEwEOQKZarB1+EjgPBEUwbUIcOw a6Cebh2embH0BEtpzpkCU43NuWkGUQt3dMqib3C0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 075/148] x86/entry: Remove skip_r11rcx Date: Sat, 23 Jul 2022 11:54:47 +0200 Message-Id: <20220723095245.198907767@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 1b331eeea7b8676fc5dbdf80d0a07e41be226177 upstream. Yes, r11 and rcx have been restored previously, but since they're being popped anyway (into rsi) might as well pop them into their own regs -- setting them to the value they already are. Less magical code. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/20220506121631.365070674@infradead.org [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/calling.h | 10 +--------- arch/x86/entry/entry_64.S | 3 +-- 2 files changed, 2 insertions(+), 11 deletions(-) --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -146,27 +146,19 @@ For 32-bit we have the following convent =20 .endm =20 -.macro POP_REGS pop_rdi=3D1 skip_r11rcx=3D0 +.macro POP_REGS pop_rdi=3D1 popq %r15 popq %r14 popq %r13 popq %r12 popq %rbp popq %rbx - .if \skip_r11rcx - popq %rsi - .else popq %r11 - .endif popq %r10 popq %r9 popq %r8 popq %rax - .if \skip_r11rcx - popq %rsi - .else popq %rcx - .endif popq %rdx popq %rsi .if \pop_rdi --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -191,8 +191,7 @@ SYM_INNER_LABEL(entry_SYSCALL_64_after_h * perf profiles. Nothing jumps here. */ syscall_return_via_sysret: - /* rcx and r11 are already restored (see code above) */ - POP_REGS pop_rdi=3D0 skip_r11rcx=3D1 + POP_REGS pop_rdi=3D0 =20 /* * Now all regs are restored except RSP and RDI. From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E43C5C433EF for ; Sat, 23 Jul 2022 10:05:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237888AbiGWKF0 (ORCPT ); Sat, 23 Jul 2022 06:05:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238612AbiGWKEe (ORCPT ); Sat, 23 Jul 2022 06:04:34 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01A97DF94; Sat, 23 Jul 2022 03:00:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1DAF7611BD; Sat, 23 Jul 2022 10:00:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 27199C341C0; Sat, 23 Jul 2022 10:00:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570407; bh=djZlPbfcpfDt9G8q/aQbamH/xyggUYAi64L1vsv26VY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=yCdsEl6yCfQsBBYnGUcq+mID8M8DKe0Nbnv0NAsMVYxSG5YrjIWUhxDzUQEzrQZyY LunxRcC0mI6brSWB4CyRSf7VozTi9ftA79llHf0aJv67nCBHbOGvXLLGZwayZJnnTP nPh/vtIlZpmH5k2Q7cDSbDsmhy2cYvuk63ww91wI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Mikulas Patocka , "Peter Zijlstra (Intel)" , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 076/148] objtool: Fix objtool regression on x32 systems Date: Sat, 23 Jul 2022 11:54:48 +0200 Message-Id: <20220723095245.465034649@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Mikulas Patocka commit 22682a07acc308ef78681572e19502ce8893c4d4 upstream. Commit c087c6e7b551 ("objtool: Fix type of reloc::addend") failed to appreciate cross building from ILP32 hosts, where 'int' =3D=3D 'long' and the issue persists. As such, use s64/int64_t/Elf64_Sxword for this field and suffer the pain that is ISO C99 printf formats for it. Fixes: c087c6e7b551 ("objtool: Fix type of reloc::addend") Signed-off-by: Mikulas Patocka [peterz: reword changelog, s/long long/s64/] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Cc: Link: https://lkml.kernel.org/r/alpine.LRH.2.02.2205161041260.11556@file01.= intranet.prod.int.rdu2.redhat.com Signed-off-by: Greg Kroah-Hartman Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 9 +++++---- tools/objtool/elf.c | 2 +- tools/objtool/elf.h | 4 ++-- 3 files changed, 8 insertions(+), 7 deletions(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -5,6 +5,7 @@ =20 #include #include +#include #include =20 #include "builtin.h" @@ -467,12 +468,12 @@ static int add_dead_ends(struct objtool_ else if (reloc->addend =3D=3D reloc->sym->sec->len) { insn =3D find_last_insn(file, reloc->sym->sec); if (!insn) { - WARN("can't find unreachable insn at %s+0x%lx", + WARN("can't find unreachable insn at %s+0x%" PRIx64, reloc->sym->sec->name, reloc->addend); return -1; } } else { - WARN("can't find unreachable insn at %s+0x%lx", + WARN("can't find unreachable insn at %s+0x%" PRIx64, reloc->sym->sec->name, reloc->addend); return -1; } @@ -502,12 +503,12 @@ reachable: else if (reloc->addend =3D=3D reloc->sym->sec->len) { insn =3D find_last_insn(file, reloc->sym->sec); if (!insn) { - WARN("can't find reachable insn at %s+0x%lx", + WARN("can't find reachable insn at %s+0x%" PRIx64, reloc->sym->sec->name, reloc->addend); return -1; } } else { - WARN("can't find reachable insn at %s+0x%lx", + WARN("can't find reachable insn at %s+0x%" PRIx64, reloc->sym->sec->name, reloc->addend); return -1; } --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -510,7 +510,7 @@ static struct section *elf_create_reloc_ int reltype); =20 int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, - unsigned int type, struct symbol *sym, long addend) + unsigned int type, struct symbol *sym, s64 addend) { struct reloc *reloc; =20 --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -73,7 +73,7 @@ struct reloc { struct symbol *sym; unsigned long offset; unsigned int type; - long addend; + s64 addend; int idx; bool jump_table_start; }; @@ -127,7 +127,7 @@ struct elf *elf_open_read(const char *na struct section *elf_create_section(struct elf *elf, const char *name, unsi= gned int sh_flags, size_t entsize, int nr); =20 int elf_add_reloc(struct elf *elf, struct section *sec, unsigned long offs= et, - unsigned int type, struct symbol *sym, long addend); + unsigned int type, struct symbol *sym, s64 addend); int elf_add_reloc_to_insn(struct elf *elf, struct section *sec, unsigned long offset, unsigned int type, struct section *insn_sec, unsigned long insn_off); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC3AECCA483 for ; Sat, 23 Jul 2022 10:05:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238498AbiGWKFr (ORCPT ); Sat, 23 Jul 2022 06:05:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238242AbiGWKEo (ORCPT ); Sat, 23 Jul 2022 06:04:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4631E18E0C; Sat, 23 Jul 2022 03:00:18 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0509B611CD; Sat, 23 Jul 2022 10:00:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0FB78C341C0; Sat, 23 Jul 2022 10:00:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570410; bh=vg9lEJu/LLQWjpmK0UYvHuBX4PCo2gXqubrhZY/0Pjs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KdqwM1RloZyOm4Eq9fuSlLXrgPdWz8mN6mbG1FfIDGa5rYyJhquqzSN1L6x0NvGU1 I6s7IgRw7xN+BZfFdFg6HVhGMAd307jXupqjLBbnUOw5s4hW99E9T6BUbP9GMNs6pb 4wG9AoGcMLoqoWZfB5Y8KFpUhCHLeTCrzqpUkyRE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 077/148] x86/realmode: build with -D__DISABLE_EXPORTS Date: Sat, 23 Jul 2022 11:54:49 +0200 Message-Id: <20220723095245.734927578@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thadeu Lima de Souza Cascardo Commit 156ff4a544ae ("x86/ibt: Base IBT bits") added this option when building realmode in order to disable IBT there. This is also needed in order to disable return thunks. Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -31,7 +31,7 @@ endif CODE16GCC_CFLAGS :=3D -m32 -Wa,$(srctree)/arch/x86/boot/code16gcc.h M16_CFLAGS :=3D $(call cc-option, -m16, $(CODE16GCC_CFLAGS)) =20 -REALMODE_CFLAGS :=3D $(M16_CFLAGS) -g -Os -DDISABLE_BRANCH_PROFILING \ +REALMODE_CFLAGS :=3D $(M16_CFLAGS) -g -Os -DDISABLE_BRANCH_PROFILING -D__D= ISABLE_EXPORTS \ -Wall -Wstrict-prototypes -march=3Di386 -mregparm=3D3 \ -fno-strict-aliasing -fomit-frame-pointer -fno-pic \ -mno-mmx -mno-sse $(call cc-option,-fcf-protection=3Dnone) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F31A8CCA483 for ; Sat, 23 Jul 2022 10:05:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238485AbiGWKFl (ORCPT ); Sat, 23 Jul 2022 06:05:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238195AbiGWKEl (ORCPT ); Sat, 23 Jul 2022 06:04:41 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BDC314D00; Sat, 23 Jul 2022 03:00:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A0D4CB82C1A; Sat, 23 Jul 2022 10:00:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E7899C341C0; Sat, 23 Jul 2022 10:00:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570413; bh=arvzEwfuEM6okV3AKGV59B3YaE0j1+/tS4TEz+YQBDc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kOT/1AoqmW3d5wgJ+FbC1ii9JK5hDjLu+ArkKOvU66PbAV5U2GX+ZIt4A8I39NnnJ eSbl/oD111LdrWK7keRUq7QVDApzEpJzAC+iLeomJPx9gqRCUmU9dM3cHUP6DbTUe3 92W96uZN+pnnNxfQo3ZkAlcyKEIgXxbx1gcsBF5o= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 078/148] x86/kvm/vmx: Make noinstr clean Date: Sat, 23 Jul 2022 11:54:50 +0200 Message-Id: <20220723095246.085386556@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 742ab6df974ae8384a2dd213db1a3a06cf6d8936 upstream. The recent mmio_stale_data fixes broke the noinstr constraints: vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x15b: call to wrmsrl.co= nstprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x1bf: call to kvm_arch_= has_assigned_device() leaves .noinstr.text section make it all happy again. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/vmx/vmx.c | 6 +++--- arch/x86/kvm/x86.c | 4 ++-- include/linux/kvm_host.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -380,9 +380,9 @@ static __always_inline void vmx_disable_ if (!vmx->disable_fb_clear) return; =20 - rdmsrl(MSR_IA32_MCU_OPT_CTRL, msr); + msr =3D __rdmsr(MSR_IA32_MCU_OPT_CTRL); msr |=3D FB_CLEAR_DIS; - wrmsrl(MSR_IA32_MCU_OPT_CTRL, msr); + native_wrmsrl(MSR_IA32_MCU_OPT_CTRL, msr); /* Cache the MSR value to avoid reading it later */ vmx->msr_ia32_mcu_opt_ctrl =3D msr; } @@ -393,7 +393,7 @@ static __always_inline void vmx_enable_f return; =20 vmx->msr_ia32_mcu_opt_ctrl &=3D ~FB_CLEAR_DIS; - wrmsrl(MSR_IA32_MCU_OPT_CTRL, vmx->msr_ia32_mcu_opt_ctrl); + native_wrmsrl(MSR_IA32_MCU_OPT_CTRL, vmx->msr_ia32_mcu_opt_ctrl); } =20 static void vmx_update_fb_clear_dis(struct kvm_vcpu *vcpu, struct vcpu_vmx= *vmx) --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11173,9 +11173,9 @@ void kvm_arch_end_assignment(struct kvm } EXPORT_SYMBOL_GPL(kvm_arch_end_assignment); =20 -bool kvm_arch_has_assigned_device(struct kvm *kvm) +bool noinstr kvm_arch_has_assigned_device(struct kvm *kvm) { - return atomic_read(&kvm->arch.assigned_device_count); + return arch_atomic_read(&kvm->arch.assigned_device_count); } EXPORT_SYMBOL_GPL(kvm_arch_has_assigned_device); =20 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -988,7 +988,7 @@ static inline void kvm_arch_end_assignme { } =20 -static inline bool kvm_arch_has_assigned_device(struct kvm *kvm) +static __always_inline bool kvm_arch_has_assigned_device(struct kvm *kvm) { return false; } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CEDBC43334 for ; Sat, 23 Jul 2022 10:05:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238371AbiGWKFq (ORCPT ); Sat, 23 Jul 2022 06:05:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238219AbiGWKEn (ORCPT ); Sat, 23 Jul 2022 06:04:43 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37DAE16598; Sat, 23 Jul 2022 03:00:16 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CF89761204; Sat, 23 Jul 2022 10:00:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D9AA2C341C0; Sat, 23 Jul 2022 10:00:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570416; bh=ByLN6L1kQ5tDLJy2YF+90FMHIkenQB5ANQ1jfWxqXcU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=suiovti4hXv+tpcLUAXPTqf9EkhdXdAAZxBzRNl1uQlV/1/sV0y/j20JQz1mjbM7H yLos+NKl+wQi8JnOl1xi+TmqGCWK7bQ2bsoyEjRrmpIxBbNpe6njK1h+utseimA6s3 baxKCBvJZwfKqWC4ftM31OKpdDjhKeVKR7BfDhY8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 079/148] x86/cpufeatures: Move RETPOLINE flags to word 11 Date: Sat, 23 Jul 2022 11:54:51 +0200 Message-Id: <20220723095246.400129259@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit a883d624aed463c84c22596006e5a96f5b44db31 upstream. In order to extend the RETPOLINE features to 4, move them to word 11 where there is still room. This mostly keeps DISABLE_RETPOLINE simple. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: bits 8 and 9 of word 11 are also free here, so comment them accordingly] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/cpufeatures.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -203,8 +203,8 @@ #define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9) /* AMD ProcFeedbackInterface = */ #define X86_FEATURE_SME ( 7*32+10) /* AMD Secure Memory Encryption */ #define X86_FEATURE_PTI ( 7*32+11) /* Kernel Page Table Isolation enable= d */ -#define X86_FEATURE_RETPOLINE ( 7*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ -#define X86_FEATURE_RETPOLINE_LFENCE ( 7*32+13) /* "" Use LFENCE for Spect= re variant 2 */ +/* FREE! ( 7*32+12) */ +/* FREE! ( 7*32+13) */ #define X86_FEATURE_INTEL_PPIN ( 7*32+14) /* Intel Processor Inventory Nu= mber */ #define X86_FEATURE_CDP_L2 ( 7*32+15) /* Code and Data Prioritization L2 = */ #define X86_FEATURE_MSR_SPEC_CTRL ( 7*32+16) /* "" MSR SPEC_CTRL is implem= ented */ @@ -290,6 +290,12 @@ #define X86_FEATURE_FENCE_SWAPGS_KERNEL (11*32+ 5) /* "" LFENCE in kernel = entry SWAPGS path */ #define X86_FEATURE_SPLIT_LOCK_DETECT (11*32+ 6) /* #AC for split lock */ #define X86_FEATURE_PER_THREAD_MBA (11*32+ 7) /* "" Per-thread Memory Band= width Allocation */ +/* FREE! (11*32+ 8) */ +/* FREE! (11*32+ 9) */ +/* FREE! (11*32+10) */ +/* FREE! (11*32+11) */ +#define X86_FEATURE_RETPOLINE (11*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ +#define X86_FEATURE_RETPOLINE_LFENCE (11*32+13) /* "" Use LFENCE for Spect= re variant 2 */ =20 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instruction= s */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F2DC433EF for ; Sat, 23 Jul 2022 10:05:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238507AbiGWKFt (ORCPT ); Sat, 23 Jul 2022 06:05:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238169AbiGWKFE (ORCPT ); Sat, 23 Jul 2022 06:05:04 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F7291DA4E; Sat, 23 Jul 2022 03:00:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AEDEA61263; Sat, 23 Jul 2022 10:00:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BA7E5C341C7; Sat, 23 Jul 2022 10:00:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570419; bh=xUZao+YZQ5hWJQmX+X+GqLQnsVGFxF9clzCVR/E8YC4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dB/QD8HZLlg1CK9kU8tXH2vMPbu1vk5zZen/mqCH1CAknxMFHAee6UCw74TYLVisL DHlOoQTguGYYgoqKHTg0y0v+dY/hza4YNxAfpbCfbbpejLoM7Sgsa+dQgtTwc6H9EL kX8UIHKPmGzai9Yvo1hh1JQrBtexz2rCzPQF/W4Y= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 080/148] x86/retpoline: Cleanup some #ifdefery Date: Sat, 23 Jul 2022 11:54:52 +0200 Message-Id: <20220723095246.670920943@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 369ae6ffc41a3c1137cab697635a84d0cc7cdcea upstream. On it's own not much of a cleanup but it prepares for more/similar code. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: conflict fixup because of DISABLE_ENQCMD] [cascardo: no changes at nospec-branch.h and bpf_jit_comp.c] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/disabled-features.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -56,6 +56,13 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif =20 +#ifdef CONFIG_RETPOLINE +# define DISABLE_RETPOLINE 0 +#else +# define DISABLE_RETPOLINE ((1 << (X86_FEATURE_RETPOLINE & 31)) | \ + (1 << (X86_FEATURE_RETPOLINE_LFENCE & 31))) +#endif + /* Force disable because it's broken beyond repair */ #define DISABLE_ENQCMD (1 << (X86_FEATURE_ENQCMD & 31)) =20 @@ -73,7 +80,7 @@ #define DISABLED_MASK8 0 #define DISABLED_MASK9 (DISABLE_SMAP) #define DISABLED_MASK10 0 -#define DISABLED_MASK11 0 +#define DISABLED_MASK11 (DISABLE_RETPOLINE) #define DISABLED_MASK12 0 #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AA8DC433EF for ; Sat, 23 Jul 2022 10:06:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238542AbiGWKGH (ORCPT ); Sat, 23 Jul 2022 06:06:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238481AbiGWKFl (ORCPT ); Sat, 23 Jul 2022 06:05:41 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEBF09F07D; Sat, 23 Jul 2022 03:00:29 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6C56661212; Sat, 23 Jul 2022 10:00:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7D25FC341C0; Sat, 23 Jul 2022 10:00:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570421; bh=U/+eLCYI4YDKjT6BcIXjKCgdRxC0oXdMaxjk/f6TANk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PeXvjQjvnc09Lgaf0k/BhTv7lTmXLbVXqrytOCFSiUvP4n+US/HJkNqyoUMLkyYjy R6U7os+ZfnycC6WMtVM+K8vKGc1DaMSgxSi+MKzKQRD5NVW6K7aACLRwsHRbEZPFFQ AGDzm+Rxj92g8RjhLm60Cb8ojAxxuJomiE6/FcZE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 081/148] x86/retpoline: Swizzle retpoline thunk Date: Sat, 23 Jul 2022 11:54:53 +0200 Message-Id: <20220723095247.034926462@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 00e1533325fd1fb5459229fe37f235462649f668 upstream. Put the actual retpoline thunk as the original code so that it can become more complicated. Specifically, it allows RET to be a JMP, which can't be .altinstr_replacement since that doesn't do relocations (except for the very first instruction). Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/lib/retpoline.S | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -32,9 +32,9 @@ SYM_INNER_LABEL(__x86_indirect_thunk_\reg, SYM_L_GLOBAL) UNWIND_HINT_EMPTY =20 - ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ - __stringify(RETPOLINE \reg), X86_FEATURE_RETPOLINE, \ - __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg; int3), X8= 6_FEATURE_RETPOLINE_LFENCE + ALTERNATIVE_2 __stringify(RETPOLINE \reg), \ + __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg; int3), X8= 6_FEATURE_RETPOLINE_LFENCE, \ + __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), ALT_NOT(X86_FEAT= URE_RETPOLINE) =20 .endm From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A38CCCA483 for ; Sat, 23 Jul 2022 10:05:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238051AbiGWKF6 (ORCPT ); Sat, 23 Jul 2022 06:05:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238424AbiGWKFY (ORCPT ); Sat, 23 Jul 2022 06:05:24 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92B9DF54; Sat, 23 Jul 2022 03:00:25 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 45B4361277; Sat, 23 Jul 2022 10:00:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 535F3C341C0; Sat, 23 Jul 2022 10:00:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570424; bh=NDj9XORo/OeKYd/gw/mw/rUhZ8m5yMraNTBvIwn97J8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hshp7uoDoAvtbQZXVy6eyy/D+FRgdPmDqbolii/RQbF5/pyv3n/u6h/P4rFbCrnRA 5uqwv5/oVKHMzoFmzjwjKL7OeHKKSrHqoFiJzfQYWjyjrryRcwOuezFkC4OlPnxnXz H0mnN87elrS4MSLtGT1JT+xVFPAzjgMZt7/zqe7Q= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Ben Hutchings Subject: [PATCH 5.10 082/148] Makefile: Set retpoline cflags based on CONFIG_CC_IS_{CLANG,GCC} Date: Sat, 23 Jul 2022 11:54:54 +0200 Message-Id: <20220723095247.327282663@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Ben Hutchings This was done as part of commit 7d73c3e9c51400d3e0e755488050804e4d44737a "Makefile: remove stale cc-option checks" upstream, and is needed to support backporting further retpoline changes. Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Makefile | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) --- a/Makefile +++ b/Makefile @@ -670,12 +670,14 @@ ifdef CONFIG_FUNCTION_TRACER CC_FLAGS_FTRACE :=3D -pg endif =20 -RETPOLINE_CFLAGS_GCC :=3D -mindirect-branch=3Dthunk-extern -mindirect-bran= ch-register -RETPOLINE_VDSO_CFLAGS_GCC :=3D -mindirect-branch=3Dthunk-inline -mindirect= -branch-register -RETPOLINE_CFLAGS_CLANG :=3D -mretpoline-external-thunk -RETPOLINE_VDSO_CFLAGS_CLANG :=3D -mretpoline -RETPOLINE_CFLAGS :=3D $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-o= ption,$(RETPOLINE_CFLAGS_CLANG))) -RETPOLINE_VDSO_CFLAGS :=3D $(call cc-option,$(RETPOLINE_VDSO_CFLAGS_GCC),$= (call cc-option,$(RETPOLINE_VDSO_CFLAGS_CLANG))) +ifdef CONFIG_CC_IS_GCC +RETPOLINE_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-extern -m= indirect-branch-register) +RETPOLINE_VDSO_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-inli= ne -mindirect-branch-register) +endif +ifdef CONFIG_CC_IS_CLANG +RETPOLINE_CFLAGS :=3D -mretpoline-external-thunk +RETPOLINE_VDSO_CFLAGS :=3D -mretpoline +endif export RETPOLINE_CFLAGS export RETPOLINE_VDSO_CFLAGS From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9921FC43334 for ; Sat, 23 Jul 2022 10:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238318AbiGWKFH (ORCPT ); Sat, 23 Jul 2022 06:05:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238560AbiGWKEa (ORCPT ); Sat, 23 Jul 2022 06:04:30 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22E24DEC; Sat, 23 Jul 2022 03:00:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 610DBCE0DBE; Sat, 23 Jul 2022 10:00:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E017C341CB; Sat, 23 Jul 2022 09:59:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570398; bh=CiilJ/57YWWPXmSTZIsCpZokOcdA6EH6fCCm2gWX1Fo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=d9BZ3yG5f0dRAkTL6kIViTzITNav4o3V5z2KDLxQIGqMI7UfeGta2lNV0NTZzfh3w I+H0yJaFRUTB7ELO4BK9QdzTfSK6dQrX2CY1eWr64+PlmCepfbeVj/WaRTMi6Xa87I v8kEj+ta75Zewy0sGS/7ppz1XAk48J15ExjpnHlE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Nick Desaulniers , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 083/148] x86/retpoline: Use -mfunction-return Date: Sat, 23 Jul 2022 11:54:55 +0200 Message-Id: <20220723095247.615053520@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 0b53c374b9eff2255a386f1f1cfb9a928e52a5ae upstream. Utilize -mfunction-return=3Dthunk-extern when available to have the compiler replace RET instructions with direct JMPs to the symbol __x86_return_thunk. This does not affect assembler (.S) sources, only C sources. -mfunction-return=3Dthunk-extern has been available since gcc 7.3 and clang 15. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Nick Desaulniers Reviewed-by: Josh Poimboeuf Tested-by: Nick Desaulniers Signed-off-by: Borislav Petkov [cascardo: RETPOLINE_CFLAGS is at Makefile] [cascardo: remove ANNOTATE_NOENDBR from __x86_return_thunk] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Makefile | 2 ++ arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/lib/retpoline.S | 12 ++++++++++++ 3 files changed, 16 insertions(+) --- a/Makefile +++ b/Makefile @@ -672,11 +672,13 @@ endif =20 ifdef CONFIG_CC_IS_GCC RETPOLINE_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-extern -m= indirect-branch-register) +RETPOLINE_CFLAGS +=3D $(call cc-option,-mfunction-return=3Dthunk-extern) RETPOLINE_VDSO_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-inli= ne -mindirect-branch-register) endif ifdef CONFIG_CC_IS_CLANG RETPOLINE_CFLAGS :=3D -mretpoline-external-thunk RETPOLINE_VDSO_CFLAGS :=3D -mretpoline +RETPOLINE_CFLAGS +=3D $(call cc-option,-mfunction-return=3Dthunk-extern) endif export RETPOLINE_CFLAGS export RETPOLINE_VDSO_CFLAGS --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -120,6 +120,8 @@ _ASM_PTR " 999b\n\t" \ ".popsection\n\t" =20 +extern void __x86_return_thunk(void); + #ifdef CONFIG_RETPOLINE =20 typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE]; --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -66,3 +66,15 @@ SYM_CODE_END(__x86_indirect_thunk_array) #define GEN(reg) EXPORT_THUNK(reg) #include #undef GEN + +/* + * This function name is magical and is used by -mfunction-return=3Dthunk-= extern + * for the compiler to generate JMPs to it. + */ +SYM_CODE_START(__x86_return_thunk) + UNWIND_HINT_EMPTY + ret + int3 +SYM_CODE_END(__x86_return_thunk) + +__EXPORT_THUNK(__x86_return_thunk) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58CB1C433EF for ; Sat, 23 Jul 2022 10:05:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238440AbiGWKFa (ORCPT ); Sat, 23 Jul 2022 06:05:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238615AbiGWKEe (ORCPT ); Sat, 23 Jul 2022 06:04:34 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55F3511823; Sat, 23 Jul 2022 03:00:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4F0D96116A; Sat, 23 Jul 2022 10:00:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 51605C341C0; Sat, 23 Jul 2022 10:00:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570401; bh=GdgQnVw35ejBBdRQk4JC4U3Dhgfkk09O80TBVdJvkNY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L12zXGNX9ITbahJibsY4+rUe42WvsjzqYPdBJAizi+CmGRGtpJeGHMMDHCFfb+QfK QR011PxMpmlhhaXOHimKvIJeP2Si5KKiSnNmHWkb2TaI5WfMJtH2UxzC5bZdvngzwh XLsKNLgU4Mn1tsscOrfEgBYhE635BqDLgE0QP7qc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 084/148] x86: Undo return-thunk damage Date: Sat, 23 Jul 2022 11:54:56 +0200 Message-Id: <20220723095247.887292450@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 15e67227c49a57837108acfe1c80570e1bd9f962 upstream. Introduce X86_FEATURE_RETHUNK for those afflicted with needing this. [ bp: Do only INT3 padding - simpler. ] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: CONFIG_STACK_VALIDATION vs CONFIG_OBJTOOL] [cascardo: no IBT support] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/alternative.h | 1=20 arch/x86/include/asm/cpufeatures.h | 1=20 arch/x86/include/asm/disabled-features.h | 3 + arch/x86/kernel/alternative.c | 60 ++++++++++++++++++++++++++= +++++ arch/x86/kernel/module.c | 8 +++- arch/x86/kernel/vmlinux.lds.S | 7 +++ 6 files changed, 78 insertions(+), 2 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -76,6 +76,7 @@ extern int alternatives_patched; extern void alternative_instructions(void); extern void apply_alternatives(struct alt_instr *start, struct alt_instr *= end); extern void apply_retpolines(s32 *start, s32 *end); +extern void apply_returns(s32 *start, s32 *end); =20 struct module; =20 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -296,6 +296,7 @@ /* FREE! (11*32+11) */ #define X86_FEATURE_RETPOLINE (11*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ #define X86_FEATURE_RETPOLINE_LFENCE (11*32+13) /* "" Use LFENCE for Spect= re variant 2 */ +#define X86_FEATURE_RETHUNK (11*32+14) /* "" Use REturn THUNK */ =20 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instruction= s */ --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -60,7 +60,8 @@ # define DISABLE_RETPOLINE 0 #else # define DISABLE_RETPOLINE ((1 << (X86_FEATURE_RETPOLINE & 31)) | \ - (1 << (X86_FEATURE_RETPOLINE_LFENCE & 31))) + (1 << (X86_FEATURE_RETPOLINE_LFENCE & 31)) | \ + (1 << (X86_FEATURE_RETHUNK & 31))) #endif =20 /* Force disable because it's broken beyond repair */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -270,6 +270,7 @@ static void __init_or_module add_nops(vo } =20 extern s32 __retpoline_sites[], __retpoline_sites_end[]; +extern s32 __return_sites[], __return_sites_end[]; extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; void text_poke_early(void *addr, const void *opcode, size_t len); @@ -661,9 +662,67 @@ void __init_or_module noinline apply_ret } } =20 +/* + * Rewrite the compiler generated return thunk tail-calls. + * + * For example, convert: + * + * JMP __x86_return_thunk + * + * into: + * + * RET + */ +static int patch_return(void *addr, struct insn *insn, u8 *bytes) +{ + int i =3D 0; + + if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) + return -1; + + bytes[i++] =3D RET_INSN_OPCODE; + + for (; i < insn->length;) + bytes[i++] =3D INT3_INSN_OPCODE; + + return i; +} + +void __init_or_module noinline apply_returns(s32 *start, s32 *end) +{ + s32 *s; + + for (s =3D start; s < end; s++) { + void *addr =3D (void *)s + *s; + struct insn insn; + int len, ret; + u8 bytes[16]; + u8 op1; + + ret =3D insn_decode_kernel(&insn, addr); + if (WARN_ON_ONCE(ret < 0)) + continue; + + op1 =3D insn.opcode.bytes[0]; + if (WARN_ON_ONCE(op1 !=3D JMP32_INSN_OPCODE)) + continue; + + DPRINTK("return thunk at: %pS (%px) len: %d to: %pS", + addr, addr, insn.length, + addr + insn.length + insn.immediate.value); + + len =3D patch_return(addr, &insn, bytes); + if (len =3D=3D insn.length) { + DUMP_BYTES(((u8*)addr), len, "%px: orig: ", addr); + DUMP_BYTES(((u8*)bytes), len, "%px: repl: ", addr); + text_poke_early(addr, bytes, len); + } + } +} #else /* !RETPOLINES || !CONFIG_STACK_VALIDATION */ =20 void __init_or_module noinline apply_retpolines(s32 *start, s32 *end) { } +void __init_or_module noinline apply_returns(s32 *start, s32 *end) { } =20 #endif /* CONFIG_RETPOLINE && CONFIG_STACK_VALIDATION */ =20 @@ -956,6 +1015,7 @@ void __init alternative_instructions(voi * those can rewrite the retpoline thunks. */ apply_retpolines(__retpoline_sites, __retpoline_sites_end); + apply_returns(__return_sites, __return_sites_end); =20 apply_alternatives(__alt_instructions, __alt_instructions_end); =20 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -252,7 +252,7 @@ int module_finalize(const Elf_Ehdr *hdr, { const Elf_Shdr *s, *text =3D NULL, *alt =3D NULL, *locks =3D NULL, *para =3D NULL, *orc =3D NULL, *orc_ip =3D NULL, - *retpolines =3D NULL; + *retpolines =3D NULL, *returns =3D NULL; char *secstrings =3D (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; =20 for (s =3D sechdrs; s < sechdrs + hdr->e_shnum; s++) { @@ -270,12 +270,18 @@ int module_finalize(const Elf_Ehdr *hdr, orc_ip =3D s; if (!strcmp(".retpoline_sites", secstrings + s->sh_name)) retpolines =3D s; + if (!strcmp(".return_sites", secstrings + s->sh_name)) + returns =3D s; } =20 if (retpolines) { void *rseg =3D (void *)retpolines->sh_addr; apply_retpolines(rseg, rseg + retpolines->sh_size); } + if (returns) { + void *rseg =3D (void *)returns->sh_addr; + apply_returns(rseg, rseg + returns->sh_size); + } if (alt) { /* patch .altinstructions */ void *aseg =3D (void *)alt->sh_addr; --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -284,6 +284,13 @@ SECTIONS *(.retpoline_sites) __retpoline_sites_end =3D .; } + + . =3D ALIGN(8); + .return_sites : AT(ADDR(.return_sites) - LOAD_OFFSET) { + __return_sites =3D .; + *(.return_sites) + __return_sites_end =3D .; + } #endif =20 /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2421C433EF for ; Sat, 23 Jul 2022 10:07:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238338AbiGWKHS (ORCPT ); Sat, 23 Jul 2022 06:07:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238463AbiGWKGv (ORCPT ); Sat, 23 Jul 2022 06:06:51 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 080C7C1DE0; Sat, 23 Jul 2022 03:01:04 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 58CAC61204; Sat, 23 Jul 2022 10:01:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67B7CC341C0; Sat, 23 Jul 2022 10:01:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570461; bh=NbqSanLB3Qxt8nxUvYutjP8Hct7V+q/cUJf7O//Vvd4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wlu6o4MbL2vGEzSYJAdJw77FKPgPufGu1o0G7upyWUJAl+0W1+RmDtrCFNCjf0onQ Y3PtscVjuniPi3Dv4XMUqeF8/NjKBF/xFsl7x7PmCmyeLm5CFPsIf5Xl76eZKWK/uQ P4vJN4fxzp2XHKdwOVyKFKyyFiM+1lFOiuQDBzZ8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 085/148] x86,objtool: Create .return_sites Date: Sat, 23 Jul 2022 11:54:57 +0200 Message-Id: <20220723095248.217682747@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit d9e9d2300681d68a775c28de6aa6e5290ae17796 upstream. Find all the return-thunk sites and record them in a .return_sites section such that the kernel can undo this. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: conflict fixup because of functions added to support IBT] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/arch.h | 1=20 tools/objtool/arch/x86/decode.c | 5 ++ tools/objtool/check.c | 75 +++++++++++++++++++++++++++++++++++= +++++ tools/objtool/elf.h | 1=20 tools/objtool/objtool.c | 1=20 tools/objtool/objtool.h | 1=20 6 files changed, 84 insertions(+) --- a/tools/objtool/arch.h +++ b/tools/objtool/arch.h @@ -89,6 +89,7 @@ const char *arch_ret_insn(int len); int arch_decode_hint_reg(u8 sp_reg, int *base); =20 bool arch_is_retpoline(struct symbol *sym); +bool arch_is_rethunk(struct symbol *sym); =20 int arch_rewrite_retpolines(struct objtool_file *file); =20 --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -649,3 +649,8 @@ bool arch_is_retpoline(struct symbol *sy { return !strncmp(sym->name, "__x86_indirect_", 15); } + +bool arch_is_rethunk(struct symbol *sym) +{ + return !strcmp(sym->name, "__x86_return_thunk"); +} --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -653,6 +653,52 @@ static int create_retpoline_sites_sectio return 0; } =20 +static int create_return_sites_sections(struct objtool_file *file) +{ + struct instruction *insn; + struct section *sec; + int idx; + + sec =3D find_section_by_name(file->elf, ".return_sites"); + if (sec) { + WARN("file already has .return_sites, skipping"); + return 0; + } + + idx =3D 0; + list_for_each_entry(insn, &file->return_thunk_list, call_node) + idx++; + + if (!idx) + return 0; + + sec =3D elf_create_section(file->elf, ".return_sites", 0, + sizeof(int), idx); + if (!sec) { + WARN("elf_create_section: .return_sites"); + return -1; + } + + idx =3D 0; + list_for_each_entry(insn, &file->return_thunk_list, call_node) { + + int *site =3D (int *)sec->data->d_buf + idx; + *site =3D 0; + + if (elf_add_reloc_to_insn(file->elf, sec, + idx * sizeof(int), + R_X86_64_PC32, + insn->sec, insn->offset)) { + WARN("elf_add_reloc_to_insn: .return_sites"); + return -1; + } + + idx++; + } + + return 0; +} + /* * Warnings shouldn't be reported for ignored functions. */ @@ -888,6 +934,11 @@ __weak bool arch_is_retpoline(struct sym return false; } =20 +__weak bool arch_is_rethunk(struct symbol *sym) +{ + return false; +} + #define NEGATIVE_RELOC ((void *)-1L) =20 static struct reloc *insn_reloc(struct objtool_file *file, struct instruct= ion *insn) @@ -1029,6 +1080,19 @@ static void add_retpoline_call(struct ob =20 annotate_call_site(file, insn, false); } + +static void add_return_call(struct objtool_file *file, struct instruction = *insn) +{ + /* + * Return thunk tail calls are really just returns in disguise, + * so convert them accordingly. + */ + insn->type =3D INSN_RETURN; + insn->retpoline_safe =3D true; + + list_add_tail(&insn->call_node, &file->return_thunk_list); +} + /* * Find the destination instructions for all jumps. */ @@ -1053,6 +1117,9 @@ static int add_jump_destinations(struct } else if (reloc->sym->retpoline_thunk) { add_retpoline_call(file, insn); continue; + } else if (reloc->sym->return_thunk) { + add_return_call(file, insn); + continue; } else if (insn->func) { /* internal or external sibling call (with reloc) */ add_call_dest(file, insn, reloc->sym, true); @@ -1842,6 +1909,9 @@ static int classify_symbols(struct objto if (arch_is_retpoline(func)) func->retpoline_thunk =3D true; =20 + if (arch_is_rethunk(func)) + func->return_thunk =3D true; + if (!strcmp(func->name, "__fentry__")) func->fentry =3D true; =20 @@ -3235,6 +3305,11 @@ int check(struct objtool_file *file) if (ret < 0) goto out; warnings +=3D ret; + + ret =3D create_return_sites_sections(file); + if (ret < 0) + goto out; + warnings +=3D ret; } =20 if (stats) { --- a/tools/objtool/elf.h +++ b/tools/objtool/elf.h @@ -58,6 +58,7 @@ struct symbol { u8 uaccess_safe : 1; u8 static_call_tramp : 1; u8 retpoline_thunk : 1; + u8 return_thunk : 1; u8 fentry : 1; u8 kcov : 1; }; --- a/tools/objtool/objtool.c +++ b/tools/objtool/objtool.c @@ -62,6 +62,7 @@ struct objtool_file *objtool_open_read(c INIT_LIST_HEAD(&file.insn_list); hash_init(file.insn_hash); INIT_LIST_HEAD(&file.retpoline_call_list); + INIT_LIST_HEAD(&file.return_thunk_list); INIT_LIST_HEAD(&file.static_call_list); file.c_file =3D !vmlinux && find_section_by_name(file.elf, ".comment"); file.ignore_unreachables =3D no_unreachable; --- a/tools/objtool/objtool.h +++ b/tools/objtool/objtool.h @@ -19,6 +19,7 @@ struct objtool_file { struct list_head insn_list; DECLARE_HASHTABLE(insn_hash, 20); struct list_head retpoline_call_list; + struct list_head return_thunk_list; struct list_head static_call_list; bool ignore_unreachables, c_file, hints, rodata; }; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14F0EC43334 for ; Sat, 23 Jul 2022 10:06:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238567AbiGWKGP (ORCPT ); Sat, 23 Jul 2022 06:06:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238410AbiGWKFz (ORCPT ); Sat, 23 Jul 2022 06:05:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC2EBA0273; Sat, 23 Jul 2022 03:00:31 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 07FE4611D4; Sat, 23 Jul 2022 10:00:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0CA36C341C7; Sat, 23 Jul 2022 10:00:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570430; bh=0A1ecswK9RXwOpzgxQteXqxtREg2wIgo60mGWUG+KJs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PbrzQCquYevCfV4MN2H93gXuVrji3QjFfqsGj2tOnZu0+SxoHZz+JeZRmDU34zO8P /EnOyv7jGVSeRoDymp+4OTI9CPoQjxlKJbs/p4nsnqGhAnHWO4p1aUq/dI2CLTDNvI wL9jmyhOr1oHwB5SEweXM3MXf2Z800NXnN9xYq0k= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Thadeu Lima de Souza Cascardo , Josh Poimboeuf , Ben Hutchings Subject: [PATCH 5.10 086/148] objtool: skip non-text sections when adding return-thunk sites Date: Sat, 23 Jul 2022 11:54:58 +0200 Message-Id: <20220723095248.478393950@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thadeu Lima de Souza Cascardo The .discard.text section is added in order to reserve BRK, with a temporary function just so it can give it a size. This adds a relocation to the return thunk, which objtool will add to the .return_sites section. Linking will then fail as there are references to the .discard.text section. Do not add instructions from non-text sections to the list of return thunk calls, avoiding the reference to .discard.text. Signed-off-by: Thadeu Lima de Souza Cascardo Acked-by: Josh Poimboeuf Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1090,7 +1090,9 @@ static void add_return_call(struct objto insn->type =3D INSN_RETURN; insn->retpoline_safe =3D true; =20 - list_add_tail(&insn->call_node, &file->return_thunk_list); + /* Skip the non-text sections, specially .discard ones */ + if (insn->sec->text) + list_add_tail(&insn->call_node, &file->return_thunk_list); } =20 /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECBF6CCA483 for ; Sat, 23 Jul 2022 10:07:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238666AbiGWKHJ (ORCPT ); Sat, 23 Jul 2022 06:07:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238600AbiGWKGo (ORCPT ); Sat, 23 Jul 2022 06:06:44 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B63B076EA6; Sat, 23 Jul 2022 03:01:00 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4E33DB82C1F; Sat, 23 Jul 2022 10:00:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A58BEC341C0; Sat, 23 Jul 2022 10:00:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570439; bh=5hXOOZIyVXu5H3P4QzuLuk/qpBj2HGskshf6QCS+reA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IFbTBqFvTbyflYgf7FRH/ZLrIXVed9FP0qRuLx2ddztts104oBYduVFrqcjkPZ1+R opNX7rHZU8g7UlT1amj0pg5dH/f8UCn1mFkBZWutVY1yZzvdS9ZO3cCXdk9OuPSzh2 4V68rT8cHeUMMzJ2xMrdbfhhAQrwcq6/T43K9LdM= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 087/148] x86,static_call: Use alternative RET encoding Date: Sat, 23 Jul 2022 11:54:59 +0200 Message-Id: <20220723095248.801393697@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit ee88d363d15617ff50ac24fab0ffec11113b2aeb upstream. In addition to teaching static_call about the new way to spell 'RET', there is an added complication in that static_call() is allowed to rewrite text before it is known which particular spelling is required. In order to deal with this; have a static_call specific fixup in the apply_return() 'alternative' patching routine that will rewrite the static_call trampoline to match the definite sequence. This in turn creates the problem of uniquely identifying static call trampolines. Currently trampolines are 8 bytes, the first 5 being the jmp.d32/ret sequence and the final 3 a byte sequence that spells out 'SCT'. This sequence is used in __static_call_validate() to ensure it is patching a trampoline and not a random other jmp.d32. That is, false-positives shouldn't be plenty, but aren't a big concern. OTOH the new __static_call_fixup() must not have false-positives, and 'SCT' decodes to the somewhat weird but semi plausible sequence: push %rbx rex.XB push %r12 Additionally, there are SLS concerns with immediate jumps. Combined it seems like a good moment to change the signature to a single 3 byte trap instruction that is unique to this usage and will not ever get generated by accident. As such, change the signature to: '0x0f, 0xb9, 0xcc', which decodes to: ud1 %esp, %ecx Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: skip validation as introduced by 2105a92748e8 ("static_call,x86:= Robustify trampoline patching")] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/static_call.h | 17 ++++++++++++++++ arch/x86/kernel/alternative.c | 12 +++++++---- arch/x86/kernel/static_call.c | 38 ++++++++++++++++++++++++++++++++= ++++- 3 files changed, 62 insertions(+), 5 deletions(-) --- a/arch/x86/include/asm/static_call.h +++ b/arch/x86/include/asm/static_call.h @@ -21,6 +21,16 @@ * relative displacement across sections. */ =20 +/* + * The trampoline is 8 bytes and of the general form: + * + * jmp.d32 \func + * ud1 %esp, %ecx + * + * That trailing #UD provides both a speculation stop and serves as a uniq= ue + * 3 byte signature identifying static call trampolines. Also see tramp_ud= [] + * and __static_call_fixup(). + */ #define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insns) \ asm(".pushsection .static_call.text, \"ax\" \n" \ ".align 4 \n" \ @@ -34,8 +44,13 @@ #define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \ __ARCH_DEFINE_STATIC_CALL_TRAMP(name, ".byte 0xe9; .long " #func " - (. += 4)") =20 +#ifdef CONFIG_RETPOLINE +#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ + __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "jmp __x86_return_thunk") +#else #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "ret; int3; nop; nop; nop") +#endif =20 =20 #define ARCH_ADD_TRAMP_KEY(name) \ @@ -44,4 +59,6 @@ ".long " STATIC_CALL_KEY_STR(name) " - . \n" \ ".popsection \n") =20 +extern bool __static_call_fixup(void *tramp, u8 op, void *dest); + #endif /* _ASM_STATIC_CALL_H */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -693,18 +693,22 @@ void __init_or_module noinline apply_ret s32 *s; =20 for (s =3D start; s < end; s++) { - void *addr =3D (void *)s + *s; + void *dest =3D NULL, *addr =3D (void *)s + *s; struct insn insn; int len, ret; u8 bytes[16]; - u8 op1; + u8 op; =20 ret =3D insn_decode_kernel(&insn, addr); if (WARN_ON_ONCE(ret < 0)) continue; =20 - op1 =3D insn.opcode.bytes[0]; - if (WARN_ON_ONCE(op1 !=3D JMP32_INSN_OPCODE)) + op =3D insn.opcode.bytes[0]; + if (op =3D=3D JMP32_INSN_OPCODE) + dest =3D addr + insn.length + insn.immediate.value; + + if (__static_call_fixup(addr, op, dest) || + WARN_ON_ONCE(dest !=3D &__x86_return_thunk)) continue; =20 DPRINTK("return thunk at: %pS (%px) len: %d to: %pS", --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -11,6 +11,13 @@ enum insn_type { RET =3D 3, /* tramp / site cond-tail-call */ }; =20 +/* + * ud1 %esp, %ecx - a 3 byte #UD that is unique to trampolines, chosen such + * that there is no false-positive trampoline identification while also be= ing a + * speculation stop. + */ +static const u8 tramp_ud[] =3D { 0x0f, 0xb9, 0xcc }; + static const u8 retinsn[] =3D { RET_INSN_OPCODE, 0xcc, 0xcc, 0xcc, 0xcc }; =20 static void __ref __static_call_transform(void *insn, enum insn_type type,= void *func) @@ -32,7 +39,10 @@ static void __ref __static_call_transfor break; =20 case RET: - code =3D &retinsn; + if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) + code =3D text_gen_insn(JMP32_INSN_OPCODE, insn, &__x86_return_thunk); + else + code =3D &retinsn; break; } =20 @@ -97,3 +107,29 @@ void arch_static_call_transform(void *si mutex_unlock(&text_mutex); } EXPORT_SYMBOL_GPL(arch_static_call_transform); + +#ifdef CONFIG_RETPOLINE +/* + * This is called by apply_returns() to fix up static call trampolines, + * specifically ARCH_DEFINE_STATIC_CALL_NULL_TRAMP which is recorded as + * having a return trampoline. + * + * The problem is that static_call() is available before determining + * X86_FEATURE_RETHUNK and, by implication, running alternatives. + * + * This means that __static_call_transform() above can have overwritten the + * return trampoline and we now need to fix things up to be consistent. + */ +bool __static_call_fixup(void *tramp, u8 op, void *dest) +{ + if (memcmp(tramp+5, tramp_ud, 3)) { + /* Not a trampoline site, not our problem. */ + return false; + } + + if (op =3D=3D RET_INSN_OPCODE || dest =3D=3D &__x86_return_thunk) + __static_call_transform(tramp, RET, NULL); + + return true; +} +#endif From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB0C1C433EF for ; Sat, 23 Jul 2022 10:07:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238293AbiGWKHN (ORCPT ); Sat, 23 Jul 2022 06:07:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238447AbiGWKGu (ORCPT ); Sat, 23 Jul 2022 06:06:50 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CFF67754C; Sat, 23 Jul 2022 03:01:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3088CB82C1B; Sat, 23 Jul 2022 10:00:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7C0BCC341C0; Sat, 23 Jul 2022 10:00:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570441; bh=uKk+3Op0t3ZhHfR38XvrZeuKdiesDxkqpIqWRpOsNHc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mE3zeWyeRQ8EHoaMX8HGYXHdP76ca8y5FBKy2NDopNZAH8Z9Ht/eX3na4HfjvHncg GHyZ5hkM0hMZwKNTIAv3EDCCcqS+twZIE+MTniMFB/NPB1+qpkuCJj5AbSSyQ4wHyH +3CcPjmc2eajJQE7n3tWvF3NsJvVX19ndOkT9VSs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 088/148] x86/ftrace: Use alternative RET encoding Date: Sat, 23 Jul 2022 11:55:00 +0200 Message-Id: <20220723095249.100961896@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 1f001e9da6bbf482311e45e48f53c2bd2179e59c upstream. Use the return thunk in ftrace trampolines, if needed. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: still copy return from ftrace_stub] [cascardo: use memcpy(text_gen_insn) as there is no __text_gen_insn] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/ftrace.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -308,7 +308,7 @@ union ftrace_op_code_union { } __attribute__((packed)); }; =20 -#define RET_SIZE 1 + IS_ENABLED(CONFIG_SLS) +#define RET_SIZE (IS_ENABLED(CONFIG_RETPOLINE) ? 5 : 1 + IS_ENABLED(CONFI= G_SLS)) =20 static unsigned long create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) @@ -367,7 +367,10 @@ create_trampoline(struct ftrace_ops *ops =20 /* The trampoline ends with ret(q) */ retq =3D (unsigned long)ftrace_stub; - ret =3D copy_from_kernel_nofault(ip, (void *)retq, RET_SIZE); + if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) + memcpy(ip, text_gen_insn(JMP32_INSN_OPCODE, ip, &__x86_return_thunk), JM= P32_INSN_SIZE); + else + ret =3D copy_from_kernel_nofault(ip, (void *)retq, RET_SIZE); if (WARN_ON(ret < 0)) goto fail; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22FDDC43334 for ; Sat, 23 Jul 2022 10:07:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238709AbiGWKHW (ORCPT ); Sat, 23 Jul 2022 06:07:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238475AbiGWKGy (ORCPT ); Sat, 23 Jul 2022 06:06:54 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EFED377546; Sat, 23 Jul 2022 03:01:06 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 027F4B82C21; Sat, 23 Jul 2022 10:00:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4B7B9C341C0; Sat, 23 Jul 2022 10:00:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570444; bh=QtnsdZLxc5e//iy3s2gLvf5/wn1Pnwcij+P+Wg6G5ag=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=1ZYzrpfZRDLnwLz4epgHmUuqyEgbtu4ecHLrb4yrSdUKT47jOnJwwKmKgQYKNNV5M wjXujk19fAEiaQo8C5T5fMh2YD+mRLEaeTnUpjmFf9yVRLaqg4xfzimMq5DGWQxe/I 3+rSyFapyoYVHUB+sPKzz2yKjJYZmz9aB4VzJQcE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 089/148] x86/bpf: Use alternative RET encoding Date: Sat, 23 Jul 2022 11:55:01 +0200 Message-Id: <20220723095249.358079262@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit d77cfe594ad50e0bf95d457e02ccd578791b2a15 upstream. Use the return thunk in eBPF generated code, if needed. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: add the necessary cnt variable to emit_return()] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/net/bpf_jit_comp.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -399,6 +399,22 @@ static void emit_indirect_jump(u8 **ppro *pprog =3D prog; } =20 +static void emit_return(u8 **pprog, u8 *ip) +{ + u8 *prog =3D *pprog; + int cnt =3D 0; + + if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { + emit_jump(&prog, &__x86_return_thunk, ip); + } else { + EMIT1(0xC3); /* ret */ + if (IS_ENABLED(CONFIG_SLS)) + EMIT1(0xCC); /* int3 */ + } + + *pprog =3D prog; +} + /* * Generate the following code: * @@ -1443,7 +1459,7 @@ emit_jmp: ctx->cleanup_addr =3D proglen; pop_callee_regs(&prog, callee_regs_used); EMIT1(0xC9); /* leave */ - EMIT1(0xC3); /* ret */ + emit_return(&prog, image + addrs[i - 1] + (prog - temp)); break; =20 default: @@ -1884,7 +1900,7 @@ int arch_prepare_bpf_trampoline(struct b if (flags & BPF_TRAMP_F_SKIP_FRAME) /* skip our return address and return to parent */ EMIT4(0x48, 0x83, 0xC4, 8); /* add rsp, 8 */ - EMIT1(0xC3); /* ret */ + emit_return(&prog, prog); /* Make sure the trampoline generation logic doesn't overflow */ if (WARN_ON_ONCE(prog > (u8 *)image_end - BPF_INSN_SAFETY)) { ret =3D -EFAULT; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23B97C433EF for ; Sat, 23 Jul 2022 10:07:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238720AbiGWKHY (ORCPT ); Sat, 23 Jul 2022 06:07:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238494AbiGWKG4 (ORCPT ); Sat, 23 Jul 2022 06:06:56 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75E70C3805; Sat, 23 Jul 2022 03:01:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1BA2361260; Sat, 23 Jul 2022 10:00:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 296EAC341C0; Sat, 23 Jul 2022 10:00:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570447; bh=9U0r2uHXT1NBk+EpHeWfpqK2tFsMJfPQcA+s6ZHAngc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YwRK+LbnMO2woH1uu9U+bGqmWSUcNZZ0Nxsr2ALIBm+pfYvolJhW4BkuuJZEpDgLV 5fIZyNcKmbWbG6oEVBOtEBE7XT0e03TNGYNfwkdEsC0kX9+GambvNfSfHW13ovfiRM g1ioFS+fvFX62DZMKJys8vAPITFjsVZkGqfJXrw8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 090/148] x86/kvm: Fix SETcc emulation for return thunks Date: Sat, 23 Jul 2022 11:55:02 +0200 Message-Id: <20220723095249.632470153@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit af2e140f34208a5dfb6b7a8ad2d56bda88f0524d upstream. Prepare the SETcc fastop stuff for when RET can be larger still. The tricky bit here is that the expressions should not only be constant C expressions, but also absolute GAS expressions. This means no ?: and 'true' is ~0. Also ensure em_setcc() has the same alignment as the actual FOP_SETCC() ops, this ensures there cannot be an alignment hole between em_setcc() and the first op. Additionally, add a .skip directive to the FOP_SETCC() macro to fill any remaining space with INT3 traps; however the primary purpose of this directive is to generate AS warnings when the remaining space goes negative. Which is a very good indication the alignment magic went side-ways. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: ignore ENDBR when computing SETCC_LENGTH] [cascardo: conflict fixup] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/emulate.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -322,13 +322,15 @@ static int fastop(struct x86_emulate_ctx #define FOP_RET(name) \ __FOP_RET(#name) =20 -#define FOP_START(op) \ +#define __FOP_START(op, align) \ extern void em_##op(struct fastop *fake); \ asm(".pushsection .text, \"ax\" \n\t" \ ".global em_" #op " \n\t" \ - ".align " __stringify(FASTOP_SIZE) " \n\t" \ + ".align " __stringify(align) " \n\t" \ "em_" #op ":\n\t" =20 +#define FOP_START(op) __FOP_START(op, FASTOP_SIZE) + #define FOP_END \ ".popsection") =20 @@ -432,15 +434,14 @@ static int fastop(struct x86_emulate_ctx /* * Depending on .config the SETcc functions look like: * - * SETcc %al [3 bytes] - * RET [1 byte] - * INT3 [1 byte; CONFIG_SLS] - * - * Which gives possible sizes 4 or 5. When rounded up to the - * next power-of-two alignment they become 4 or 8. + * SETcc %al [3 bytes] + * RET | JMP __x86_return_thunk [1,5 bytes; CONFIG_RETPOLINE] + * INT3 [1 byte; CONFIG_SLS] */ -#define SETCC_LENGTH (4 + IS_ENABLED(CONFIG_SLS)) -#define SETCC_ALIGN (4 << IS_ENABLED(CONFIG_SLS)) +#define RET_LENGTH (1 + (4 * IS_ENABLED(CONFIG_RETPOLINE)) + \ + IS_ENABLED(CONFIG_SLS)) +#define SETCC_LENGTH (3 + RET_LENGTH) +#define SETCC_ALIGN (4 << ((SETCC_LENGTH > 4) & 1) << ((SETCC_LENGTH > 8) = & 1)) static_assert(SETCC_LENGTH <=3D SETCC_ALIGN); =20 #define FOP_SETCC(op) \ @@ -448,14 +449,15 @@ static_assert(SETCC_LENGTH <=3D SETCC_ALIG ".type " #op ", @function \n\t" \ #op ": \n\t" \ #op " %al \n\t" \ - __FOP_RET(#op) + __FOP_RET(#op) \ + ".skip " __stringify(SETCC_ALIGN) " - (.-" #op "), 0xcc \n\t" =20 asm(".pushsection .fixup, \"ax\"\n" ".global kvm_fastop_exception \n" "kvm_fastop_exception: xor %esi, %esi; " ASM_RET ".popsection"); =20 -FOP_START(setcc) +__FOP_START(setcc, SETCC_ALIGN) FOP_SETCC(seto) FOP_SETCC(setno) FOP_SETCC(setc) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24D32C433EF for ; Sat, 23 Jul 2022 10:07:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238738AbiGWKHh (ORCPT ); Sat, 23 Jul 2022 06:07:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238529AbiGWKG6 (ORCPT ); Sat, 23 Jul 2022 06:06:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 820DFC3825; Sat, 23 Jul 2022 03:01:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2440F611D4; Sat, 23 Jul 2022 10:00:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 05896C341C0; Sat, 23 Jul 2022 10:00:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570450; bh=I1nVtJhBaBbQvFmmxP+Ixri0tvBlRaN4Uz+xRtIocyY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PzdVgff7I6wmrXz3Yk18MY2u32PPwfr6fboP2pV78y0CtBLBxzZAhCWmHV8g0cH/p db9h8Bh2gayPkjBICoz6XxkvEiZp8eGi1ecT/K37kW28zDA+sWr8we8gdawqLTkjS+ RfIW4vA6rUZXTmhM/SH295ZMJlis6sm7FGkxPFjo= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 091/148] x86/vsyscall_emu/64: Dont use RET in vsyscall emulation Date: Sat, 23 Jul 2022 11:55:03 +0200 Message-Id: <20220723095249.907083729@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 15583e514eb16744b80be85dea0774ece153177d upstream. This is userspace code and doesn't play by the normal kernel rules. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/vsyscall/vsyscall_emu_64.S | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) --- a/arch/x86/entry/vsyscall/vsyscall_emu_64.S +++ b/arch/x86/entry/vsyscall/vsyscall_emu_64.S @@ -19,17 +19,20 @@ __vsyscall_page: =20 mov $__NR_gettimeofday, %rax syscall - RET + ret + int3 =20 .balign 1024, 0xcc mov $__NR_time, %rax syscall - RET + ret + int3 =20 .balign 1024, 0xcc mov $__NR_getcpu, %rax syscall - RET + ret + int3 =20 .balign 4096, 0xcc From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CA22C433EF for ; Sat, 23 Jul 2022 10:06:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238592AbiGWKG6 (ORCPT ); Sat, 23 Jul 2022 06:06:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238572AbiGWKGf (ORCPT ); Sat, 23 Jul 2022 06:06:35 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FF62A0256; Sat, 23 Jul 2022 03:00:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8A31DB82C1D; Sat, 23 Jul 2022 10:00:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01F5DC341C0; Sat, 23 Jul 2022 10:00:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570453; bh=ufygdNxYvRwqap/gxVvpqhpvRh4OFs5NCTipEhRPTSI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UspT52ihYLQx7uLfrpuOePuXgRbiDwDCdf8e4a+wFSyV8PLD4dOZ2rpcsNJlfQvs7 AhHZQzoLpNHoMwi5JN3gRfqBJtcbikU9E3dLuuM+Uw3DXavuX6cu75MzxKwVLC3eYk I8jDUpJjb0aMIGf/hQYNzpoTXBovooKWXteAC9Jc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Kim Phillips , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 092/148] x86/sev: Avoid using __x86_return_thunk Date: Sat, 23 Jul 2022 11:55:04 +0200 Message-Id: <20220723095250.164785084@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Kim Phillips commit 0ee9073000e8791f8b134a8ded31bcc767f7f232 upstream. Specifically, it's because __enc_copy() encrypts the kernel after being relocated outside the kernel in sme_encrypt_execute(), and the RET macro's jmp offset isn't amended prior to execution. Signed-off-by: Kim Phillips Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/mm/mem_encrypt_boot.S | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) --- a/arch/x86/mm/mem_encrypt_boot.S +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -65,7 +65,9 @@ SYM_FUNC_START(sme_encrypt_execute) movq %rbp, %rsp /* Restore original stack pointer */ pop %rbp =20 - RET + /* Offset to __x86_return_thunk would be wrong here */ + ret + int3 SYM_FUNC_END(sme_encrypt_execute) =20 SYM_FUNC_START(__enc_copy) @@ -151,6 +153,8 @@ SYM_FUNC_START(__enc_copy) pop %r12 pop %r15 =20 - RET + /* Offset to __x86_return_thunk would be wrong here */ + ret + int3 .L__enc_copy_end: SYM_FUNC_END(__enc_copy) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 666FEC43334 for ; Sat, 23 Jul 2022 10:07:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238457AbiGWKHF (ORCPT ); Sat, 23 Jul 2022 06:07:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238112AbiGWKGk (ORCPT ); Sat, 23 Jul 2022 06:06:40 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CE8376955; Sat, 23 Jul 2022 03:00:59 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6F082B82C22; Sat, 23 Jul 2022 10:00:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2DF6C341C0; Sat, 23 Jul 2022 10:00:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570456; bh=BFyioED30kwJD9PGGxY9ZF6TW0CFgre5ZqlPuYrWpL8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eMiu8dTBqOz0iMWuu4dmBRinvClG07BErClAeVmQw7ZbOCibktUK1n8UX8JWY3HHq Jgkwnk9VQoj547b+Tzn2GYv5GlbIR43kgj6yBF3pTHsyZ8bTnOvMF2gEgZ3QJfCmxi BqVA5qCaubz/vScT65dy0lnrLhyhdcWoO+C9uUHI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 093/148] x86: Use return-thunk in asm code Date: Sat, 23 Jul 2022 11:55:05 +0200 Message-Id: <20220723095250.449568556@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit aa3d480315ba6c3025a60958e1981072ea37c3df upstream. Use the return thunk in asm code. If the thunk isn't needed, it will get patched into a RET instruction during boot by apply_returns(). Since alternatives can't handle relocations outside of the first instruction, putting a 'jmp __x86_return_thunk' in one is not valid, therefore carve out the memmove ERMS path into a separate label and jump to it. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: no RANDSTRUCT_CFLAGS] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/vdso/Makefile | 1 + arch/x86/include/asm/linkage.h | 8 ++++++++ arch/x86/lib/memmove_64.S | 7 ++++++- 3 files changed, 15 insertions(+), 1 deletion(-) --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -91,6 +91,7 @@ endif endif =20 $(vobjs): KBUILD_CFLAGS :=3D $(filter-out $(GCC_PLUGINS_CFLAGS) $(RETPOLIN= E_CFLAGS),$(KBUILD_CFLAGS)) $(CFL) +$(vobjs): KBUILD_AFLAGS +=3D -DBUILD_VDSO =20 # # vDSO code runs in userspace and -pg doesn't help with profiling anyway. --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -18,19 +18,27 @@ #define __ALIGN_STR __stringify(__ALIGN) #endif =20 +#if defined(CONFIG_RETPOLINE) && !defined(__DISABLE_EXPORTS) && !defined(B= UILD_VDSO) +#define RET jmp __x86_return_thunk +#else /* CONFIG_RETPOLINE */ #ifdef CONFIG_SLS #define RET ret; int3 #else #define RET ret #endif +#endif /* CONFIG_RETPOLINE */ =20 #else /* __ASSEMBLY__ */ =20 +#if defined(CONFIG_RETPOLINE) && !defined(__DISABLE_EXPORTS) && !defined(B= UILD_VDSO) +#define ASM_RET "jmp __x86_return_thunk\n\t" +#else /* CONFIG_RETPOLINE */ #ifdef CONFIG_SLS #define ASM_RET "ret; int3\n\t" #else #define ASM_RET "ret\n\t" #endif +#endif /* CONFIG_RETPOLINE */ =20 #endif /* __ASSEMBLY__ */ =20 --- a/arch/x86/lib/memmove_64.S +++ b/arch/x86/lib/memmove_64.S @@ -40,7 +40,7 @@ SYM_FUNC_START(__memmove) /* FSRM implies ERMS =3D> no length checks, do the copy directly */ .Lmemmove_begin_forward: ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM - ALTERNATIVE "", __stringify(movq %rdx, %rcx; rep movsb; RET), X86_FEATURE= _ERMS + ALTERNATIVE "", "jmp .Lmemmove_erms", X86_FEATURE_ERMS =20 /* * movsq instruction have many startup latency @@ -206,6 +206,11 @@ SYM_FUNC_START(__memmove) movb %r11b, (%rdi) 13: RET + +.Lmemmove_erms: + movq %rdx, %rcx + rep movsb + RET SYM_FUNC_END(__memmove) SYM_FUNC_END_ALIAS(memmove) EXPORT_SYMBOL(__memmove) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB674C433EF for ; Sat, 23 Jul 2022 10:07:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238103AbiGWKHA (ORCPT ); Sat, 23 Jul 2022 06:07:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238430AbiGWKGg (ORCPT ); Sat, 23 Jul 2022 06:06:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A496BE9CC; Sat, 23 Jul 2022 03:01:00 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8DCB9611BF; Sat, 23 Jul 2022 10:00:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 982B2C341C0; Sat, 23 Jul 2022 10:00:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570459; bh=oAN6Q4dQiIIJu3yvwAyYImnSQUKBxZrz7e4pEBocnjg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=1vv5QUkoDaMtURlGC8rSwUr9mlNQd2F4RcpkDJVee6gohyzjZtF4iHg9cmeTASUfH e7N7bUjxQPxp/8wuYXYUf3gRUiEHMnucK4kb9PrZLSTz/msJKaOKazfcDp9rm9DQis F1cRLg9vSMIXPWXxLUMSE13ilCSGH4ps53YOhiHc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 094/148] objtool: Treat .text.__x86.* as noinstr Date: Sat, 23 Jul 2022 11:55:06 +0200 Message-Id: <20220723095250.747563428@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 951ddecf435659553ed15a9214e153a3af43a9a1 upstream. Needed because zen_untrain_ret() will be called from noinstr code. Also makes sense since the thunks MUST NOT contain instrumentation nor be poked with dynamic instrumentation. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/check.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -366,7 +366,8 @@ static int decode_instructions(struct ob sec->text =3D true; =20 if (!strcmp(sec->name, ".noinstr.text") || - !strcmp(sec->name, ".entry.text")) + !strcmp(sec->name, ".entry.text") || + !strncmp(sec->name, ".text.__x86.", 12)) sec->noinstr =3D true; =20 for (offset =3D 0; offset < sec->len; offset +=3D insn->len) { From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6D9FC433EF for ; Sat, 23 Jul 2022 10:06:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238552AbiGWKGK (ORCPT ); Sat, 23 Jul 2022 06:06:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238517AbiGWKFu (ORCPT ); Sat, 23 Jul 2022 06:05:50 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7939EA896A; Sat, 23 Jul 2022 03:00:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D2E7D611BD; Sat, 23 Jul 2022 10:00:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E0092C341C0; Sat, 23 Jul 2022 10:00:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570433; bh=2gFvAKY2qUtOBiisicG0EG4pjl3AtXjl2CIWBvwmbLs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZqD37LGCRT4iw6KBEUYlj+kI3wyFYw+ptTRZQqF9lWrV+O65EDUBnr3I3x3RvQWVN wIClBBXJf1AXJp8lIOyhiMdwBveY8B5clVCxnYCd7lvU8ASylrUuh7HljzeDWVDXKW fN/OR8efc7RaCVDf4bbBriDwdVU44jdGISW/6GKE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Andrew Cooper , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 095/148] x86: Add magic AMD return-thunk Date: Sat, 23 Jul 2022 11:55:07 +0200 Message-Id: <20220723095251.031058766@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit a149180fbcf336e97ce4eb2cdc13672727feb94d upstream. Note: needs to be in a section distinct from Retpolines such that the Retpoline RET substitution cannot possibly use immediate jumps. ORC unwinding for zen_untrain_ret() and __x86_return_thunk() is a little tricky but works due to the fact that zen_untrain_ret() doesn't have any stack ops and as such will emit a single ORC entry at the start (+0x3f). Meanwhile, unwinding an IP, including the __x86_return_thunk() one (+0x40) will search for the largest ORC entry smaller or equal to the IP, these will find the one ORC entry (+0x3f) and all works. [ Alexandre: SVM part. ] [ bp: Build fix, massages. ] Suggested-by: Andrew Cooper Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: conflicts at arch/x86/entry/entry_64_compat.S] [cascardo: there is no ANNOTATE_NOENDBR] [cascardo: objtool commit 34c861e806478ac2ea4032721defbf1d6967df08 missing] [cascardo: conflict fixup] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: SEV-ES is not supported, so drop the change in arch/x86/kvm/svm/vmenter.S] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/entry_64.S | 6 ++ arch/x86/entry/entry_64_compat.S | 4 + arch/x86/include/asm/cpufeatures.h | 1=20 arch/x86/include/asm/disabled-features.h | 3 - arch/x86/include/asm/nospec-branch.h | 17 ++++++++ arch/x86/kernel/vmlinux.lds.S | 2=20 arch/x86/kvm/svm/vmenter.S | 9 ++++ arch/x86/lib/retpoline.S | 63 ++++++++++++++++++++++++++= +++-- tools/objtool/check.c | 20 ++++++++- 9 files changed, 117 insertions(+), 8 deletions(-) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -102,6 +102,7 @@ SYM_CODE_START(entry_SYSCALL_64) movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) + UNTRAIN_RET =20 /* Construct struct pt_regs on stack */ pushq $__USER_DS /* pt_regs->ss */ @@ -675,6 +676,7 @@ native_irq_return_ldt: pushq %rdi /* Stash user RDI */ swapgs /* to kernel GS */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdi /* to kernel CR3 */ + UNTRAIN_RET =20 movq PER_CPU_VAR(espfix_waddr), %rdi movq %rax, (0*8)(%rdi) /* user RAX */ @@ -910,6 +912,7 @@ SYM_CODE_START_LOCAL(paranoid_entry) * be retrieved from a kernel internal table. */ SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax save_reg=3D%r14 + UNTRAIN_RET =20 /* * Handling GSBASE depends on the availability of FSGSBASE. @@ -1022,6 +1025,7 @@ SYM_CODE_START_LOCAL(error_entry) FENCE_SWAPGS_USER_ENTRY /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax + UNTRAIN_RET =20 .Lerror_entry_from_usermode_after_swapgs: /* Put us onto the real thread stack. */ @@ -1077,6 +1081,7 @@ SYM_CODE_START_LOCAL(error_entry) SWAPGS FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax + UNTRAIN_RET =20 /* * Pretend that the exception came from user mode: set up pt_regs @@ -1171,6 +1176,7 @@ SYM_CODE_START(asm_exc_nmi) movq %rsp, %rdx movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp UNWIND_HINT_IRET_REGS base=3D%rdx offset=3D8 + UNTRAIN_RET pushq 5*8(%rdx) /* pt_regs->ss */ pushq 4*8(%rdx) /* pt_regs->rsp */ pushq 3*8(%rdx) /* pt_regs->flags */ --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include =20 @@ -71,6 +72,7 @@ SYM_CODE_START(entry_SYSENTER_compat) pushq $__USER32_CS /* pt_regs->cs */ pushq $0 /* pt_regs->ip =3D 0 (placeholder) */ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) + UNTRAIN_RET =20 /* * User tracing code (ptrace or signal handlers) might assume that @@ -211,6 +213,7 @@ SYM_CODE_START(entry_SYSCALL_compat) movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_compat_safe_stack, SYM_L_GLOBAL) + UNTRAIN_RET =20 /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ @@ -377,6 +380,7 @@ SYM_CODE_START(entry_INT80_compat) pushq (%rdi) /* pt_regs->di */ .Lint80_keep_stack: =20 + UNTRAIN_RET pushq %rsi /* pt_regs->si */ xorl %esi, %esi /* nospec si */ pushq %rdx /* pt_regs->dx */ --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -297,6 +297,7 @@ #define X86_FEATURE_RETPOLINE (11*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ #define X86_FEATURE_RETPOLINE_LFENCE (11*32+13) /* "" Use LFENCE for Spect= re variant 2 */ #define X86_FEATURE_RETHUNK (11*32+14) /* "" Use REturn THUNK */ +#define X86_FEATURE_UNRET (11*32+15) /* "" AMD BTB untrain return */ =20 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instruction= s */ --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -61,7 +61,8 @@ #else # define DISABLE_RETPOLINE ((1 << (X86_FEATURE_RETPOLINE & 31)) | \ (1 << (X86_FEATURE_RETPOLINE_LFENCE & 31)) | \ - (1 << (X86_FEATURE_RETHUNK & 31))) + (1 << (X86_FEATURE_RETHUNK & 31)) | \ + (1 << (X86_FEATURE_UNRET & 31))) #endif =20 /* Force disable because it's broken beyond repair */ --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -112,6 +112,22 @@ #endif .endm =20 +/* + * Mitigate RETBleed for AMD/Hygon Zen uarch. Requires KERNEL CR3 because = the + * return thunk isn't mapped into the userspace tables (then again, AMD + * typically has NO_MELTDOWN). + * + * Doesn't clobber any registers but does require a stable stack. + * + * As such, this must be placed after every *SWITCH_TO_KERNEL_CR3 at a poi= nt + * where we have a stack but before any RET instruction. + */ +.macro UNTRAIN_RET +#ifdef CONFIG_RETPOLINE + ALTERNATIVE "", "call zen_untrain_ret", X86_FEATURE_UNRET +#endif +.endm + #else /* __ASSEMBLY__ */ =20 #define ANNOTATE_RETPOLINE_SAFE \ @@ -121,6 +137,7 @@ ".popsection\n\t" =20 extern void __x86_return_thunk(void); +extern void zen_untrain_ret(void); =20 #ifdef CONFIG_RETPOLINE =20 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -142,7 +142,7 @@ SECTIONS =20 #ifdef CONFIG_RETPOLINE __indirect_thunk_start =3D .; - *(.text.__x86.indirect_thunk) + *(.text.__x86.*) __indirect_thunk_end =3D .; #endif } :text =3D0xcccc --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -129,6 +129,15 @@ SYM_FUNC_START(__svm_vcpu_run) #endif =20 /* + * Mitigate RETBleed for AMD/Hygon Zen uarch. RET should be + * untrained as soon as we exit the VM and are back to the + * kernel. This should be done before re-enabling interrupts + * because interrupt handlers won't sanitize 'ret' if the return is + * from the kernel. + */ + UNTRAIN_RET + + /* * Clear all general purpose registers except RSP and RAX to prevent * speculative use of the guest's values, even those that are reloaded * via the stack. In theory, an L1 cache miss when restoring registers --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -71,10 +71,67 @@ SYM_CODE_END(__x86_indirect_thunk_array) * This function name is magical and is used by -mfunction-return=3Dthunk-= extern * for the compiler to generate JMPs to it. */ -SYM_CODE_START(__x86_return_thunk) - UNWIND_HINT_EMPTY + .section .text.__x86.return_thunk + +/* + * Safety details here pertain to the AMD Zen{1,2} microarchitecture: + * 1) The RET at __x86_return_thunk must be on a 64 byte boundary, for + * alignment within the BTB. + * 2) The instruction at zen_untrain_ret must contain, and not + * end with, the 0xc3 byte of the RET. + * 3) STIBP must be enabled, or SMT disabled, to prevent the sibling thread + * from re-poisioning the BTB prediction. + */ + .align 64 + .skip 63, 0xcc +SYM_FUNC_START_NOALIGN(zen_untrain_ret); + + /* + * As executed from zen_untrain_ret, this is: + * + * TEST $0xcc, %bl + * LFENCE + * JMP __x86_return_thunk + * + * Executing the TEST instruction has a side effect of evicting any BTB + * prediction (potentially attacker controlled) attached to the RET, as + * __x86_return_thunk + 1 isn't an instruction boundary at the moment. + */ + .byte 0xf6 + + /* + * As executed from __x86_return_thunk, this is a plain RET. + * + * As part of the TEST above, RET is the ModRM byte, and INT3 the imm8. + * + * We subsequently jump backwards and architecturally execute the RET. + * This creates a correct BTB prediction (type=3Dret), but in the + * meantime we suffer Straight Line Speculation (because the type was + * no branch) which is halted by the INT3. + * + * With SMT enabled and STIBP active, a sibling thread cannot poison + * RET's prediction to a type of its choice, but can evict the + * prediction due to competitive sharing. If the prediction is + * evicted, __x86_return_thunk will suffer Straight Line Speculation + * which will be contained safely by the INT3. + */ +SYM_INNER_LABEL(__x86_return_thunk, SYM_L_GLOBAL) ret int3 SYM_CODE_END(__x86_return_thunk) =20 -__EXPORT_THUNK(__x86_return_thunk) + /* + * Ensure the TEST decoding / BTB invalidation is complete. + */ + lfence + + /* + * Jump back and execute the RET in the middle of the TEST instruction. + * INT3 is for SLS protection. + */ + jmp __x86_return_thunk + int3 +SYM_FUNC_END(zen_untrain_ret) +__EXPORT_THUNK(zen_untrain_ret) + +EXPORT_SYMBOL(__x86_return_thunk) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1082,7 +1082,7 @@ static void add_retpoline_call(struct ob annotate_call_site(file, insn, false); } =20 -static void add_return_call(struct objtool_file *file, struct instruction = *insn) +static void add_return_call(struct objtool_file *file, struct instruction = *insn, bool add) { /* * Return thunk tail calls are really just returns in disguise, @@ -1092,7 +1092,7 @@ static void add_return_call(struct objto insn->retpoline_safe =3D true; =20 /* Skip the non-text sections, specially .discard ones */ - if (insn->sec->text) + if (add && insn->sec->text) list_add_tail(&insn->call_node, &file->return_thunk_list); } =20 @@ -1121,7 +1121,7 @@ static int add_jump_destinations(struct add_retpoline_call(file, insn); continue; } else if (reloc->sym->return_thunk) { - add_return_call(file, insn); + add_return_call(file, insn, true); continue; } else if (insn->func) { /* internal or external sibling call (with reloc) */ @@ -1138,6 +1138,7 @@ static int add_jump_destinations(struct =20 insn->jump_dest =3D find_insn(file, dest_sec, dest_off); if (!insn->jump_dest) { + struct symbol *sym =3D find_symbol_by_offset(dest_sec, dest_off); =20 /* * This is a special case where an alt instruction @@ -1147,6 +1148,19 @@ static int add_jump_destinations(struct if (!strcmp(insn->sec->name, ".altinstr_replacement")) continue; =20 + /* + * This is a special case for zen_untrain_ret(). + * It jumps to __x86_return_thunk(), but objtool + * can't find the thunk's starting RET + * instruction, because the RET is also in the + * middle of another instruction. Objtool only + * knows about the outer instruction. + */ + if (sym && sym->return_thunk) { + add_return_call(file, insn, false); + continue; + } + WARN_FUNC("can't find jump dest instruction at %s+0x%lx", insn->sec, insn->offset, dest_sec->name, dest_off); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAA07C433EF for ; Sat, 23 Jul 2022 10:06:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238588AbiGWKGf (ORCPT ); Sat, 23 Jul 2022 06:06:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238500AbiGWKGJ (ORCPT ); Sat, 23 Jul 2022 06:06:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94ECDA9B8F; Sat, 23 Jul 2022 03:00:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B8ABF61263; Sat, 23 Jul 2022 10:00:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C1D98C341C0; Sat, 23 Jul 2022 10:00:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570436; bh=xp4hUKui/6IFtmVwexHXR6RltV7kLR8FPniaMYkkTbU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=xkYYyQKRN+PulEEJIvEI1KUAGMsYgDT5D6sjwKzoYZmzUJRCatcU/WgqX2n+UCqo5 qkGGjp8HHH9/I3ilP0/+2IxfOQexaOYu2kkmHLLQcGM5mTmVXzpKFNZ0uelSzxuf/v oOSRTu33gVdTmImsvAKF15n5UkOACJaDgG3tYDsg= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Kim Phillips , Alexandre Chartre , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 096/148] x86/bugs: Report AMD retbleed vulnerability Date: Sat, 23 Jul 2022 11:55:08 +0200 Message-Id: <20220723095251.281444229@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexandre Chartre commit 6b80b59b3555706508008f1f127b5412c89c7fd8 upstream. Report that AMD x86 CPUs are vulnerable to the RETBleed (Arbitrary Speculative Code Execution with Return Instructions) attack. [peterz: add hygon] [kim: invert parity; fam15h] Co-developed-by: Kim Phillips Signed-off-by: Kim Phillips Signed-off-by: Alexandre Chartre Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/bugs.c | 13 +++++++++++++ arch/x86/kernel/cpu/common.c | 19 +++++++++++++++++++ drivers/base/cpu.c | 8 ++++++++ include/linux/cpu.h | 2 ++ 5 files changed, 43 insertions(+) --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -426,5 +426,6 @@ #define X86_BUG_ITLB_MULTIHIT X86_BUG(23) /* CPU may incur MCE during cer= tain page attribute changes */ #define X86_BUG_SRBDS X86_BUG(24) /* CPU may leak RNG bits if not mitiga= ted */ #define X86_BUG_MMIO_STALE_DATA X86_BUG(25) /* CPU is affected by Process= or MMIO Stale Data vulnerabilities */ +#define X86_BUG_RETBLEED X86_BUG(26) /* CPU is affected by RETBleed */ =20 #endif /* _ASM_X86_CPUFEATURES_H */ --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1917,6 +1917,11 @@ static ssize_t srbds_show_state(char *bu return sprintf(buf, "%s\n", srbds_strings[srbds_mitigation]); } =20 +static ssize_t retbleed_show_state(char *buf) +{ + return sprintf(buf, "Vulnerable\n"); +} + static ssize_t cpu_show_common(struct device *dev, struct device_attribute= *attr, char *buf, unsigned int bug) { @@ -1962,6 +1967,9 @@ static ssize_t cpu_show_common(struct de case X86_BUG_MMIO_STALE_DATA: return mmio_stale_data_show_state(buf); =20 + case X86_BUG_RETBLEED: + return retbleed_show_state(buf); + default: break; } @@ -2018,4 +2026,9 @@ ssize_t cpu_show_mmio_stale_data(struct { return cpu_show_common(dev, attr, buf, X86_BUG_MMIO_STALE_DATA); } + +ssize_t cpu_show_retbleed(struct device *dev, struct device_attribute *att= r, char *buf) +{ + return cpu_show_common(dev, attr, buf, X86_BUG_RETBLEED); +} #endif --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1092,16 +1092,27 @@ static const __initconst struct x86_cpu_ {} }; =20 +#define VULNBL(vendor, family, model, blacklist) \ + X86_MATCH_VENDOR_FAM_MODEL(vendor, family, model, blacklist) + #define VULNBL_INTEL_STEPPINGS(model, steppings, issues) \ X86_MATCH_VENDOR_FAM_MODEL_STEPPINGS_FEATURE(INTEL, 6, \ INTEL_FAM6_##model, steppings, \ X86_FEATURE_ANY, issues) =20 +#define VULNBL_AMD(family, blacklist) \ + VULNBL(AMD, family, X86_MODEL_ANY, blacklist) + +#define VULNBL_HYGON(family, blacklist) \ + VULNBL(HYGON, family, X86_MODEL_ANY, blacklist) + #define SRBDS BIT(0) /* CPU is affected by X86_BUG_MMIO_STALE_DATA */ #define MMIO BIT(1) /* CPU is affected by Shared Buffers Data Sampling (SBDS), a variant of X8= 6_BUG_MMIO_STALE_DATA */ #define MMIO_SBDS BIT(2) +/* CPU is affected by RETbleed, speculating where you would not expect it = */ +#define RETBLEED BIT(3) =20 static const struct x86_cpu_id cpu_vuln_blacklist[] __initconst =3D { VULNBL_INTEL_STEPPINGS(IVYBRIDGE, X86_STEPPING_ANY, SRBDS), @@ -1134,6 +1145,11 @@ static const struct x86_cpu_id cpu_vuln_ VULNBL_INTEL_STEPPINGS(ATOM_TREMONT, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO= _SBDS), VULNBL_INTEL_STEPPINGS(ATOM_TREMONT_D, X86_STEPPING_ANY, MMIO), VULNBL_INTEL_STEPPINGS(ATOM_TREMONT_L, X86_STEPPINGS(0x0, 0x0), MMIO | MM= IO_SBDS), + + VULNBL_AMD(0x15, RETBLEED), + VULNBL_AMD(0x16, RETBLEED), + VULNBL_AMD(0x17, RETBLEED), + VULNBL_HYGON(0x18, RETBLEED), {} }; =20 @@ -1235,6 +1251,9 @@ static void __init cpu_set_bug_bits(stru !arch_cap_mmio_immune(ia32_cap)) setup_force_cpu_bug(X86_BUG_MMIO_STALE_DATA); =20 + if (cpu_matches(cpu_vuln_blacklist, RETBLEED)) + setup_force_cpu_bug(X86_BUG_RETBLEED); + if (cpu_matches(cpu_vuln_whitelist, NO_MELTDOWN)) return; =20 --- a/drivers/base/cpu.c +++ b/drivers/base/cpu.c @@ -572,6 +572,12 @@ ssize_t __weak cpu_show_mmio_stale_data( return sysfs_emit(buf, "Not affected\n"); } =20 +ssize_t __weak cpu_show_retbleed(struct device *dev, + struct device_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "Not affected\n"); +} + static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL); static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL); static DEVICE_ATTR(spectre_v2, 0444, cpu_show_spectre_v2, NULL); @@ -582,6 +588,7 @@ static DEVICE_ATTR(tsx_async_abort, 0444 static DEVICE_ATTR(itlb_multihit, 0444, cpu_show_itlb_multihit, NULL); static DEVICE_ATTR(srbds, 0444, cpu_show_srbds, NULL); static DEVICE_ATTR(mmio_stale_data, 0444, cpu_show_mmio_stale_data, NULL); +static DEVICE_ATTR(retbleed, 0444, cpu_show_retbleed, NULL); =20 static struct attribute *cpu_root_vulnerabilities_attrs[] =3D { &dev_attr_meltdown.attr, @@ -594,6 +601,7 @@ static struct attribute *cpu_root_vulner &dev_attr_itlb_multihit.attr, &dev_attr_srbds.attr, &dev_attr_mmio_stale_data.attr, + &dev_attr_retbleed.attr, NULL }; =20 --- a/include/linux/cpu.h +++ b/include/linux/cpu.h @@ -68,6 +68,8 @@ extern ssize_t cpu_show_srbds(struct dev extern ssize_t cpu_show_mmio_stale_data(struct device *dev, struct device_attribute *attr, char *buf); +extern ssize_t cpu_show_retbleed(struct device *dev, + struct device_attribute *attr, char *buf); =20 extern __printf(4, 5) struct device *cpu_device_create(struct device *parent, void *drvdata, From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD2CAC43334 for ; Sat, 23 Jul 2022 10:08:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238431AbiGWKIf (ORCPT ); Sat, 23 Jul 2022 06:08:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238806AbiGWKIA (ORCPT ); Sat, 23 Jul 2022 06:08:00 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 624A3C5D63; Sat, 23 Jul 2022 03:01:40 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BABD6B82C1D; Sat, 23 Jul 2022 10:01:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B057C341C0; Sat, 23 Jul 2022 10:01:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570498; bh=7NjT53egA3ATlJ9aTxXrE1cWTZ5hf+G2pbGremAuhQQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mziMBHQtaz31usY0NRquPJzbU6eCKzLPX9SSTY/Kkuf/8bVSfZJxMcBTxSlprvoJ2 t4tTgwMMIVaXicoCJb6RhhAMIKgNttvrcWjFGyi6en9ZF6febj0xYraEaE9REkIono +YuPdMVt1EkfmyFjCB/77g2ZV5mcgHYGiz+fM+Gk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Alexandre Chartre , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 097/148] x86/bugs: Add AMD retbleed= boot parameter Date: Sat, 23 Jul 2022 11:55:09 +0200 Message-Id: <20220723095251.551421670@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexandre Chartre commit 7fbf47c7ce50b38a64576b150e7011ae73d54669 upstream. Add the "retbleed=3D" boot parameter to select a mitigation for RETBleed. Possible values are "off", "auto" and "unret" (JMP2RET mitigation). The default value is "auto". Currently, "retbleed=3Dauto" will select the unret mitigation on AMD and Hygon and no mitigation on Intel (JMP2RET is not effective on Intel). [peterz: rebase; add hygon] [jpoimboe: cleanups] Signed-off-by: Alexandre Chartre Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Documentation/admin-guide/kernel-parameters.txt | 15 +++ arch/x86/Kconfig | 3=20 arch/x86/kernel/cpu/bugs.c | 108 +++++++++++++++++++= ++++- 3 files changed, 125 insertions(+), 1 deletion(-) --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4656,6 +4656,21 @@ =20 retain_initrd [RAM] Keep initrd memory after extraction =20 + retbleed=3D [X86] Control mitigation of RETBleed (Arbitrary + Speculative Code Execution with Return Instructions) + vulnerability. + + off - unconditionally disable + auto - automatically select a migitation + unret - force enable untrained return thunks, + only effective on AMD Zen {1,2} + based systems. + + Selecting 'auto' will choose a mitigation method at run + time according to the CPU. + + Not specifying this option is equivalent to retbleed=3Dauto. + rfkill.default_state=3D 0 "airplane mode". All wifi, bluetooth, wimax, gps, fm, etc. communication is blocked by default. --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -465,6 +465,9 @@ config RETPOLINE config CC_HAS_SLS def_bool $(cc-option,-mharden-sls=3Dall) =20 +config CC_HAS_RETURN_THUNK + def_bool $(cc-option,-mfunction-return=3Dthunk-extern) + config SLS bool "Mitigate Straight-Line-Speculation" depends on CC_HAS_SLS && X86_64 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -37,6 +37,7 @@ #include "cpu.h" =20 static void __init spectre_v1_select_mitigation(void); +static void __init retbleed_select_mitigation(void); static void __init spectre_v2_select_mitigation(void); static void __init ssb_select_mitigation(void); static void __init l1tf_select_mitigation(void); @@ -112,6 +113,12 @@ void __init check_bugs(void) =20 /* Select the proper CPU mitigations before patching alternatives: */ spectre_v1_select_mitigation(); + retbleed_select_mitigation(); + /* + * spectre_v2_select_mitigation() relies on the state set by + * retbleed_select_mitigation(); specifically the STIBP selection is + * forced for UNRET. + */ spectre_v2_select_mitigation(); ssb_select_mitigation(); l1tf_select_mitigation(); @@ -709,6 +716,100 @@ static int __init nospectre_v1_cmdline(c early_param("nospectre_v1", nospectre_v1_cmdline); =20 #undef pr_fmt +#define pr_fmt(fmt) "RETBleed: " fmt + +enum retbleed_mitigation { + RETBLEED_MITIGATION_NONE, + RETBLEED_MITIGATION_UNRET, +}; + +enum retbleed_mitigation_cmd { + RETBLEED_CMD_OFF, + RETBLEED_CMD_AUTO, + RETBLEED_CMD_UNRET, +}; + +const char * const retbleed_strings[] =3D { + [RETBLEED_MITIGATION_NONE] =3D "Vulnerable", + [RETBLEED_MITIGATION_UNRET] =3D "Mitigation: untrained return thunk", +}; + +static enum retbleed_mitigation retbleed_mitigation __ro_after_init =3D + RETBLEED_MITIGATION_NONE; +static enum retbleed_mitigation_cmd retbleed_cmd __ro_after_init =3D + RETBLEED_CMD_AUTO; + +static int __init retbleed_parse_cmdline(char *str) +{ + if (!str) + return -EINVAL; + + if (!strcmp(str, "off")) + retbleed_cmd =3D RETBLEED_CMD_OFF; + else if (!strcmp(str, "auto")) + retbleed_cmd =3D RETBLEED_CMD_AUTO; + else if (!strcmp(str, "unret")) + retbleed_cmd =3D RETBLEED_CMD_UNRET; + else + pr_err("Unknown retbleed option (%s). Defaulting to 'auto'\n", str); + + return 0; +} +early_param("retbleed", retbleed_parse_cmdline); + +#define RETBLEED_UNTRAIN_MSG "WARNING: BTB untrained return thunk mitigati= on is only effective on AMD/Hygon!\n" +#define RETBLEED_COMPILER_MSG "WARNING: kernel not compiled with RETPOLINE= or -mfunction-return capable compiler!\n" + +static void __init retbleed_select_mitigation(void) +{ + if (!boot_cpu_has_bug(X86_BUG_RETBLEED) || cpu_mitigations_off()) + return; + + switch (retbleed_cmd) { + case RETBLEED_CMD_OFF: + return; + + case RETBLEED_CMD_UNRET: + retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + break; + + case RETBLEED_CMD_AUTO: + default: + if (!boot_cpu_has_bug(X86_BUG_RETBLEED)) + break; + + if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_AMD || + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) + retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + break; + } + + switch (retbleed_mitigation) { + case RETBLEED_MITIGATION_UNRET: + + if (!IS_ENABLED(CONFIG_RETPOLINE) || + !IS_ENABLED(CONFIG_CC_HAS_RETURN_THUNK)) { + pr_err(RETBLEED_COMPILER_MSG); + retbleed_mitigation =3D RETBLEED_MITIGATION_NONE; + break; + } + + setup_force_cpu_cap(X86_FEATURE_RETHUNK); + setup_force_cpu_cap(X86_FEATURE_UNRET); + + if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && + boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) + pr_err(RETBLEED_UNTRAIN_MSG); + break; + + default: + break; + } + + pr_info("%s\n", retbleed_strings[retbleed_mitigation]); +} + +#undef pr_fmt #define pr_fmt(fmt) "Spectre V2 : " fmt =20 static enum spectre_v2_mitigation spectre_v2_enabled __ro_after_init =3D @@ -1919,7 +2020,12 @@ static ssize_t srbds_show_state(char *bu =20 static ssize_t retbleed_show_state(char *buf) { - return sprintf(buf, "Vulnerable\n"); + if (retbleed_mitigation =3D=3D RETBLEED_MITIGATION_UNRET && + (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && + boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON)) + return sprintf(buf, "Vulnerable: untrained return thunk on non-Zen uarch= \n"); + + return sprintf(buf, "%s\n", retbleed_strings[retbleed_mitigation]); } =20 static ssize_t cpu_show_common(struct device *dev, struct device_attribute= *attr, From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0136C433EF for ; Sat, 23 Jul 2022 10:07:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238703AbiGWKHT (ORCPT ); Sat, 23 Jul 2022 06:07:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238468AbiGWKGx (ORCPT ); Sat, 23 Jul 2022 06:06:53 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 752B4C1DF8; Sat, 23 Jul 2022 03:01:06 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2994B611CD; Sat, 23 Jul 2022 10:01:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 38962C341C0; Sat, 23 Jul 2022 10:01:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570464; bh=r1KYi8CpJ9PMhlMf7z/gAKIbGBUAcbhymfilRfURaEo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XJoNDOcLEL/l7cqg3wVpN+kDbmHAGLrBBHXyA/pat/dqvG/E87ZloIcm8A0DW4EB/ 69d9rNl1CMVErdpidJ206Y40eOfwxXldc+hCQJneDDai8HEp4Hw0+/WKZqoi9T63OO W+vRAq/nuUOV2MPGqC2hDSz4yzbtw6eZdKxrDKoI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Kim Phillips , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 098/148] x86/bugs: Enable STIBP for JMP2RET Date: Sat, 23 Jul 2022 11:55:10 +0200 Message-Id: <20220723095251.864209452@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Kim Phillips commit e8ec1b6e08a2102d8755ccb06fa26d540f26a2fa upstream. For untrained return thunks to be fully effective, STIBP must be enabled or SMT disabled. Co-developed-by: Josh Poimboeuf Signed-off-by: Josh Poimboeuf Signed-off-by: Kim Phillips Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Documentation/admin-guide/kernel-parameters.txt | 16 ++++-- arch/x86/kernel/cpu/bugs.c | 58 +++++++++++++++++++= ----- 2 files changed, 57 insertions(+), 17 deletions(-) --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4660,11 +4660,17 @@ Speculative Code Execution with Return Instructions) vulnerability. =20 - off - unconditionally disable - auto - automatically select a migitation - unret - force enable untrained return thunks, - only effective on AMD Zen {1,2} - based systems. + off - no mitigation + auto - automatically select a migitation + auto,nosmt - automatically select a mitigation, + disabling SMT if necessary for + the full mitigation (only on Zen1 + and older without STIBP). + unret - force enable untrained return thunks, + only effective on AMD f15h-f17h + based systems. + unret,nosmt - like unret, will disable SMT when STIBP + is not available. =20 Selecting 'auto' will choose a mitigation method at run time according to the CPU. --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -739,19 +739,34 @@ static enum retbleed_mitigation retbleed static enum retbleed_mitigation_cmd retbleed_cmd __ro_after_init =3D RETBLEED_CMD_AUTO; =20 +static int __ro_after_init retbleed_nosmt =3D false; + static int __init retbleed_parse_cmdline(char *str) { if (!str) return -EINVAL; =20 - if (!strcmp(str, "off")) - retbleed_cmd =3D RETBLEED_CMD_OFF; - else if (!strcmp(str, "auto")) - retbleed_cmd =3D RETBLEED_CMD_AUTO; - else if (!strcmp(str, "unret")) - retbleed_cmd =3D RETBLEED_CMD_UNRET; - else - pr_err("Unknown retbleed option (%s). Defaulting to 'auto'\n", str); + while (str) { + char *next =3D strchr(str, ','); + if (next) { + *next =3D 0; + next++; + } + + if (!strcmp(str, "off")) { + retbleed_cmd =3D RETBLEED_CMD_OFF; + } else if (!strcmp(str, "auto")) { + retbleed_cmd =3D RETBLEED_CMD_AUTO; + } else if (!strcmp(str, "unret")) { + retbleed_cmd =3D RETBLEED_CMD_UNRET; + } else if (!strcmp(str, "nosmt")) { + retbleed_nosmt =3D true; + } else { + pr_err("Ignoring unknown retbleed option (%s).", str); + } + + str =3D next; + } =20 return 0; } @@ -797,6 +812,10 @@ static void __init retbleed_select_mitig setup_force_cpu_cap(X86_FEATURE_RETHUNK); setup_force_cpu_cap(X86_FEATURE_UNRET); =20 + if (!boot_cpu_has(X86_FEATURE_STIBP) && + (retbleed_nosmt || cpu_mitigations_auto_nosmt())) + cpu_smt_disable(false); + if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) pr_err(RETBLEED_UNTRAIN_MSG); @@ -1043,6 +1062,13 @@ spectre_v2_user_select_mitigation(enum s boot_cpu_has(X86_FEATURE_AMD_STIBP_ALWAYS_ON)) mode =3D SPECTRE_V2_USER_STRICT_PREFERRED; =20 + if (retbleed_mitigation =3D=3D RETBLEED_MITIGATION_UNRET) { + if (mode !=3D SPECTRE_V2_USER_STRICT && + mode !=3D SPECTRE_V2_USER_STRICT_PREFERRED) + pr_info("Selecting STIBP always-on mode to complement retbleed mitigati= on'\n"); + mode =3D SPECTRE_V2_USER_STRICT_PREFERRED; + } + spectre_v2_user_stibp =3D mode; =20 set_mode: @@ -2020,10 +2046,18 @@ static ssize_t srbds_show_state(char *bu =20 static ssize_t retbleed_show_state(char *buf) { - if (retbleed_mitigation =3D=3D RETBLEED_MITIGATION_UNRET && - (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && - boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON)) - return sprintf(buf, "Vulnerable: untrained return thunk on non-Zen uarch= \n"); + if (retbleed_mitigation =3D=3D RETBLEED_MITIGATION_UNRET) { + if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && + boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) + return sprintf(buf, "Vulnerable: untrained return thunk on non-Zen u= arch\n"); + + return sprintf(buf, "%s; SMT %s\n", + retbleed_strings[retbleed_mitigation], + !sched_smt_active() ? "disabled" : + spectre_v2_user_stibp =3D=3D SPECTRE_V2_USER_STRICT || + spectre_v2_user_stibp =3D=3D SPECTRE_V2_USER_STRICT_PREFERRED ? + "enabled with STIBP protection" : "vulnerable"); + } =20 return sprintf(buf, "%s\n", retbleed_strings[retbleed_mitigation]); } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2F34C433EF for ; Sat, 23 Jul 2022 10:07:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238112AbiGWKHy (ORCPT ); Sat, 23 Jul 2022 06:07:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238407AbiGWKHD (ORCPT ); Sat, 23 Jul 2022 06:07:03 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A560D78596; Sat, 23 Jul 2022 03:01:16 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6E4C261212; Sat, 23 Jul 2022 10:01:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A6EBC341C0; Sat, 23 Jul 2022 10:01:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570475; bh=3SxuAmCWCYNcIuuFMjjQd8FxJhJUQ1ERaFp9yFZaXFk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SaQgJ4LazZOocYbdaRyCwyr8AtUt3oMck3VqmRYozMkTC8cltrhDjuiYUIg6PucMG 9D4mBvgwGfYWkn/vRS4SCw0CH5Xl0IyRyj64T13OSt2Ujs+Pb9YNlfQmRYT9yypre5 cAOmUXebOz7QaINx50IxeNhkCHbpiLJ+UaZ8AUC4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 099/148] x86/bugs: Keep a per-CPU IA32_SPEC_CTRL value Date: Sat, 23 Jul 2022 11:55:11 +0200 Message-Id: <20220723095252.138766172@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit caa0ff24d5d0e02abce5e65c3d2b7f20a6617be5 upstream. Due to TIF_SSBD and TIF_SPEC_IB the actual IA32_SPEC_CTRL value can differ from x86_spec_ctrl_base. As such, keep a per-CPU value reflecting the current task's MSR content. [jpoimboe: rename] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 1 + arch/x86/kernel/cpu/bugs.c | 28 +++++++++++++++++++++++----- arch/x86/kernel/process.c | 2 +- 3 files changed, 25 insertions(+), 6 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -254,6 +254,7 @@ static inline void indirect_branch_predi =20 /* The Intel SPEC CTRL MSR base value cache */ extern u64 x86_spec_ctrl_base; +extern void write_spec_ctrl_current(u64 val); =20 /* * With retpoline, we must use IBRS to restrict branch prediction --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -48,12 +48,30 @@ static void __init taa_select_mitigation static void __init mmio_select_mitigation(void); static void __init srbds_select_mitigation(void); =20 -/* The base value of the SPEC_CTRL MSR that always has to be preserved. */ +/* The base value of the SPEC_CTRL MSR without task-specific bits set */ u64 x86_spec_ctrl_base; EXPORT_SYMBOL_GPL(x86_spec_ctrl_base); + +/* The current value of the SPEC_CTRL MSR with task-specific bits set */ +DEFINE_PER_CPU(u64, x86_spec_ctrl_current); +EXPORT_SYMBOL_GPL(x86_spec_ctrl_current); + static DEFINE_MUTEX(spec_ctrl_mutex); =20 /* + * Keep track of the SPEC_CTRL MSR value for the current task, which may d= iffer + * from x86_spec_ctrl_base due to STIBP/SSB in __speculation_ctrl_update(). + */ +void write_spec_ctrl_current(u64 val) +{ + if (this_cpu_read(x86_spec_ctrl_current) =3D=3D val) + return; + + this_cpu_write(x86_spec_ctrl_current, val); + wrmsrl(MSR_IA32_SPEC_CTRL, val); +} + +/* * The vendor and possibly platform specific bits which can be modified in * x86_spec_ctrl_base. */ @@ -1235,7 +1253,7 @@ static void __init spectre_v2_select_mit if (spectre_v2_in_eibrs_mode(mode)) { /* Force it so VMEXIT will restore correctly */ x86_spec_ctrl_base |=3D SPEC_CTRL_IBRS; - wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base); } =20 switch (mode) { @@ -1290,7 +1308,7 @@ static void __init spectre_v2_select_mit =20 static void update_stibp_msr(void * __unused) { - wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base); } =20 /* Update x86_spec_ctrl_base in case SMT state changed. */ @@ -1533,7 +1551,7 @@ static enum ssb_mitigation __init __ssb_ x86_amd_ssb_disable(); } else { x86_spec_ctrl_base |=3D SPEC_CTRL_SSBD; - wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base); } } =20 @@ -1751,7 +1769,7 @@ int arch_prctl_spec_ctrl_get(struct task void x86_spec_ctrl_setup_ap(void) { if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL)) - wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base); =20 if (ssb_mode =3D=3D SPEC_STORE_BYPASS_DISABLE) x86_amd_ssb_disable(); --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -556,7 +556,7 @@ static __always_inline void __speculatio } =20 if (updmsr) - wrmsrl(MSR_IA32_SPEC_CTRL, msr); + write_spec_ctrl_current(msr); } =20 static unsigned long speculation_ctrl_update_tif(struct task_struct *tsk) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1025FCCA483 for ; Sat, 23 Jul 2022 10:08:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238689AbiGWKH6 (ORCPT ); Sat, 23 Jul 2022 06:07:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238465AbiGWKHH (ORCPT ); Sat, 23 Jul 2022 06:07:07 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C918C47E5; Sat, 23 Jul 2022 03:01:22 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E5FF7B82C1D; Sat, 23 Jul 2022 10:01:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 53224C341C0; Sat, 23 Jul 2022 10:01:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570478; bh=AlwIaBZ2g8sMPWPN0Jz1aONq8AbTKSIosVdoN6z6qbo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EhWGfyncJTRknglMvfTl9IpKjg7lYKMk7asW11g4OzZ0EGkrEnNr0fvDUXmYlczb3 DrRXO0gUEu/+rYJr04yLhQM1QIpDdvqUNbB0ZtGHVppeHiLP1aneHsCt6kEeQVE4JR sMIIKhMB2fk9757rR79BB66neDpH6s1EvQvZ2HBM= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 100/148] x86/entry: Add kernel IBRS implementation Date: Sat, 23 Jul 2022 11:55:12 +0200 Message-Id: <20220723095252.429231691@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 2dbb887e875b1de3ca8f40ddf26bcfe55798c609 upstream. Implement Kernel IBRS - currently the only known option to mitigate RSB underflow speculation issues on Skylake hardware. Note: since IBRS_ENTER requires fuller context established than UNTRAIN_RET, it must be placed after it. However, since UNTRAIN_RET itself implies a RET, it must come after IBRS_ENTER. This means IBRS_ENTER needs to also move UNTRAIN_RET. Note 2: KERNEL_IBRS is sub-optimal for XenPV. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: conflict at arch/x86/entry/entry_64.S, skip_r11rcx] [cascardo: conflict at arch/x86/entry/entry_64_compat.S] [cascardo: conflict fixups, no ANNOTATE_NOENDBR] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/calling.h | 58 ++++++++++++++++++++++++++++++++= +++++ arch/x86/entry/entry_64.S | 44 ++++++++++++++++++++++++---- arch/x86/entry/entry_64_compat.S | 17 ++++++++-- arch/x86/include/asm/cpufeatures.h | 2 - 4 files changed, 111 insertions(+), 10 deletions(-) --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -6,6 +6,8 @@ #include #include #include +#include +#include =20 /* =20 @@ -309,6 +311,62 @@ For 32-bit we have the following convent #endif =20 /* + * IBRS kernel mitigation for Spectre_v2. + * + * Assumes full context is established (PUSH_REGS, CR3 and GS) and it clob= bers + * the regs it uses (AX, CX, DX). Must be called before the first RET + * instruction (NOTE! UNTRAIN_RET includes a RET instruction) + * + * The optional argument is used to save/restore the current value, + * which is used on the paranoid paths. + * + * Assumes x86_spec_ctrl_{base,current} to have SPEC_CTRL_IBRS set. + */ +.macro IBRS_ENTER save_reg + ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_KERNEL_IBRS + movl $MSR_IA32_SPEC_CTRL, %ecx + +.ifnb \save_reg + rdmsr + shl $32, %rdx + or %rdx, %rax + mov %rax, \save_reg + test $SPEC_CTRL_IBRS, %eax + jz .Ldo_wrmsr_\@ + lfence + jmp .Lend_\@ +.Ldo_wrmsr_\@: +.endif + + movq PER_CPU_VAR(x86_spec_ctrl_current), %rdx + movl %edx, %eax + shr $32, %rdx + wrmsr +.Lend_\@: +.endm + +/* + * Similar to IBRS_ENTER, requires KERNEL GS,CR3 and clobbers (AX, CX, DX) + * regs. Must be called after the last RET. + */ +.macro IBRS_EXIT save_reg + ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_KERNEL_IBRS + movl $MSR_IA32_SPEC_CTRL, %ecx + +.ifnb \save_reg + mov \save_reg, %rdx +.else + movq PER_CPU_VAR(x86_spec_ctrl_current), %rdx + andl $(~SPEC_CTRL_IBRS), %edx +.endif + + movl %edx, %eax + shr $32, %rdx + wrmsr +.Lend_\@: +.endm + +/* * Mitigate Spectre v1 for conditional swapgs code paths. * * FENCE_SWAPGS_USER_ENTRY is used in the user entry swapgs code path, to --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -102,7 +102,6 @@ SYM_CODE_START(entry_SYSCALL_64) movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) - UNTRAIN_RET =20 /* Construct struct pt_regs on stack */ pushq $__USER_DS /* pt_regs->ss */ @@ -118,6 +117,11 @@ SYM_INNER_LABEL(entry_SYSCALL_64_after_h /* IRQs are off. */ movq %rax, %rdi movq %rsp, %rsi + + /* clobbers %rax, make sure it is after saving the syscall nr */ + IBRS_ENTER + UNTRAIN_RET + call do_syscall_64 /* returns with IRQs disabled */ =20 /* @@ -192,6 +196,7 @@ SYM_INNER_LABEL(entry_SYSCALL_64_after_h * perf profiles. Nothing jumps here. */ syscall_return_via_sysret: + IBRS_EXIT POP_REGS pop_rdi=3D0 =20 /* @@ -569,6 +574,7 @@ __irqentry_text_end: =20 SYM_CODE_START_LOCAL(common_interrupt_return) SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) + IBRS_EXIT #ifdef CONFIG_DEBUG_ENTRY /* Assert that pt_regs indicates user mode. */ testb $3, CS(%rsp) @@ -889,6 +895,9 @@ SYM_CODE_END(xen_failsafe_callback) * 1 -> no SWAPGS on exit * * Y GSBASE value at entry, must be restored in paranoid_exit + * + * R14 - old CR3 + * R15 - old SPEC_CTRL */ SYM_CODE_START_LOCAL(paranoid_entry) UNWIND_HINT_FUNC @@ -912,7 +921,6 @@ SYM_CODE_START_LOCAL(paranoid_entry) * be retrieved from a kernel internal table. */ SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax save_reg=3D%r14 - UNTRAIN_RET =20 /* * Handling GSBASE depends on the availability of FSGSBASE. @@ -934,7 +942,7 @@ SYM_CODE_START_LOCAL(paranoid_entry) * is needed here. */ SAVE_AND_SET_GSBASE scratch_reg=3D%rax save_reg=3D%rbx - RET + jmp .Lparanoid_gsbase_done =20 .Lparanoid_entry_checkgs: /* EBX =3D 1 -> kernel GSBASE active, no restore required */ @@ -953,8 +961,16 @@ SYM_CODE_START_LOCAL(paranoid_entry) xorl %ebx, %ebx swapgs .Lparanoid_kernel_gsbase: - FENCE_SWAPGS_KERNEL_ENTRY +.Lparanoid_gsbase_done: + + /* + * Once we have CR3 and %GS setup save and set SPEC_CTRL. Just like + * CR3 above, keep the old value in a callee saved register. + */ + IBRS_ENTER save_reg=3D%r15 + UNTRAIN_RET + RET SYM_CODE_END(paranoid_entry) =20 @@ -976,9 +992,19 @@ SYM_CODE_END(paranoid_entry) * 1 -> no SWAPGS on exit * * Y User space GSBASE, must be restored unconditionally + * + * R14 - old CR3 + * R15 - old SPEC_CTRL */ SYM_CODE_START_LOCAL(paranoid_exit) UNWIND_HINT_REGS + + /* + * Must restore IBRS state before both CR3 and %GS since we need access + * to the per-CPU x86_spec_ctrl_shadow variable. + */ + IBRS_EXIT save_reg=3D%r15 + /* * The order of operations is important. RESTORE_CR3 requires * kernel GSBASE. @@ -1025,9 +1051,11 @@ SYM_CODE_START_LOCAL(error_entry) FENCE_SWAPGS_USER_ENTRY /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax + IBRS_ENTER UNTRAIN_RET =20 .Lerror_entry_from_usermode_after_swapgs: + /* Put us onto the real thread stack. */ popq %r12 /* save return addr in %12 */ movq %rsp, %rdi /* arg0 =3D pt_regs pointer */ @@ -1081,6 +1109,7 @@ SYM_CODE_START_LOCAL(error_entry) SWAPGS FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax + IBRS_ENTER UNTRAIN_RET =20 /* @@ -1176,7 +1205,6 @@ SYM_CODE_START(asm_exc_nmi) movq %rsp, %rdx movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp UNWIND_HINT_IRET_REGS base=3D%rdx offset=3D8 - UNTRAIN_RET pushq 5*8(%rdx) /* pt_regs->ss */ pushq 4*8(%rdx) /* pt_regs->rsp */ pushq 3*8(%rdx) /* pt_regs->flags */ @@ -1187,6 +1215,9 @@ SYM_CODE_START(asm_exc_nmi) PUSH_AND_CLEAR_REGS rdx=3D(%rdx) ENCODE_FRAME_POINTER =20 + IBRS_ENTER + UNTRAIN_RET + /* * At this point we no longer need to worry about stack damage * due to nesting -- we're on the normal thread stack and we're @@ -1409,6 +1440,9 @@ end_repeat_nmi: movq $-1, %rsi call exc_nmi =20 + /* Always restore stashed SPEC_CTRL value (see paranoid_entry) */ + IBRS_EXIT save_reg=3D%r15 + /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=3D%r15 save_reg=3D%r14 =20 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -4,7 +4,6 @@ * * Copyright 2000-2002 Andi Kleen, SuSE Labs. */ -#include "calling.h" #include #include #include @@ -18,6 +17,8 @@ #include #include =20 +#include "calling.h" + .section .entry.text, "ax" =20 /* @@ -72,7 +73,6 @@ SYM_CODE_START(entry_SYSENTER_compat) pushq $__USER32_CS /* pt_regs->cs */ pushq $0 /* pt_regs->ip =3D 0 (placeholder) */ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) - UNTRAIN_RET =20 /* * User tracing code (ptrace or signal handlers) might assume that @@ -114,6 +114,9 @@ SYM_INNER_LABEL(entry_SYSENTER_compat_af =20 cld =20 + IBRS_ENTER + UNTRAIN_RET + /* * SYSENTER doesn't filter flags, so we need to clear NT and AC * ourselves. To save a few cycles, we can check whether @@ -213,7 +216,6 @@ SYM_CODE_START(entry_SYSCALL_compat) movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_compat_safe_stack, SYM_L_GLOBAL) - UNTRAIN_RET =20 /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ @@ -255,6 +257,9 @@ SYM_INNER_LABEL(entry_SYSCALL_compat_aft =20 UNWIND_HINT_REGS =20 + IBRS_ENTER + UNTRAIN_RET + movq %rsp, %rdi call do_fast_syscall_32 /* XEN PV guests always use IRET path */ @@ -269,6 +274,8 @@ sysret32_from_system_call: */ STACKLEAK_ERASE =20 + IBRS_EXIT + movq RBX(%rsp), %rbx /* pt_regs->rbx */ movq RBP(%rsp), %rbp /* pt_regs->rbp */ movq EFLAGS(%rsp), %r11 /* pt_regs->flags (in r11) */ @@ -380,7 +387,6 @@ SYM_CODE_START(entry_INT80_compat) pushq (%rdi) /* pt_regs->di */ .Lint80_keep_stack: =20 - UNTRAIN_RET pushq %rsi /* pt_regs->si */ xorl %esi, %esi /* nospec si */ pushq %rdx /* pt_regs->dx */ @@ -413,6 +419,9 @@ SYM_CODE_START(entry_INT80_compat) =20 cld =20 + IBRS_ENTER + UNTRAIN_RET + movq %rsp, %rdi call do_int80_syscall_32 jmp swapgs_restore_regs_and_return_to_usermode --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -203,7 +203,7 @@ #define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9) /* AMD ProcFeedbackInterface = */ #define X86_FEATURE_SME ( 7*32+10) /* AMD Secure Memory Encryption */ #define X86_FEATURE_PTI ( 7*32+11) /* Kernel Page Table Isolation enable= d */ -/* FREE! ( 7*32+12) */ +#define X86_FEATURE_KERNEL_IBRS ( 7*32+12) /* "" Set/clear IBRS on kernel= entry/exit */ /* FREE! ( 7*32+13) */ #define X86_FEATURE_INTEL_PPIN ( 7*32+14) /* Intel Processor Inventory Nu= mber */ #define X86_FEATURE_CDP_L2 ( 7*32+15) /* Code and Data Prioritization L2 = */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25E44CCA489 for ; Sat, 23 Jul 2022 10:08:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238807AbiGWKIB (ORCPT ); Sat, 23 Jul 2022 06:08:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238692AbiGWKHS (ORCPT ); Sat, 23 Jul 2022 06:07:18 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 873E8C47F9; Sat, 23 Jul 2022 03:01:24 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B9BD9B82C1F; Sat, 23 Jul 2022 10:01:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23DCCC341C0; Sat, 23 Jul 2022 10:01:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570481; bh=kcMSOh3DA5LA50BmAg8peRwkcHb2FZBw+YEVoL5xv/Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RfJd0RMxPwZ647VcF6MVqwHnL6E0KlE79pfIqqkVuW5iLxiy2MVd//FF71ZUV4bj5 w24WGvnZBScwgjt2qNuyXCqC4poA2kUNbkrRYUJ5yEV5P5/PsLVRH9BS3tcemVWW+t 9+ngstt07AL83mvknI4EVfXbhAya2luDT0n0MQKI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 101/148] x86/bugs: Optimize SPEC_CTRL MSR writes Date: Sat, 23 Jul 2022 11:55:13 +0200 Message-Id: <20220723095252.687980835@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit c779bc1a9002fa474175b80e72b85c9bf628abb0 upstream. When changing SPEC_CTRL for user control, the WRMSR can be delayed until return-to-user when KERNEL_IBRS has been enabled. This avoids an MSR write during context switch. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 2 +- arch/x86/kernel/cpu/bugs.c | 18 ++++++++++++------ arch/x86/kernel/process.c | 2 +- 3 files changed, 14 insertions(+), 8 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -254,7 +254,7 @@ static inline void indirect_branch_predi =20 /* The Intel SPEC CTRL MSR base value cache */ extern u64 x86_spec_ctrl_base; -extern void write_spec_ctrl_current(u64 val); +extern void write_spec_ctrl_current(u64 val, bool force); =20 /* * With retpoline, we must use IBRS to restrict branch prediction --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -62,13 +62,19 @@ static DEFINE_MUTEX(spec_ctrl_mutex); * Keep track of the SPEC_CTRL MSR value for the current task, which may d= iffer * from x86_spec_ctrl_base due to STIBP/SSB in __speculation_ctrl_update(). */ -void write_spec_ctrl_current(u64 val) +void write_spec_ctrl_current(u64 val, bool force) { if (this_cpu_read(x86_spec_ctrl_current) =3D=3D val) return; =20 this_cpu_write(x86_spec_ctrl_current, val); - wrmsrl(MSR_IA32_SPEC_CTRL, val); + + /* + * When KERNEL_IBRS this MSR is written on return-to-user, unless + * forced the update can be delayed until that time. + */ + if (force || !cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS)) + wrmsrl(MSR_IA32_SPEC_CTRL, val); } =20 /* @@ -1253,7 +1259,7 @@ static void __init spectre_v2_select_mit if (spectre_v2_in_eibrs_mode(mode)) { /* Force it so VMEXIT will restore correctly */ x86_spec_ctrl_base |=3D SPEC_CTRL_IBRS; - write_spec_ctrl_current(x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base, true); } =20 switch (mode) { @@ -1308,7 +1314,7 @@ static void __init spectre_v2_select_mit =20 static void update_stibp_msr(void * __unused) { - write_spec_ctrl_current(x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base, true); } =20 /* Update x86_spec_ctrl_base in case SMT state changed. */ @@ -1551,7 +1557,7 @@ static enum ssb_mitigation __init __ssb_ x86_amd_ssb_disable(); } else { x86_spec_ctrl_base |=3D SPEC_CTRL_SSBD; - write_spec_ctrl_current(x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base, true); } } =20 @@ -1769,7 +1775,7 @@ int arch_prctl_spec_ctrl_get(struct task void x86_spec_ctrl_setup_ap(void) { if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL)) - write_spec_ctrl_current(x86_spec_ctrl_base); + write_spec_ctrl_current(x86_spec_ctrl_base, true); =20 if (ssb_mode =3D=3D SPEC_STORE_BYPASS_DISABLE) x86_amd_ssb_disable(); --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -556,7 +556,7 @@ static __always_inline void __speculatio } =20 if (updmsr) - write_spec_ctrl_current(msr); + write_spec_ctrl_current(msr, false); } =20 static unsigned long speculation_ctrl_update_tif(struct task_struct *tsk) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CDF1C433EF for ; Sat, 23 Jul 2022 10:09:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238820AbiGWKJD (ORCPT ); Sat, 23 Jul 2022 06:09:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238800AbiGWKId (ORCPT ); Sat, 23 Jul 2022 06:08:33 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D014C7AF0; Sat, 23 Jul 2022 03:01:52 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EDA09611BD; Sat, 23 Jul 2022 10:01:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 070C5C341C7; Sat, 23 Jul 2022 10:01:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570484; bh=2pGZJtOLIxxXgJY9pcGVlt7LN9hTnd6QKofUs3FJ3cI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H1IpQPETw7mQutbpwli4w0G9mf3iMlhjYayBhaQYNbl1f+Md0G45dX2airc6mtERi D+PW9zx39Q9IPA3XgkB5Uv+4vAVhdMuyqhJ8+rQotoJ3xpdIJ2GozavGN/eVIdbWCn oSgLpvuCG4OXm7XaqNvIW0dS5IR7NwpcjzkrmszU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Pawan Gupta , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 102/148] x86/speculation: Add spectre_v2=ibrs option to support Kernel IBRS Date: Sat, 23 Jul 2022 11:55:14 +0200 Message-Id: <20220723095252.960679070@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Pawan Gupta commit 7c693f54c873691a4b7da05c7e0f74e67745d144 upstream. Extend spectre_v2=3D boot option with Kernel IBRS. [jpoimboe: no STIBP with IBRS] Signed-off-by: Pawan Gupta Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Documentation/admin-guide/kernel-parameters.txt | 1=20 arch/x86/include/asm/nospec-branch.h | 1=20 arch/x86/kernel/cpu/bugs.c | 66 ++++++++++++++++++-= ----- 3 files changed, 54 insertions(+), 14 deletions(-) --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5026,6 +5026,7 @@ eibrs - enhanced IBRS eibrs,retpoline - enhanced IBRS + Retpolines eibrs,lfence - enhanced IBRS + LFENCE + ibrs - use IBRS to protect kernel =20 Not specifying this option is equivalent to spectre_v2=3Dauto. --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -212,6 +212,7 @@ enum spectre_v2_mitigation { SPECTRE_V2_EIBRS, SPECTRE_V2_EIBRS_RETPOLINE, SPECTRE_V2_EIBRS_LFENCE, + SPECTRE_V2_IBRS, }; =20 /* The indirect branch speculation control variants */ --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -928,6 +928,7 @@ enum spectre_v2_mitigation_cmd { SPECTRE_V2_CMD_EIBRS, SPECTRE_V2_CMD_EIBRS_RETPOLINE, SPECTRE_V2_CMD_EIBRS_LFENCE, + SPECTRE_V2_CMD_IBRS, }; =20 enum spectre_v2_user_cmd { @@ -1000,11 +1001,12 @@ spectre_v2_parse_user_cmdline(enum spect return SPECTRE_V2_USER_CMD_AUTO; } =20 -static inline bool spectre_v2_in_eibrs_mode(enum spectre_v2_mitigation mod= e) +static inline bool spectre_v2_in_ibrs_mode(enum spectre_v2_mitigation mode) { - return (mode =3D=3D SPECTRE_V2_EIBRS || - mode =3D=3D SPECTRE_V2_EIBRS_RETPOLINE || - mode =3D=3D SPECTRE_V2_EIBRS_LFENCE); + return mode =3D=3D SPECTRE_V2_IBRS || + mode =3D=3D SPECTRE_V2_EIBRS || + mode =3D=3D SPECTRE_V2_EIBRS_RETPOLINE || + mode =3D=3D SPECTRE_V2_EIBRS_LFENCE; } =20 static void __init @@ -1069,12 +1071,12 @@ spectre_v2_user_select_mitigation(enum s } =20 /* - * If no STIBP, enhanced IBRS is enabled or SMT impossible, STIBP is not - * required. + * If no STIBP, IBRS or enhanced IBRS is enabled, or SMT impossible, + * STIBP is not required. */ if (!boot_cpu_has(X86_FEATURE_STIBP) || !smt_possible || - spectre_v2_in_eibrs_mode(spectre_v2_enabled)) + spectre_v2_in_ibrs_mode(spectre_v2_enabled)) return; =20 /* @@ -1106,6 +1108,7 @@ static const char * const spectre_v2_str [SPECTRE_V2_EIBRS] =3D "Mitigation: Enhanced IBRS", [SPECTRE_V2_EIBRS_LFENCE] =3D "Mitigation: Enhanced IBRS + LFENCE", [SPECTRE_V2_EIBRS_RETPOLINE] =3D "Mitigation: Enhanced IBRS + Retpolines= ", + [SPECTRE_V2_IBRS] =3D "Mitigation: IBRS", }; =20 static const struct { @@ -1123,6 +1126,7 @@ static const struct { { "eibrs,lfence", SPECTRE_V2_CMD_EIBRS_LFENCE, false }, { "eibrs,retpoline", SPECTRE_V2_CMD_EIBRS_RETPOLINE, false }, { "auto", SPECTRE_V2_CMD_AUTO, false }, + { "ibrs", SPECTRE_V2_CMD_IBRS, false }, }; =20 static void __init spec_v2_print_cond(const char *reason, bool secure) @@ -1185,6 +1189,24 @@ static enum spectre_v2_mitigation_cmd __ return SPECTRE_V2_CMD_AUTO; } =20 + if (cmd =3D=3D SPECTRE_V2_CMD_IBRS && boot_cpu_data.x86_vendor !=3D X86_V= ENDOR_INTEL) { + pr_err("%s selected but not Intel CPU. Switching to AUTO select\n", + mitigation_options[i].option); + return SPECTRE_V2_CMD_AUTO; + } + + if (cmd =3D=3D SPECTRE_V2_CMD_IBRS && !boot_cpu_has(X86_FEATURE_IBRS)) { + pr_err("%s selected but CPU doesn't have IBRS. Switching to AUTO select\= n", + mitigation_options[i].option); + return SPECTRE_V2_CMD_AUTO; + } + + if (cmd =3D=3D SPECTRE_V2_CMD_IBRS && boot_cpu_has(X86_FEATURE_XENPV)) { + pr_err("%s selected but running as XenPV guest. Switching to AUTO select= \n", + mitigation_options[i].option); + return SPECTRE_V2_CMD_AUTO; + } + spec_v2_print_cond(mitigation_options[i].option, mitigation_options[i].secure); return cmd; @@ -1224,6 +1246,14 @@ static void __init spectre_v2_select_mit break; } =20 + if (boot_cpu_has_bug(X86_BUG_RETBLEED) && + retbleed_cmd !=3D RETBLEED_CMD_OFF && + boot_cpu_has(X86_FEATURE_IBRS) && + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_INTEL) { + mode =3D SPECTRE_V2_IBRS; + break; + } + mode =3D spectre_v2_select_retpoline(); break; =20 @@ -1240,6 +1270,10 @@ static void __init spectre_v2_select_mit mode =3D spectre_v2_select_retpoline(); break; =20 + case SPECTRE_V2_CMD_IBRS: + mode =3D SPECTRE_V2_IBRS; + break; + case SPECTRE_V2_CMD_EIBRS: mode =3D SPECTRE_V2_EIBRS; break; @@ -1256,7 +1290,7 @@ static void __init spectre_v2_select_mit if (mode =3D=3D SPECTRE_V2_EIBRS && unprivileged_ebpf_enabled()) pr_err(SPECTRE_V2_EIBRS_EBPF_MSG); =20 - if (spectre_v2_in_eibrs_mode(mode)) { + if (spectre_v2_in_ibrs_mode(mode)) { /* Force it so VMEXIT will restore correctly */ x86_spec_ctrl_base |=3D SPEC_CTRL_IBRS; write_spec_ctrl_current(x86_spec_ctrl_base, true); @@ -1267,6 +1301,10 @@ static void __init spectre_v2_select_mit case SPECTRE_V2_EIBRS: break; =20 + case SPECTRE_V2_IBRS: + setup_force_cpu_cap(X86_FEATURE_KERNEL_IBRS); + break; + case SPECTRE_V2_LFENCE: case SPECTRE_V2_EIBRS_LFENCE: setup_force_cpu_cap(X86_FEATURE_RETPOLINE_LFENCE); @@ -1293,17 +1331,17 @@ static void __init spectre_v2_select_mit pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context switc= h\n"); =20 /* - * Retpoline means the kernel is safe because it has no indirect - * branches. Enhanced IBRS protects firmware too, so, enable restricted - * speculation around firmware calls only when Enhanced IBRS isn't - * supported. + * Retpoline protects the kernel, but doesn't protect firmware. IBRS + * and Enhanced IBRS protect firmware too, so enable IBRS around + * firmware calls only when IBRS / Enhanced IBRS aren't otherwise + * enabled. * * Use "mode" to check Enhanced IBRS instead of boot_cpu_has(), because * the user might select retpoline on the kernel command line and if * the CPU supports Enhanced IBRS, kernel might un-intentionally not * enable IBRS around firmware calls. */ - if (boot_cpu_has(X86_FEATURE_IBRS) && !spectre_v2_in_eibrs_mode(mode)) { + if (boot_cpu_has(X86_FEATURE_IBRS) && !spectre_v2_in_ibrs_mode(mode)) { setup_force_cpu_cap(X86_FEATURE_USE_IBRS_FW); pr_info("Enabling Restricted Speculation for firmware calls\n"); } @@ -2012,7 +2050,7 @@ static ssize_t mmio_stale_data_show_stat =20 static char *stibp_state(void) { - if (spectre_v2_in_eibrs_mode(spectre_v2_enabled)) + if (spectre_v2_in_ibrs_mode(spectre_v2_enabled)) return ""; =20 switch (spectre_v2_user_stibp) { From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCD6FC433EF for ; Sat, 23 Jul 2022 10:08:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238810AbiGWKIE (ORCPT ); Sat, 23 Jul 2022 06:08:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238702AbiGWKHT (ORCPT ); Sat, 23 Jul 2022 06:07:19 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED12FC5207; Sat, 23 Jul 2022 03:01:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D23C360C81; Sat, 23 Jul 2022 10:01:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DDB1AC341C7; Sat, 23 Jul 2022 10:01:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570487; bh=aLjl1sz+9TKvEHYRGFyleZovb2+KqWCsBPr2RrNcDeM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u3U0Xo/axxSRwzAs/+TeP8CQ5pot2Och/OctWNeQHrlGawFtE7fb3HF3kVUrWXeWN b2ZzQmBmnw6Wa/0MEBHIoVusYXjHTjb3TQCGTid1Cahl7a7aH9DRsCK3DwaYGXqvuF hnphVeXF9RdT6J1D0IkX2zyuvyiRRyJH9WqgPbEU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 103/148] x86/bugs: Split spectre_v2_select_mitigation() and spectre_v2_user_select_mitigation() Date: Sat, 23 Jul 2022 11:55:15 +0200 Message-Id: <20220723095253.224246000@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 166115c08a9b0b846b783088808a27d739be6e8d upstream. retbleed will depend on spectre_v2, while spectre_v2_user depends on retbleed. Break this cycle. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/bugs.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -37,8 +37,9 @@ #include "cpu.h" =20 static void __init spectre_v1_select_mitigation(void); -static void __init retbleed_select_mitigation(void); static void __init spectre_v2_select_mitigation(void); +static void __init retbleed_select_mitigation(void); +static void __init spectre_v2_user_select_mitigation(void); static void __init ssb_select_mitigation(void); static void __init l1tf_select_mitigation(void); static void __init mds_select_mitigation(void); @@ -137,13 +138,19 @@ void __init check_bugs(void) =20 /* Select the proper CPU mitigations before patching alternatives: */ spectre_v1_select_mitigation(); + spectre_v2_select_mitigation(); + /* + * retbleed_select_mitigation() relies on the state set by + * spectre_v2_select_mitigation(); specifically it wants to know about + * spectre_v2=3Dibrs. + */ retbleed_select_mitigation(); /* - * spectre_v2_select_mitigation() relies on the state set by + * spectre_v2_user_select_mitigation() relies on the state set by * retbleed_select_mitigation(); specifically the STIBP selection is * forced for UNRET. */ - spectre_v2_select_mitigation(); + spectre_v2_user_select_mitigation(); ssb_select_mitigation(); l1tf_select_mitigation(); md_clear_select_mitigation(); @@ -969,13 +976,15 @@ static void __init spec_v2_user_print_co pr_info("spectre_v2_user=3D%s forced on command line.\n", reason); } =20 +static __ro_after_init enum spectre_v2_mitigation_cmd spectre_v2_cmd; + static enum spectre_v2_user_cmd __init -spectre_v2_parse_user_cmdline(enum spectre_v2_mitigation_cmd v2_cmd) +spectre_v2_parse_user_cmdline(void) { char arg[20]; int ret, i; =20 - switch (v2_cmd) { + switch (spectre_v2_cmd) { case SPECTRE_V2_CMD_NONE: return SPECTRE_V2_USER_CMD_NONE; case SPECTRE_V2_CMD_FORCE: @@ -1010,7 +1019,7 @@ static inline bool spectre_v2_in_ibrs_mo } =20 static void __init -spectre_v2_user_select_mitigation(enum spectre_v2_mitigation_cmd v2_cmd) +spectre_v2_user_select_mitigation(void) { enum spectre_v2_user_mitigation mode =3D SPECTRE_V2_USER_NONE; bool smt_possible =3D IS_ENABLED(CONFIG_SMP); @@ -1023,7 +1032,7 @@ spectre_v2_user_select_mitigation(enum s cpu_smt_control =3D=3D CPU_SMT_NOT_SUPPORTED) smt_possible =3D false; =20 - cmd =3D spectre_v2_parse_user_cmdline(v2_cmd); + cmd =3D spectre_v2_parse_user_cmdline(); switch (cmd) { case SPECTRE_V2_USER_CMD_NONE: goto set_mode; @@ -1347,7 +1356,7 @@ static void __init spectre_v2_select_mit } =20 /* Set up IBPB and STIBP depending on the general spectre V2 command */ - spectre_v2_user_select_mitigation(cmd); + spectre_v2_cmd =3D cmd; } =20 static void update_stibp_msr(void * __unused) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B204BC433EF for ; Sat, 23 Jul 2022 10:10:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239123AbiGWKKV (ORCPT ); Sat, 23 Jul 2022 06:10:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234789AbiGWKI5 (ORCPT ); Sat, 23 Jul 2022 06:08:57 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C78AC7AE1; Sat, 23 Jul 2022 03:02:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5A90EB82C23; Sat, 23 Jul 2022 10:01:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A9BE6C341C0; Sat, 23 Jul 2022 10:01:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570490; bh=e/qiPrngS/1Wu1+7y5cBa/qZBIs933QZx4qBbSZgxJs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WbiBB4cpafbOtEGEpJh2ZKBElpYLQ1JqWdKeUhS64hPAcGYAo+3yJrgAeedu3YzAr 2m0ovrxvWXw5T9ouHeuz04nHEEOlHLhRHGPrC63v6RNU51Cr1BHwj5Hw32kVgA92yy JoyyL++CWckHkUIbdrFtzh8nmrGBQp3L2/OTxVro= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 104/148] x86/bugs: Report Intel retbleed vulnerability Date: Sat, 23 Jul 2022 11:55:16 +0200 Message-Id: <20220723095253.478179422@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 6ad0ad2bf8a67e27d1f9d006a1dabb0e1c360cc3 upstream. Skylake suffers from RSB underflow speculation issues; report this vulnerability and it's mitigation (spectre_v2=3Dibrs). [jpoimboe: cleanups, eibrs] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/bugs.c | 39 +++++++++++++++++++++++++++++++++-= ----- arch/x86/kernel/cpu/common.c | 24 ++++++++++++------------ 3 files changed, 46 insertions(+), 18 deletions(-) --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -91,6 +91,7 @@ #define MSR_IA32_ARCH_CAPABILITIES 0x0000010a #define ARCH_CAP_RDCL_NO BIT(0) /* Not susceptible to Meltdown */ #define ARCH_CAP_IBRS_ALL BIT(1) /* Enhanced IBRS support */ +#define ARCH_CAP_RSBA BIT(2) /* RET may use alternative branch predictor= s */ #define ARCH_CAP_SKIP_VMENTRY_L1DFLUSH BIT(3) /* Skip L1D flush on vmentry= */ #define ARCH_CAP_SSB_NO BIT(4) /* * Not susceptible to Speculative Store Bypass --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -746,12 +746,17 @@ static int __init nospectre_v1_cmdline(c } early_param("nospectre_v1", nospectre_v1_cmdline); =20 +static enum spectre_v2_mitigation spectre_v2_enabled __ro_after_init =3D + SPECTRE_V2_NONE; + #undef pr_fmt #define pr_fmt(fmt) "RETBleed: " fmt =20 enum retbleed_mitigation { RETBLEED_MITIGATION_NONE, RETBLEED_MITIGATION_UNRET, + RETBLEED_MITIGATION_IBRS, + RETBLEED_MITIGATION_EIBRS, }; =20 enum retbleed_mitigation_cmd { @@ -763,6 +768,8 @@ enum retbleed_mitigation_cmd { const char * const retbleed_strings[] =3D { [RETBLEED_MITIGATION_NONE] =3D "Vulnerable", [RETBLEED_MITIGATION_UNRET] =3D "Mitigation: untrained return thunk", + [RETBLEED_MITIGATION_IBRS] =3D "Mitigation: IBRS", + [RETBLEED_MITIGATION_EIBRS] =3D "Mitigation: Enhanced IBRS", }; =20 static enum retbleed_mitigation retbleed_mitigation __ro_after_init =3D @@ -805,6 +812,7 @@ early_param("retbleed", retbleed_parse_c =20 #define RETBLEED_UNTRAIN_MSG "WARNING: BTB untrained return thunk mitigati= on is only effective on AMD/Hygon!\n" #define RETBLEED_COMPILER_MSG "WARNING: kernel not compiled with RETPOLINE= or -mfunction-return capable compiler!\n" +#define RETBLEED_INTEL_MSG "WARNING: Spectre v2 mitigation leaves CPU vuln= erable to RETBleed attacks, data leaks possible!\n" =20 static void __init retbleed_select_mitigation(void) { @@ -821,12 +829,15 @@ static void __init retbleed_select_mitig =20 case RETBLEED_CMD_AUTO: default: - if (!boot_cpu_has_bug(X86_BUG_RETBLEED)) - break; - if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_AMD || boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + + /* + * The Intel mitigation (IBRS) was already selected in + * spectre_v2_select_mitigation(). + */ + break; } =20 @@ -856,15 +867,31 @@ static void __init retbleed_select_mitig break; } =20 + /* + * Let IBRS trump all on Intel without affecting the effects of the + * retbleed=3D cmdline option. + */ + if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_INTEL) { + switch (spectre_v2_enabled) { + case SPECTRE_V2_IBRS: + retbleed_mitigation =3D RETBLEED_MITIGATION_IBRS; + break; + case SPECTRE_V2_EIBRS: + case SPECTRE_V2_EIBRS_RETPOLINE: + case SPECTRE_V2_EIBRS_LFENCE: + retbleed_mitigation =3D RETBLEED_MITIGATION_EIBRS; + break; + default: + pr_err(RETBLEED_INTEL_MSG); + } + } + pr_info("%s\n", retbleed_strings[retbleed_mitigation]); } =20 #undef pr_fmt #define pr_fmt(fmt) "Spectre V2 : " fmt =20 -static enum spectre_v2_mitigation spectre_v2_enabled __ro_after_init =3D - SPECTRE_V2_NONE; - static enum spectre_v2_user_mitigation spectre_v2_user_stibp __ro_after_in= it =3D SPECTRE_V2_USER_NONE; static enum spectre_v2_user_mitigation spectre_v2_user_ibpb __ro_after_ini= t =3D --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1124,24 +1124,24 @@ static const struct x86_cpu_id cpu_vuln_ VULNBL_INTEL_STEPPINGS(BROADWELL_G, X86_STEPPING_ANY, SRBDS), VULNBL_INTEL_STEPPINGS(BROADWELL_X, X86_STEPPING_ANY, MMIO), VULNBL_INTEL_STEPPINGS(BROADWELL, X86_STEPPING_ANY, SRBDS), - VULNBL_INTEL_STEPPINGS(SKYLAKE_L, X86_STEPPINGS(0x3, 0x3), SRBDS | MMIO), + VULNBL_INTEL_STEPPINGS(SKYLAKE_L, X86_STEPPINGS(0x3, 0x3), SRBDS | MMIO |= RETBLEED), VULNBL_INTEL_STEPPINGS(SKYLAKE_L, X86_STEPPING_ANY, SRBDS), VULNBL_INTEL_STEPPINGS(SKYLAKE_X, BIT(3) | BIT(4) | BIT(6) | - BIT(7) | BIT(0xB), MMIO), - VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPINGS(0x3, 0x3), SRBDS | MMIO), + BIT(7) | BIT(0xB), MMIO | RETBLEED), + VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPINGS(0x3, 0x3), SRBDS | MMIO | = RETBLEED), VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPING_ANY, SRBDS), - VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPINGS(0x9, 0xC), SRBDS | MMIO), + VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPINGS(0x9, 0xC), SRBDS | MMIO = | RETBLEED), VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPINGS(0x0, 0x8), SRBDS), - VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPINGS(0x9, 0xD), SRBDS | MMIO), + VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPINGS(0x9, 0xD), SRBDS | MMIO | = RETBLEED), VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPINGS(0x0, 0x8), SRBDS), - VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPINGS(0x5, 0x5), MMIO | MMIO_SB= DS), + VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPINGS(0x5, 0x5), MMIO | MMIO_SB= DS | RETBLEED), VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPINGS(0x1, 0x1), MMIO), VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPINGS(0x4, 0x6), MMIO), - VULNBL_INTEL_STEPPINGS(COMETLAKE, BIT(2) | BIT(3) | BIT(5), MMIO | MMIO_S= BDS), - VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO_= SBDS), - VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x0, 0x0), MMIO), - VULNBL_INTEL_STEPPINGS(LAKEFIELD, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO_SB= DS), - VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPINGS(0x1, 0x1), MMIO), + VULNBL_INTEL_STEPPINGS(COMETLAKE, BIT(2) | BIT(3) | BIT(5), MMIO | MMIO_S= BDS | RETBLEED), + VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO_= SBDS | RETBLEED), + VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x0, 0x0), MMIO | RETBL= EED), + VULNBL_INTEL_STEPPINGS(LAKEFIELD, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO_SB= DS | RETBLEED), + VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPINGS(0x1, 0x1), MMIO | RETBLE= ED), VULNBL_INTEL_STEPPINGS(ATOM_TREMONT, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO= _SBDS), VULNBL_INTEL_STEPPINGS(ATOM_TREMONT_D, X86_STEPPING_ANY, MMIO), VULNBL_INTEL_STEPPINGS(ATOM_TREMONT_L, X86_STEPPINGS(0x0, 0x0), MMIO | MM= IO_SBDS), @@ -1251,7 +1251,7 @@ static void __init cpu_set_bug_bits(stru !arch_cap_mmio_immune(ia32_cap)) setup_force_cpu_bug(X86_BUG_MMIO_STALE_DATA); =20 - if (cpu_matches(cpu_vuln_blacklist, RETBLEED)) + if ((cpu_matches(cpu_vuln_blacklist, RETBLEED) || (ia32_cap & ARCH_CAP_RS= BA))) setup_force_cpu_bug(X86_BUG_RETBLEED); =20 if (cpu_matches(cpu_vuln_whitelist, NO_MELTDOWN)) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BD40C43334 for ; Sat, 23 Jul 2022 10:08:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238173AbiGWKIU (ORCPT ); Sat, 23 Jul 2022 06:08:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238748AbiGWKHv (ORCPT ); Sat, 23 Jul 2022 06:07:51 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86013C5234; Sat, 23 Jul 2022 03:01:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6DDB361272; Sat, 23 Jul 2022 10:01:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7AF5BC341C0; Sat, 23 Jul 2022 10:01:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570492; bh=M1JAbX+riJW6E/VEEoTjpyBBCJ7lGmUIn2OY4L/SFy8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=di9W/rQripDU9eKZYIwauxxxn2PM9Nusc3UO0cULqob2SDwseZXY75iZN3bW7ICXb JDT9TawDnMijbiPFYKYra7DfId1vzMhSs6dBunOUjLa6kuxwS3r6yX/f+WGjaKxa20 Csiz7IYOyr+QYXQ7d5lUU4NyUwSj8pjY6thGF/hU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Tim Chen , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 105/148] intel_idle: Disable IBRS during long idle Date: Sat, 23 Jul 2022 11:55:17 +0200 Message-Id: <20220723095253.758095406@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit bf5835bcdb9635c97f85120dba9bfa21e111130f upstream. Having IBRS enabled while the SMT sibling is idle unnecessarily slows down the running sibling. OTOH, disabling IBRS around idle takes two MSR writes, which will increase the idle latency. Therefore, only disable IBRS around deeper idle states. Shallow idle states are bounded by the tick in duration, since NOHZ is not allowed for them by virtue of their short target residency. Only do this for mwait-driven idle, since that keeps interrupts disabled across idle, which makes disabling IBRS vs IRQ-entry a non-issue. Note: C6 is a random threshold, most importantly C1 probably shouldn't disable IBRS, benchmarking needed. Suggested-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: no CPUIDLE_FLAG_IRQ_ENABLE] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 1=20 arch/x86/kernel/cpu/bugs.c | 6 ++++ drivers/idle/intel_idle.c | 43 ++++++++++++++++++++++++++++++= ----- 3 files changed, 44 insertions(+), 6 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -256,6 +256,7 @@ static inline void indirect_branch_predi /* The Intel SPEC CTRL MSR base value cache */ extern u64 x86_spec_ctrl_base; extern void write_spec_ctrl_current(u64 val, bool force); +extern u64 spec_ctrl_current(void); =20 /* * With retpoline, we must use IBRS to restrict branch prediction --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -78,6 +78,12 @@ void write_spec_ctrl_current(u64 val, bo wrmsrl(MSR_IA32_SPEC_CTRL, val); } =20 +u64 spec_ctrl_current(void) +{ + return this_cpu_read(x86_spec_ctrl_current); +} +EXPORT_SYMBOL_GPL(spec_ctrl_current); + /* * The vendor and possibly platform specific bits which can be modified in * x86_spec_ctrl_base. --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -47,11 +47,13 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include =20 @@ -94,6 +96,12 @@ static unsigned int mwait_substates __in #define CPUIDLE_FLAG_ALWAYS_ENABLE BIT(15) =20 /* + * Disable IBRS across idle (when KERNEL_IBRS), is exclusive vs IRQ_ENABLE + * above. + */ +#define CPUIDLE_FLAG_IBRS BIT(16) + +/* * MWAIT takes an 8-bit "hint" in EAX "suggesting" * the C-state (top nibble) and sub-state (bottom nibble) * 0x00 means "MWAIT(C1)", 0x10 means "MWAIT(C2)" etc. @@ -132,6 +140,24 @@ static __cpuidle int intel_idle(struct c return index; } =20 +static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) +{ + bool smt_active =3D sched_smt_active(); + u64 spec_ctrl =3D spec_ctrl_current(); + int ret; + + if (smt_active) + wrmsrl(MSR_IA32_SPEC_CTRL, 0); + + ret =3D intel_idle(dev, drv, index); + + if (smt_active) + wrmsrl(MSR_IA32_SPEC_CTRL, spec_ctrl); + + return ret; +} + /** * intel_idle_s2idle - Ask the processor to enter the given idle state. * @dev: cpuidle device of the target CPU. @@ -653,7 +679,7 @@ static struct cpuidle_state skl_cstates[ { .name =3D "C6", .desc =3D "MWAIT 0x20", - .flags =3D MWAIT2flg(0x20) | CPUIDLE_FLAG_TLB_FLUSHED, + .flags =3D MWAIT2flg(0x20) | CPUIDLE_FLAG_TLB_FLUSHED | CPUIDLE_FLAG_IBR= S, .exit_latency =3D 85, .target_residency =3D 200, .enter =3D &intel_idle, @@ -661,7 +687,7 @@ static struct cpuidle_state skl_cstates[ { .name =3D "C7s", .desc =3D "MWAIT 0x33", - .flags =3D MWAIT2flg(0x33) | CPUIDLE_FLAG_TLB_FLUSHED, + .flags =3D MWAIT2flg(0x33) | CPUIDLE_FLAG_TLB_FLUSHED | CPUIDLE_FLAG_IBR= S, .exit_latency =3D 124, .target_residency =3D 800, .enter =3D &intel_idle, @@ -669,7 +695,7 @@ static struct cpuidle_state skl_cstates[ { .name =3D "C8", .desc =3D "MWAIT 0x40", - .flags =3D MWAIT2flg(0x40) | CPUIDLE_FLAG_TLB_FLUSHED, + .flags =3D MWAIT2flg(0x40) | CPUIDLE_FLAG_TLB_FLUSHED | CPUIDLE_FLAG_IBR= S, .exit_latency =3D 200, .target_residency =3D 800, .enter =3D &intel_idle, @@ -677,7 +703,7 @@ static struct cpuidle_state skl_cstates[ { .name =3D "C9", .desc =3D "MWAIT 0x50", - .flags =3D MWAIT2flg(0x50) | CPUIDLE_FLAG_TLB_FLUSHED, + .flags =3D MWAIT2flg(0x50) | CPUIDLE_FLAG_TLB_FLUSHED | CPUIDLE_FLAG_IBR= S, .exit_latency =3D 480, .target_residency =3D 5000, .enter =3D &intel_idle, @@ -685,7 +711,7 @@ static struct cpuidle_state skl_cstates[ { .name =3D "C10", .desc =3D "MWAIT 0x60", - .flags =3D MWAIT2flg(0x60) | CPUIDLE_FLAG_TLB_FLUSHED, + .flags =3D MWAIT2flg(0x60) | CPUIDLE_FLAG_TLB_FLUSHED | CPUIDLE_FLAG_IBR= S, .exit_latency =3D 890, .target_residency =3D 5000, .enter =3D &intel_idle, @@ -714,7 +740,7 @@ static struct cpuidle_state skx_cstates[ { .name =3D "C6", .desc =3D "MWAIT 0x20", - .flags =3D MWAIT2flg(0x20) | CPUIDLE_FLAG_TLB_FLUSHED, + .flags =3D MWAIT2flg(0x20) | CPUIDLE_FLAG_TLB_FLUSHED | CPUIDLE_FLAG_IBR= S, .exit_latency =3D 133, .target_residency =3D 600, .enter =3D &intel_idle, @@ -1501,6 +1527,11 @@ static void __init intel_idle_init_cstat /* Structure copy. */ drv->states[drv->state_count] =3D cpuidle_state_table[cstate]; =20 + if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) && + cpuidle_state_table[cstate].flags & CPUIDLE_FLAG_IBRS) { + drv->states[drv->state_count].enter =3D intel_idle_ibrs; + } + if ((disabled_states_mask & BIT(drv->state_count)) || ((icpu->use_acpi || force_use_acpi) && intel_idle_off_by_default(mwait_hint) && From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CA6BC43334 for ; Sat, 23 Jul 2022 10:10:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239072AbiGWKKN (ORCPT ); Sat, 23 Jul 2022 06:10:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238748AbiGWKIu (ORCPT ); Sat, 23 Jul 2022 06:08:50 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96821C9E7A; Sat, 23 Jul 2022 03:01:59 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4312D6116A; Sat, 23 Jul 2022 10:01:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 526D7C341C0; Sat, 23 Jul 2022 10:01:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570495; bh=s9BBdXTRzWO6rilbhoZPvTfn8C7dwgczJt4XxU408Fs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N0vlzdrpULs29z6OTQr+6AENsrGvpG29z0KA0Q9NisG0a+m+wuAhRa1mI/asybxd8 WEMhVAOg57tkEENHeDyFsTeb8jDhlnkM4+sdbubVhsXAmgSzuWbJ16GXctX0BFfmSP XuOopOHyhlR4g1cvnHC/NmZp+oDGdEezAfHG1TKs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 106/148] objtool: Update Retpoline validation Date: Sat, 23 Jul 2022 11:55:18 +0200 Message-Id: <20220723095254.035382699@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 9bb2ec608a209018080ca262f771e6a9ff203b6f upstream. Update retpoline validation with the new CONFIG_RETPOLINE requirement of not having bare naked RET instructions. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: conflict fixup at arch/x86/xen/xen-head.S] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 6 ++++++ arch/x86/mm/mem_encrypt_boot.S | 2 ++ arch/x86/xen/xen-head.S | 1 + tools/objtool/check.c | 19 +++++++++++++------ 4 files changed, 22 insertions(+), 6 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -76,6 +76,12 @@ .endm =20 /* + * (ab)use RETPOLINE_SAFE on RET to annotate away 'bare' RET instructions + * vs RETBleed validation. + */ +#define ANNOTATE_UNRET_SAFE ANNOTATE_RETPOLINE_SAFE + +/* * JMP_NOSPEC and CALL_NOSPEC macros can be used instead of a simple * indirect jmp/call which may be susceptible to the Spectre variant 2 * attack. --- a/arch/x86/mm/mem_encrypt_boot.S +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -66,6 +66,7 @@ SYM_FUNC_START(sme_encrypt_execute) pop %rbp =20 /* Offset to __x86_return_thunk would be wrong here */ + ANNOTATE_UNRET_SAFE ret int3 SYM_FUNC_END(sme_encrypt_execute) @@ -154,6 +155,7 @@ SYM_FUNC_START(__enc_copy) pop %r15 =20 /* Offset to __x86_return_thunk would be wrong here */ + ANNOTATE_UNRET_SAFE ret int3 .L__enc_copy_end: --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -70,6 +70,7 @@ SYM_CODE_START(hypercall_page) .rept (PAGE_SIZE / 32) UNWIND_HINT_FUNC .skip 31, 0x90 + ANNOTATE_UNRET_SAFE RET .endr =20 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1799,8 +1799,9 @@ static int read_retpoline_hints(struct o } =20 if (insn->type !=3D INSN_JUMP_DYNAMIC && - insn->type !=3D INSN_CALL_DYNAMIC) { - WARN_FUNC("retpoline_safe hint not an indirect jump/call", + insn->type !=3D INSN_CALL_DYNAMIC && + insn->type !=3D INSN_RETURN) { + WARN_FUNC("retpoline_safe hint not an indirect jump/call/ret", insn->sec, insn->offset); return -1; } @@ -3051,7 +3052,8 @@ static int validate_retpoline(struct obj =20 for_each_insn(file, insn) { if (insn->type !=3D INSN_JUMP_DYNAMIC && - insn->type !=3D INSN_CALL_DYNAMIC) + insn->type !=3D INSN_CALL_DYNAMIC && + insn->type !=3D INSN_RETURN) continue; =20 if (insn->retpoline_safe) @@ -3066,9 +3068,14 @@ static int validate_retpoline(struct obj if (!strcmp(insn->sec->name, ".init.text") && !module) continue; =20 - WARN_FUNC("indirect %s found in RETPOLINE build", - insn->sec, insn->offset, - insn->type =3D=3D INSN_JUMP_DYNAMIC ? "jump" : "call"); + if (insn->type =3D=3D INSN_RETURN) { + WARN_FUNC("'naked' return found in RETPOLINE build", + insn->sec, insn->offset); + } else { + WARN_FUNC("indirect %s found in RETPOLINE build", + insn->sec, insn->offset, + insn->type =3D=3D INSN_JUMP_DYNAMIC ? "jump" : "call"); + } =20 warnings++; } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5860C433EF for ; Sat, 23 Jul 2022 10:08:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238787AbiGWKI2 (ORCPT ); Sat, 23 Jul 2022 06:08:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238467AbiGWKHz (ORCPT ); Sat, 23 Jul 2022 06:07:55 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 521BCC5D44; Sat, 23 Jul 2022 03:01:36 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9324EB82B92; Sat, 23 Jul 2022 10:01:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 03581C341C0; Sat, 23 Jul 2022 10:01:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570467; bh=ay3LWH7pBox0dI17p/VUYfmLXWQBubpVZeIlarQLFx4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kmdSTK5LKt9L4et0jru278zv+SiwnM6s7fcIIetPWX0xeFRy58eo5qubSXVNOgE6v 1dGzh0aElSUEVCthRz1JC2L95wr4akXTot+l55jHiue/duZX1jHfzQro4ZZ91pOBrw vusk7OxLm+bgSNJZBAVuls75q4YSmmE5gBwruS8g= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 107/148] x86/xen: Rename SYS* entry points Date: Sat, 23 Jul 2022 11:55:19 +0200 Message-Id: <20220723095254.324728586@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit b75b7f8ef1148be1b9321ffc2f6c19238904b438 upstream. Native SYS{CALL,ENTER} entry points are called entry_SYS{CALL,ENTER}_{64,compat}, make sure the Xen versions are named consistently. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/xen/setup.c | 6 +++--- arch/x86/xen/xen-asm.S | 20 ++++++++++---------- arch/x86/xen/xen-ops.h | 6 +++--- 3 files changed, 16 insertions(+), 16 deletions(-) --- a/arch/x86/xen/setup.c +++ b/arch/x86/xen/setup.c @@ -922,7 +922,7 @@ void xen_enable_sysenter(void) if (!boot_cpu_has(sysenter_feature)) return; =20 - ret =3D register_callback(CALLBACKTYPE_sysenter, xen_sysenter_target); + ret =3D register_callback(CALLBACKTYPE_sysenter, xen_entry_SYSENTER_compa= t); if(ret !=3D 0) setup_clear_cpu_cap(sysenter_feature); } @@ -931,7 +931,7 @@ void xen_enable_syscall(void) { int ret; =20 - ret =3D register_callback(CALLBACKTYPE_syscall, xen_syscall_target); + ret =3D register_callback(CALLBACKTYPE_syscall, xen_entry_SYSCALL_64); if (ret !=3D 0) { printk(KERN_ERR "Failed to set syscall callback: %d\n", ret); /* Pretty fatal; 64-bit userspace has no other @@ -940,7 +940,7 @@ void xen_enable_syscall(void) =20 if (boot_cpu_has(X86_FEATURE_SYSCALL32)) { ret =3D register_callback(CALLBACKTYPE_syscall32, - xen_syscall32_target); + xen_entry_SYSCALL_compat); if (ret !=3D 0) setup_clear_cpu_cap(X86_FEATURE_SYSCALL32); } --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -276,7 +276,7 @@ SYM_CODE_END(xenpv_restore_regs_and_retu */ =20 /* Normal 64-bit system call target */ -SYM_CODE_START(xen_syscall_target) +SYM_CODE_START(xen_entry_SYSCALL_64) UNWIND_HINT_EMPTY popq %rcx popq %r11 @@ -290,12 +290,12 @@ SYM_CODE_START(xen_syscall_target) movq $__USER_CS, 1*8(%rsp) =20 jmp entry_SYSCALL_64_after_hwframe -SYM_CODE_END(xen_syscall_target) +SYM_CODE_END(xen_entry_SYSCALL_64) =20 #ifdef CONFIG_IA32_EMULATION =20 /* 32-bit compat syscall target */ -SYM_CODE_START(xen_syscall32_target) +SYM_CODE_START(xen_entry_SYSCALL_compat) UNWIND_HINT_EMPTY popq %rcx popq %r11 @@ -309,10 +309,10 @@ SYM_CODE_START(xen_syscall32_target) movq $__USER32_CS, 1*8(%rsp) =20 jmp entry_SYSCALL_compat_after_hwframe -SYM_CODE_END(xen_syscall32_target) +SYM_CODE_END(xen_entry_SYSCALL_compat) =20 /* 32-bit compat sysenter target */ -SYM_CODE_START(xen_sysenter_target) +SYM_CODE_START(xen_entry_SYSENTER_compat) UNWIND_HINT_EMPTY /* * NB: Xen is polite and clears TF from EFLAGS for us. This means @@ -330,18 +330,18 @@ SYM_CODE_START(xen_sysenter_target) movq $__USER32_CS, 1*8(%rsp) =20 jmp entry_SYSENTER_compat_after_hwframe -SYM_CODE_END(xen_sysenter_target) +SYM_CODE_END(xen_entry_SYSENTER_compat) =20 #else /* !CONFIG_IA32_EMULATION */ =20 -SYM_CODE_START(xen_syscall32_target) -SYM_CODE_START(xen_sysenter_target) +SYM_CODE_START(xen_entry_SYSCALL_compat) +SYM_CODE_START(xen_entry_SYSENTER_compat) UNWIND_HINT_EMPTY lea 16(%rsp), %rsp /* strip %rcx, %r11 */ mov $-ENOSYS, %rax pushq $0 jmp hypercall_iret -SYM_CODE_END(xen_sysenter_target) -SYM_CODE_END(xen_syscall32_target) +SYM_CODE_END(xen_entry_SYSENTER_compat) +SYM_CODE_END(xen_entry_SYSCALL_compat) =20 #endif /* CONFIG_IA32_EMULATION */ --- a/arch/x86/xen/xen-ops.h +++ b/arch/x86/xen/xen-ops.h @@ -10,10 +10,10 @@ /* These are code, but not functions. Defined in entry.S */ extern const char xen_failsafe_callback[]; =20 -void xen_sysenter_target(void); +void xen_entry_SYSENTER_compat(void); #ifdef CONFIG_X86_64 -void xen_syscall_target(void); -void xen_syscall32_target(void); +void xen_entry_SYSCALL_64(void); +void xen_entry_SYSCALL_compat(void); #endif =20 extern void *xen_initial_gdt; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A613C433EF for ; Sat, 23 Jul 2022 10:07:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238728AbiGWKH2 (ORCPT ); Sat, 23 Jul 2022 06:07:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238551AbiGWKG5 (ORCPT ); Sat, 23 Jul 2022 06:06:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FBE9BDA2D; Sat, 23 Jul 2022 03:01:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CF57860C81; Sat, 23 Jul 2022 10:01:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DB9E3C341C0; Sat, 23 Jul 2022 10:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570470; bh=KGe1zr18yHrc+hbFEc1yACHoLy38Zwqv/mteIlxott0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B8ykEVMccmqCJ42fR3FfZ4BS0Gx9L62fn877KMBQozq2/DXe8hdlggmqFigkU5RO6 7xfDU38HE9dVt9ipgu/eYGmpARHqMCA/KtYpKFuwtXwo+kOkUCMtKHjDc3ODanpouz GgSvRRGiVNTH8DvepBd9U+nvuS+v/CC/fiLwCAOI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Andrew Cooper , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 108/148] x86/bugs: Add retbleed=ibpb Date: Sat, 23 Jul 2022 11:55:20 +0200 Message-Id: <20220723095254.606583358@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 3ebc170068885b6fc7bedda6c667bb2c4d533159 upstream. jmp2ret mitigates the easy-to-attack case at relatively low overhead. It mitigates the long speculation windows after a mispredicted RET, but it does not mitigate the short speculation window from arbitrary instruction boundaries. On Zen2, there is a chicken bit which needs setting, which mitigates "arbitrary instruction boundaries" down to just "basic block boundaries". But there is no fix for the short speculation window on basic block boundaries, other than to flush the entire BTB to evict all attacker predictions. On the spectrum of "fast & blurry" -> "safe", there is (on top of STIBP or no-SMT): 1) Nothing System wide open 2) jmp2ret May stop a script kiddy 3) jmp2ret+chickenbit Raises the bar rather further 4) IBPB Only thing which can count as "safe". Tentative numbers put IBPB-on-entry at a 2.5x hit on Zen2, and a 10x hit on Zen1 according to lmbench. [ bp: Fixup feature bit comments, document option, 32-bit build fix. ] Suggested-by: Andrew Cooper Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Documentation/admin-guide/kernel-parameters.txt | 3 + arch/x86/entry/Makefile | 2 - arch/x86/entry/entry.S | 22 ++++++++++++ arch/x86/include/asm/cpufeatures.h | 2 - arch/x86/include/asm/nospec-branch.h | 8 +++- arch/x86/kernel/cpu/bugs.c | 43 ++++++++++++++++++-= ----- 6 files changed, 67 insertions(+), 13 deletions(-) create mode 100644 arch/x86/entry/entry.S --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4666,6 +4666,9 @@ disabling SMT if necessary for the full mitigation (only on Zen1 and older without STIBP). + ibpb - mitigate short speculation windows on + basic block boundaries too. Safe, highest + perf impact. unret - force enable untrained return thunks, only effective on AMD f15h-f17h based systems. --- a/arch/x86/entry/Makefile +++ b/arch/x86/entry/Makefile @@ -21,7 +21,7 @@ CFLAGS_syscall_64.o +=3D $(call cc-option CFLAGS_syscall_32.o +=3D $(call cc-option,-Wno-override-init,) CFLAGS_syscall_x32.o +=3D $(call cc-option,-Wno-override-init,) =20 -obj-y :=3D entry_$(BITS).o thunk_$(BITS).o syscall_$(BITS).o +obj-y :=3D entry.o entry_$(BITS).o thunk_$(BITS).o syscall_$(BITS).o obj-y +=3D common.o =20 obj-y +=3D vdso/ --- /dev/null +++ b/arch/x86/entry/entry.S @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Common place for both 32- and 64-bit entry routines. + */ + +#include +#include +#include + +.pushsection .noinstr.text, "ax" + +SYM_FUNC_START(entry_ibpb) + movl $MSR_IA32_PRED_CMD, %ecx + movl $PRED_CMD_IBPB, %eax + xorl %edx, %edx + wrmsr + RET +SYM_FUNC_END(entry_ibpb) +/* For KVM */ +EXPORT_SYMBOL_GPL(entry_ibpb); + +.popsection --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -292,7 +292,7 @@ #define X86_FEATURE_PER_THREAD_MBA (11*32+ 7) /* "" Per-thread Memory Band= width Allocation */ /* FREE! (11*32+ 8) */ /* FREE! (11*32+ 9) */ -/* FREE! (11*32+10) */ +#define X86_FEATURE_ENTRY_IBPB (11*32+10) /* "" Issue an IBPB on kernel e= ntry */ /* FREE! (11*32+11) */ #define X86_FEATURE_RETPOLINE (11*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ #define X86_FEATURE_RETPOLINE_LFENCE (11*32+13) /* "" Use LFENCE for Spect= re variant 2 */ --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -123,14 +123,17 @@ * return thunk isn't mapped into the userspace tables (then again, AMD * typically has NO_MELTDOWN). * - * Doesn't clobber any registers but does require a stable stack. + * While zen_untrain_ret() doesn't clobber anything but requires stack, + * entry_ibpb() will clobber AX, CX, DX. * * As such, this must be placed after every *SWITCH_TO_KERNEL_CR3 at a poi= nt * where we have a stack but before any RET instruction. */ .macro UNTRAIN_RET #ifdef CONFIG_RETPOLINE - ALTERNATIVE "", "call zen_untrain_ret", X86_FEATURE_UNRET + ALTERNATIVE_2 "", \ + "call zen_untrain_ret", X86_FEATURE_UNRET, \ + "call entry_ibpb", X86_FEATURE_ENTRY_IBPB #endif .endm =20 @@ -144,6 +147,7 @@ =20 extern void __x86_return_thunk(void); extern void zen_untrain_ret(void); +extern void entry_ibpb(void); =20 #ifdef CONFIG_RETPOLINE =20 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -761,6 +761,7 @@ static enum spectre_v2_mitigation spectr enum retbleed_mitigation { RETBLEED_MITIGATION_NONE, RETBLEED_MITIGATION_UNRET, + RETBLEED_MITIGATION_IBPB, RETBLEED_MITIGATION_IBRS, RETBLEED_MITIGATION_EIBRS, }; @@ -769,11 +770,13 @@ enum retbleed_mitigation_cmd { RETBLEED_CMD_OFF, RETBLEED_CMD_AUTO, RETBLEED_CMD_UNRET, + RETBLEED_CMD_IBPB, }; =20 const char * const retbleed_strings[] =3D { [RETBLEED_MITIGATION_NONE] =3D "Vulnerable", [RETBLEED_MITIGATION_UNRET] =3D "Mitigation: untrained return thunk", + [RETBLEED_MITIGATION_IBPB] =3D "Mitigation: IBPB", [RETBLEED_MITIGATION_IBRS] =3D "Mitigation: IBRS", [RETBLEED_MITIGATION_EIBRS] =3D "Mitigation: Enhanced IBRS", }; @@ -803,6 +806,8 @@ static int __init retbleed_parse_cmdline retbleed_cmd =3D RETBLEED_CMD_AUTO; } else if (!strcmp(str, "unret")) { retbleed_cmd =3D RETBLEED_CMD_UNRET; + } else if (!strcmp(str, "ibpb")) { + retbleed_cmd =3D RETBLEED_CMD_IBPB; } else if (!strcmp(str, "nosmt")) { retbleed_nosmt =3D true; } else { @@ -817,11 +822,13 @@ static int __init retbleed_parse_cmdline early_param("retbleed", retbleed_parse_cmdline); =20 #define RETBLEED_UNTRAIN_MSG "WARNING: BTB untrained return thunk mitigati= on is only effective on AMD/Hygon!\n" -#define RETBLEED_COMPILER_MSG "WARNING: kernel not compiled with RETPOLINE= or -mfunction-return capable compiler!\n" +#define RETBLEED_COMPILER_MSG "WARNING: kernel not compiled with RETPOLINE= or -mfunction-return capable compiler; falling back to IBPB!\n" #define RETBLEED_INTEL_MSG "WARNING: Spectre v2 mitigation leaves CPU vuln= erable to RETBleed attacks, data leaks possible!\n" =20 static void __init retbleed_select_mitigation(void) { + bool mitigate_smt =3D false; + if (!boot_cpu_has_bug(X86_BUG_RETBLEED) || cpu_mitigations_off()) return; =20 @@ -833,11 +840,21 @@ static void __init retbleed_select_mitig retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; break; =20 + case RETBLEED_CMD_IBPB: + retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; + break; + case RETBLEED_CMD_AUTO: default: if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_AMD || - boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) - retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) { + + if (IS_ENABLED(CONFIG_RETPOLINE) && + IS_ENABLED(CONFIG_CC_HAS_RETURN_THUNK)) + retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + else + retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; + } =20 /* * The Intel mitigation (IBRS) was already selected in @@ -853,26 +870,34 @@ static void __init retbleed_select_mitig if (!IS_ENABLED(CONFIG_RETPOLINE) || !IS_ENABLED(CONFIG_CC_HAS_RETURN_THUNK)) { pr_err(RETBLEED_COMPILER_MSG); - retbleed_mitigation =3D RETBLEED_MITIGATION_NONE; - break; + retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; + goto retbleed_force_ibpb; } =20 setup_force_cpu_cap(X86_FEATURE_RETHUNK); setup_force_cpu_cap(X86_FEATURE_UNRET); =20 - if (!boot_cpu_has(X86_FEATURE_STIBP) && - (retbleed_nosmt || cpu_mitigations_auto_nosmt())) - cpu_smt_disable(false); - if (boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD && boot_cpu_data.x86_vendor !=3D X86_VENDOR_HYGON) pr_err(RETBLEED_UNTRAIN_MSG); + + mitigate_smt =3D true; + break; + + case RETBLEED_MITIGATION_IBPB: +retbleed_force_ibpb: + setup_force_cpu_cap(X86_FEATURE_ENTRY_IBPB); + mitigate_smt =3D true; break; =20 default: break; } =20 + if (mitigate_smt && !boot_cpu_has(X86_FEATURE_STIBP) && + (retbleed_nosmt || cpu_mitigations_auto_nosmt())) + cpu_smt_disable(false); + /* * Let IBRS trump all on Intel without affecting the effects of the * retbleed=3D cmdline option. From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC422C43334 for ; Sat, 23 Jul 2022 10:08:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238755AbiGWKIi (ORCPT ); Sat, 23 Jul 2022 06:08:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238618AbiGWKIP (ORCPT ); Sat, 23 Jul 2022 06:08:15 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 285EC78205; Sat, 23 Jul 2022 03:01:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5A70FB82C1A; Sat, 23 Jul 2022 10:01:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B2AACC341C0; Sat, 23 Jul 2022 10:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570473; bh=E1G/sj44DQ1PBjU34Ar46hxoaFI0v+72xBumAP9AX4o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bdq6rxEAZpDeLvyl48wwWpumxsWIT2kfAMUdiNfjQVnMjXadHUUau9nijMxbuhkMR EaQd+zr21pZfmYO/ICwQGaBHL4ph7ZReVevolwsLAlhFgt/uF3/DQyvws35J4WTnU2 7jKmV5+cMSFHslU+LLDImWMK8M40BsqFVvBdq2Fs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 109/148] x86/bugs: Do IBPB fallback check only once Date: Sat, 23 Jul 2022 11:55:21 +0200 Message-Id: <20220723095254.884274073@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 0fe4aeea9c01baabecc8c3afc7889c809d939bc2 upstream. When booting with retbleed=3Dauto, if the kernel wasn't built with CONFIG_CC_HAS_RETURN_THUNK, the mitigation falls back to IBPB. Make sure a warning is printed in that case. The IBPB fallback check is done twice, but it really only needs to be done once. Signed-off-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/bugs.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -847,18 +847,13 @@ static void __init retbleed_select_mitig case RETBLEED_CMD_AUTO: default: if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_AMD || - boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) { - - if (IS_ENABLED(CONFIG_RETPOLINE) && - IS_ENABLED(CONFIG_CC_HAS_RETURN_THUNK)) - retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; - else - retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; - } + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) + retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; =20 /* - * The Intel mitigation (IBRS) was already selected in - * spectre_v2_select_mitigation(). + * The Intel mitigation (IBRS or eIBRS) was already selected in + * spectre_v2_select_mitigation(). 'retbleed_mitigation' will + * be set accordingly below. */ =20 break; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B57BC43334 for ; Sat, 23 Jul 2022 10:10:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238953AbiGWKKk (ORCPT ); Sat, 23 Jul 2022 06:10:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238960AbiGWKJJ (ORCPT ); Sat, 23 Jul 2022 06:09:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C6963F329; Sat, 23 Jul 2022 03:02:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D41FF61263; Sat, 23 Jul 2022 10:02:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DC755C341C0; Sat, 23 Jul 2022 10:02:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570536; bh=PR4qSsLfCebVv1MQRnxZGaYukimLYC4djFlgxVu5qwg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ExkolcrzIGz7JZfOwLsRe4FrnuWxd38XyZKuRPS3LfQYcoRBCZbUKZ3Pa/UhTaJCq RrfxTLspYrZjtmupCpiQjFerHyAipw4EPztQmRtSqlzj2oPRZukznW9LyQgeVs+vbR XxJVZXLI/rsGlh5RVL1C/DbdnE5PgDSsQgwfPutE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 110/148] objtool: Add entry UNRET validation Date: Sat, 23 Jul 2022 11:55:22 +0200 Message-Id: <20220723095255.234133949@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit a09a6e2399ba0595c3042b3164f3ca68a3cff33e upstream. Since entry asm is tricky, add a validation pass that ensures the retbleed mitigation has been done before the first actual RET instruction. Entry points are those that either have UNWIND_HINT_ENTRY, which acts as UNWIND_HINT_EMPTY but marks the instruction as an entry point, or those that have UWIND_HINT_IRET_REGS at +0. This is basically a variant of validate_branch() that is intra-function and it will simply follow all branches from marked entry points and ensures that all paths lead to ANNOTATE_UNRET_END. If a path hits RET or an indirection the path is a fail and will be reported. There are 3 ANNOTATE_UNRET_END instances: - UNTRAIN_RET itself - exception from-kernel; this path doesn't need UNTRAIN_RET - all early exceptions; these also don't need UNTRAIN_RET Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov [cascardo: arch/x86/entry/entry_64.S no pt_regs return at .Lerror_entry_don= e_lfence] [cascardo: tools/objtool/builtin-check.c no link option validation] [cascardo: tools/objtool/check.c opts.ibt is ibt] [cascardo: tools/objtool/include/objtool/builtin.h leave unret option as bo= ol, no struct opts] [cascardo: objtool is still called from scripts/link-vmlinux.sh] [cascardo: no IBT support] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: - In scripts/link-vmlinux.sh, use "test -n" instead of is_enabled - Adjust context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/entry_64.S | 3=20 arch/x86/entry/entry_64_compat.S | 6 - arch/x86/include/asm/nospec-branch.h | 12 ++ arch/x86/include/asm/unwind_hints.h | 4=20 arch/x86/kernel/head_64.S | 5 + arch/x86/xen/xen-asm.S | 10 +- include/linux/objtool.h | 3=20 scripts/link-vmlinux.sh | 3=20 tools/include/linux/objtool.h | 3=20 tools/objtool/builtin-check.c | 3=20 tools/objtool/builtin.h | 2=20 tools/objtool/check.c | 172 ++++++++++++++++++++++++++++++= ++++- tools/objtool/check.h | 6 + 13 files changed, 217 insertions(+), 15 deletions(-) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -93,7 +93,7 @@ SYM_CODE_END(native_usergs_sysret64) */ =20 SYM_CODE_START(entry_SYSCALL_64) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY =20 swapgs /* tss.sp2 is scratch space. */ @@ -1094,6 +1094,7 @@ SYM_CODE_START_LOCAL(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + ANNOTATE_UNRET_END RET =20 .Lbstep_iret: --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -49,7 +49,7 @@ * 0(%ebp) arg6 */ SYM_CODE_START(entry_SYSENTER_compat) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY /* Interrupts are off on entry. */ SWAPGS =20 @@ -202,7 +202,7 @@ SYM_CODE_END(entry_SYSENTER_compat) * 0(%esp) arg6 */ SYM_CODE_START(entry_SYSCALL_compat) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY /* Interrupts are off on entry. */ swapgs =20 @@ -349,7 +349,7 @@ SYM_CODE_END(entry_SYSCALL_compat) * ebp arg6 */ SYM_CODE_START(entry_INT80_compat) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY /* * Interrupts are off on entry. */ --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -82,6 +82,17 @@ #define ANNOTATE_UNRET_SAFE ANNOTATE_RETPOLINE_SAFE =20 /* + * Abuse ANNOTATE_RETPOLINE_SAFE on a NOP to indicate UNRET_END, should + * eventually turn into it's own annotation. + */ +.macro ANNOTATE_UNRET_END +#ifdef CONFIG_DEBUG_ENTRY + ANNOTATE_RETPOLINE_SAFE + nop +#endif +.endm + +/* * JMP_NOSPEC and CALL_NOSPEC macros can be used instead of a simple * indirect jmp/call which may be susceptible to the Spectre variant 2 * attack. @@ -131,6 +142,7 @@ */ .macro UNTRAIN_RET #ifdef CONFIG_RETPOLINE + ANNOTATE_UNRET_END ALTERNATIVE_2 "", \ "call zen_untrain_ret", X86_FEATURE_UNRET, \ "call entry_ibpb", X86_FEATURE_ENTRY_IBPB --- a/arch/x86/include/asm/unwind_hints.h +++ b/arch/x86/include/asm/unwind_hints.h @@ -11,6 +11,10 @@ UNWIND_HINT sp_reg=3DORC_REG_UNDEFINED type=3DUNWIND_HINT_TYPE_CALL end= =3D1 .endm =20 +.macro UNWIND_HINT_ENTRY + UNWIND_HINT sp_reg=3DORC_REG_UNDEFINED type=3DUNWIND_HINT_TYPE_ENTRY end= =3D1 +.endm + .macro UNWIND_HINT_REGS base=3D%rsp offset=3D0 indirect=3D0 extra=3D1 part= ial=3D0 .if \base =3D=3D %rsp .if \indirect --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -321,6 +321,8 @@ SYM_CODE_END(start_cpu0) SYM_CODE_START_NOALIGN(vc_boot_ghcb) UNWIND_HINT_IRET_REGS offset=3D8 =20 + ANNOTATE_UNRET_END + /* Build pt_regs */ PUSH_AND_CLEAR_REGS =20 @@ -378,6 +380,7 @@ SYM_CODE_START(early_idt_handler_array) SYM_CODE_END(early_idt_handler_array) =20 SYM_CODE_START_LOCAL(early_idt_handler_common) + ANNOTATE_UNRET_END /* * The stack is the hardware frame, an error code or zero, and the * vector number. @@ -424,6 +427,8 @@ SYM_CODE_END(early_idt_handler_common) SYM_CODE_START_NOALIGN(vc_no_ghcb) UNWIND_HINT_IRET_REGS offset=3D8 =20 + ANNOTATE_UNRET_END + /* Build pt_regs */ PUSH_AND_CLEAR_REGS =20 --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -148,7 +148,7 @@ SYM_FUNC_END(xen_read_cr2_direct); =20 .macro xen_pv_trap name SYM_CODE_START(xen_\name) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY pop %rcx pop %r11 jmp \name @@ -277,7 +277,7 @@ SYM_CODE_END(xenpv_restore_regs_and_retu =20 /* Normal 64-bit system call target */ SYM_CODE_START(xen_entry_SYSCALL_64) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY popq %rcx popq %r11 =20 @@ -296,7 +296,7 @@ SYM_CODE_END(xen_entry_SYSCALL_64) =20 /* 32-bit compat syscall target */ SYM_CODE_START(xen_entry_SYSCALL_compat) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY popq %rcx popq %r11 =20 @@ -313,7 +313,7 @@ SYM_CODE_END(xen_entry_SYSCALL_compat) =20 /* 32-bit compat sysenter target */ SYM_CODE_START(xen_entry_SYSENTER_compat) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY /* * NB: Xen is polite and clears TF from EFLAGS for us. This means * that we don't need to guard against single step exceptions here. @@ -336,7 +336,7 @@ SYM_CODE_END(xen_entry_SYSENTER_compat) =20 SYM_CODE_START(xen_entry_SYSCALL_compat) SYM_CODE_START(xen_entry_SYSENTER_compat) - UNWIND_HINT_EMPTY + UNWIND_HINT_ENTRY lea 16(%rsp), %rsp /* strip %rcx, %r11 */ mov $-ENOSYS, %rax pushq $0 --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -32,11 +32,14 @@ struct unwind_hint { * * UNWIND_HINT_FUNC: Generate the unwind metadata of a callable function. * Useful for code which doesn't have an ELF function annotation. + * + * UNWIND_HINT_ENTRY: machine entry without stack, SYSCALL/SYSENTER etc. */ #define UNWIND_HINT_TYPE_CALL 0 #define UNWIND_HINT_TYPE_REGS 1 #define UNWIND_HINT_TYPE_REGS_PARTIAL 2 #define UNWIND_HINT_TYPE_FUNC 3 +#define UNWIND_HINT_TYPE_ENTRY 4 =20 #ifdef CONFIG_STACK_VALIDATION =20 --- a/scripts/link-vmlinux.sh +++ b/scripts/link-vmlinux.sh @@ -65,6 +65,9 @@ objtool_link() =20 if [ -n "${CONFIG_VMLINUX_VALIDATION}" ]; then objtoolopt=3D"check" + if [ -n "${CONFIG_RETPOLINE}" ]; then + objtoolopt=3D"${objtoolopt} --unret" + fi if [ -z "${CONFIG_FRAME_POINTER}" ]; then objtoolopt=3D"${objtoolopt} --no-fp" fi --- a/tools/include/linux/objtool.h +++ b/tools/include/linux/objtool.h @@ -32,11 +32,14 @@ struct unwind_hint { * * UNWIND_HINT_FUNC: Generate the unwind metadata of a callable function. * Useful for code which doesn't have an ELF function annotation. + * + * UNWIND_HINT_ENTRY: machine entry without stack, SYSCALL/SYSENTER etc. */ #define UNWIND_HINT_TYPE_CALL 0 #define UNWIND_HINT_TYPE_REGS 1 #define UNWIND_HINT_TYPE_REGS_PARTIAL 2 #define UNWIND_HINT_TYPE_FUNC 3 +#define UNWIND_HINT_TYPE_ENTRY 4 =20 #ifdef CONFIG_STACK_VALIDATION =20 --- a/tools/objtool/builtin-check.c +++ b/tools/objtool/builtin-check.c @@ -19,7 +19,7 @@ #include "objtool.h" =20 bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, stats, - validate_dup, vmlinux, sls; + validate_dup, vmlinux, sls, unret; =20 static const char * const check_usage[] =3D { "objtool check [] file.o", @@ -30,6 +30,7 @@ const struct option check_options[] =3D { OPT_BOOLEAN('f', "no-fp", &no_fp, "Skip frame pointer validation"), OPT_BOOLEAN('u', "no-unreachable", &no_unreachable, "Skip 'unreachable in= struction' warnings"), OPT_BOOLEAN('r', "retpoline", &retpoline, "Validate retpoline assumptions= "), + OPT_BOOLEAN(0, "unret", &unret, "validate entry unret placement"), OPT_BOOLEAN('m', "module", &module, "Indicates the object will be part of= a kernel module"), OPT_BOOLEAN('b', "backtrace", &backtrace, "unwind on error"), OPT_BOOLEAN('a', "uaccess", &uaccess, "enable uaccess checking"), --- a/tools/objtool/builtin.h +++ b/tools/objtool/builtin.h @@ -9,7 +9,7 @@ =20 extern const struct option check_options[]; extern bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, = stats, - validate_dup, vmlinux, sls; + validate_dup, vmlinux, sls, unret; =20 extern int cmd_check(int argc, const char **argv); extern int cmd_orc(int argc, const char **argv); --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1752,6 +1752,19 @@ static int read_unwind_hints(struct objt =20 insn->hint =3D true; =20 + if (hint->type =3D=3D UNWIND_HINT_TYPE_REGS_PARTIAL) { + struct symbol *sym =3D find_symbol_by_offset(insn->sec, insn->offset); + + if (sym && sym->bind =3D=3D STB_GLOBAL) { + insn->entry =3D 1; + } + } + + if (hint->type =3D=3D UNWIND_HINT_TYPE_ENTRY) { + hint->type =3D UNWIND_HINT_TYPE_CALL; + insn->entry =3D 1; + } + if (hint->type =3D=3D UNWIND_HINT_TYPE_FUNC) { insn->cfi =3D &func_cfi; continue; @@ -1800,8 +1813,9 @@ static int read_retpoline_hints(struct o =20 if (insn->type !=3D INSN_JUMP_DYNAMIC && insn->type !=3D INSN_CALL_DYNAMIC && - insn->type !=3D INSN_RETURN) { - WARN_FUNC("retpoline_safe hint not an indirect jump/call/ret", + insn->type !=3D INSN_RETURN && + insn->type !=3D INSN_NOP) { + WARN_FUNC("retpoline_safe hint not an indirect jump/call/ret/nop", insn->sec, insn->offset); return -1; } @@ -2818,8 +2832,8 @@ static int validate_branch(struct objtoo return 1; } =20 - visited =3D 1 << state.uaccess; - if (insn->visited) { + visited =3D VISITED_BRANCH << state.uaccess; + if (insn->visited & VISITED_BRANCH_MASK) { if (!insn->hint && !insn_cfi_match(insn, &state.cfi)) return 1; =20 @@ -3045,6 +3059,145 @@ static int validate_unwind_hints(struct return warnings; } =20 +/* + * Validate rethunk entry constraint: must untrain RET before the first RE= T. + * + * Follow every branch (intra-function) and ensure ANNOTATE_UNRET_END comes + * before an actual RET instruction. + */ +static int validate_entry(struct objtool_file *file, struct instruction *i= nsn) +{ + struct instruction *next, *dest; + int ret, warnings =3D 0; + + for (;;) { + next =3D next_insn_to_validate(file, insn); + + if (insn->visited & VISITED_ENTRY) + return 0; + + insn->visited |=3D VISITED_ENTRY; + + if (!insn->ignore_alts && !list_empty(&insn->alts)) { + struct alternative *alt; + bool skip_orig =3D false; + + list_for_each_entry(alt, &insn->alts, list) { + if (alt->skip_orig) + skip_orig =3D true; + + ret =3D validate_entry(file, alt->insn); + if (ret) { + if (backtrace) + BT_FUNC("(alt)", insn); + return ret; + } + } + + if (skip_orig) + return 0; + } + + switch (insn->type) { + + case INSN_CALL_DYNAMIC: + case INSN_JUMP_DYNAMIC: + case INSN_JUMP_DYNAMIC_CONDITIONAL: + WARN_FUNC("early indirect call", insn->sec, insn->offset); + return 1; + + case INSN_JUMP_UNCONDITIONAL: + case INSN_JUMP_CONDITIONAL: + if (!is_sibling_call(insn)) { + if (!insn->jump_dest) { + WARN_FUNC("unresolved jump target after linking?!?", + insn->sec, insn->offset); + return -1; + } + ret =3D validate_entry(file, insn->jump_dest); + if (ret) { + if (backtrace) { + BT_FUNC("(branch%s)", insn, + insn->type =3D=3D INSN_JUMP_CONDITIONAL ? "-cond" : ""); + } + return ret; + } + + if (insn->type =3D=3D INSN_JUMP_UNCONDITIONAL) + return 0; + + break; + } + + /* fallthrough */ + case INSN_CALL: + dest =3D find_insn(file, insn->call_dest->sec, + insn->call_dest->offset); + if (!dest) { + WARN("Unresolved function after linking!?: %s", + insn->call_dest->name); + return -1; + } + + ret =3D validate_entry(file, dest); + if (ret) { + if (backtrace) + BT_FUNC("(call)", insn); + return ret; + } + /* + * If a call returns without error, it must have seen UNTRAIN_RET. + * Therefore any non-error return is a success. + */ + return 0; + + case INSN_RETURN: + WARN_FUNC("RET before UNTRAIN", insn->sec, insn->offset); + return 1; + + case INSN_NOP: + if (insn->retpoline_safe) + return 0; + break; + + default: + break; + } + + if (!next) { + WARN_FUNC("teh end!", insn->sec, insn->offset); + return -1; + } + insn =3D next; + } + + return warnings; +} + +/* + * Validate that all branches starting at 'insn->entry' encounter UNRET_END + * before RET. + */ +static int validate_unret(struct objtool_file *file) +{ + struct instruction *insn; + int ret, warnings =3D 0; + + for_each_insn(file, insn) { + if (!insn->entry) + continue; + + ret =3D validate_entry(file, insn); + if (ret < 0) { + WARN_FUNC("Failed UNRET validation", insn->sec, insn->offset); + return ret; + } + warnings +=3D ret; + } + + return warnings; +} + static int validate_retpoline(struct objtool_file *file) { struct instruction *insn; @@ -3312,6 +3465,17 @@ int check(struct objtool_file *file) goto out; warnings +=3D ret; =20 + if (unret) { + /* + * Must be after validate_branch() and friends, it plays + * further games with insn->visited. + */ + ret =3D validate_unret(file); + if (ret < 0) + return ret; + warnings +=3D ret; + } + if (!warnings) { ret =3D validate_reachable_instructions(file); if (ret < 0) --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -48,6 +48,7 @@ struct instruction { bool dead_end, ignore, ignore_alts; bool hint; bool retpoline_safe; + bool entry; s8 instr; u8 visited; struct alt_group *alt_group; @@ -62,6 +63,11 @@ struct instruction { struct cfi_state *cfi; }; =20 +#define VISITED_BRANCH 0x01 +#define VISITED_BRANCH_UACCESS 0x02 +#define VISITED_BRANCH_MASK 0x03 +#define VISITED_ENTRY 0x04 + static inline bool is_static_jump(struct instruction *insn) { return insn->type =3D=3D INSN_JUMP_CONDITIONAL || From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CF32C43334 for ; Sat, 23 Jul 2022 10:10:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238599AbiGWKKp (ORCPT ); Sat, 23 Jul 2022 06:10:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239020AbiGWKJN (ORCPT ); Sat, 23 Jul 2022 06:09:13 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85C0D64E3A; Sat, 23 Jul 2022 03:02:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7BAEAB82C24; Sat, 23 Jul 2022 10:01:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8815C341C0; Sat, 23 Jul 2022 10:01:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570501; bh=v6IcRe3m3zGjisnaOPEL/c8dPABEyZPbTJYzs5Yb47o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=v6XIXNMksBMGtXG4PZ1CQD2rWTnXc0zR+4BPClxyXwxf07mDdrSoklnvsA0GqaulM 0iCxE4MBwsGlDcHu3qM2Fu/8mCOm0AEhBhYdP5qoC30eMINLjW7nwt1KTjk9Q9c5hF PseEaF/O6kshUt9jfSoF/MW8g1ikR1NPYxKnNykc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Andrew Cooper , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 111/148] x86/cpu/amd: Add Spectral Chicken Date: Sat, 23 Jul 2022 11:55:23 +0200 Message-Id: <20220723095255.486902538@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit d7caac991feeef1b871ee6988fd2c9725df09039 upstream. Zen2 uarchs have an undocumented, unnamed, MSR that contains a chicken bit for some speculation behaviour. It needs setting. Note: very belatedly AMD released naming; it's now officially called MSR_AMD64_DE_CFG2 and MSR_AMD64_DE_CFG2_SUPPRESS_NOBR_PRED_BIT but shall remain the SPECTRAL CHICKEN. Suggested-by: Andrew Cooper Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/msr-index.h | 3 +++ arch/x86/kernel/cpu/amd.c | 23 ++++++++++++++++++++++- arch/x86/kernel/cpu/cpu.h | 2 ++ arch/x86/kernel/cpu/hygon.c | 6 ++++++ 4 files changed, 33 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -508,6 +508,9 @@ /* Fam 17h MSRs */ #define MSR_F17H_IRPERF 0xc00000e9 =20 +#define MSR_ZEN2_SPECTRAL_CHICKEN 0xc00110e3 +#define MSR_ZEN2_SPECTRAL_CHICKEN_BIT BIT_ULL(1) + /* Fam 16h MSRs */ #define MSR_F16H_L2I_PERF_CTL 0xc0010230 #define MSR_F16H_L2I_PERF_CTR 0xc0010231 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -914,6 +914,26 @@ static void init_amd_bd(struct cpuinfo_x clear_rdrand_cpuid_bit(c); } =20 +void init_spectral_chicken(struct cpuinfo_x86 *c) +{ + u64 value; + + /* + * On Zen2 we offer this chicken (bit) on the altar of Speculation. + * + * This suppresses speculation from the middle of a basic block, i.e. it + * suppresses non-branch predictions. + * + * We use STIBP as a heuristic to filter out Zen2 from the rest of F17H + */ + if (!cpu_has(c, X86_FEATURE_HYPERVISOR) && cpu_has(c, X86_FEATURE_AMD_STI= BP)) { + if (!rdmsrl_safe(MSR_ZEN2_SPECTRAL_CHICKEN, &value)) { + value |=3D MSR_ZEN2_SPECTRAL_CHICKEN_BIT; + wrmsrl_safe(MSR_ZEN2_SPECTRAL_CHICKEN, value); + } + } +} + static void init_amd_zn(struct cpuinfo_x86 *c) { set_cpu_cap(c, X86_FEATURE_ZEN); @@ -959,7 +979,8 @@ static void init_amd(struct cpuinfo_x86 case 0x12: init_amd_ln(c); break; case 0x15: init_amd_bd(c); break; case 0x16: init_amd_jg(c); break; - case 0x17: fallthrough; + case 0x17: init_spectral_chicken(c); + fallthrough; case 0x19: init_amd_zn(c); break; } =20 --- a/arch/x86/kernel/cpu/cpu.h +++ b/arch/x86/kernel/cpu/cpu.h @@ -60,6 +60,8 @@ extern void tsx_disable(void); static inline void tsx_init(void) { } #endif /* CONFIG_CPU_SUP_INTEL */ =20 +extern void init_spectral_chicken(struct cpuinfo_x86 *c); + extern void get_cpu_cap(struct cpuinfo_x86 *c); extern void get_cpu_address_sizes(struct cpuinfo_x86 *c); extern void cpu_detect_cache_sizes(struct cpuinfo_x86 *c); --- a/arch/x86/kernel/cpu/hygon.c +++ b/arch/x86/kernel/cpu/hygon.c @@ -318,6 +318,12 @@ static void init_hygon(struct cpuinfo_x8 /* get apicid instead of initial apic id from cpuid */ c->apicid =3D hard_smp_processor_id(); =20 + /* + * XXX someone from Hygon needs to confirm this DTRT + * + init_spectral_chicken(c); + */ + set_cpu_cap(c, X86_FEATURE_ZEN); set_cpu_cap(c, X86_FEATURE_CPB); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A2E1CCA483 for ; Sat, 23 Jul 2022 10:10:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238979AbiGWKKC (ORCPT ); Sat, 23 Jul 2022 06:10:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238616AbiGWKIg (ORCPT ); Sat, 23 Jul 2022 06:08:36 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E265BC9E6B; Sat, 23 Jul 2022 03:01:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 7C745CE0B68; Sat, 23 Jul 2022 10:01:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F4B5C341C7; Sat, 23 Jul 2022 10:01:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570512; bh=k0LU1Ux28pkftl2hmnXLNBpozpwfdBZccM5xFqr0Ccw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mQ5Ys1hGPih1bX7YIwjc475/ezh7LI31P1OqOce2ywiw14kJ6EXE6H9FwQc7L5ibm RQWaDUsQNG1QfvkwtgROllZQfXkuCV+NdohVud4kmzRzX9oaJTleofILjetr4NBUW4 l8Rd+8X7d54LXA7jzezQ/lYHpmfJc5Af2ie0N6Vo= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 112/148] x86/speculation: Fix RSB filling with CONFIG_RETPOLINE=n Date: Sat, 23 Jul 2022 11:55:24 +0200 Message-Id: <20220723095255.777578620@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit b2620facef4889fefcbf2e87284f34dcd4189bce upstream. If a kernel is built with CONFIG_RETPOLINE=3Dn, but the user still wants to mitigate Spectre v2 using IBRS or eIBRS, the RSB filling will be silently disabled. There's nothing retpoline-specific about RSB buffer filling. Remove the CONFIG_RETPOLINE guards around it. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/entry/entry_32.S | 2 -- arch/x86/entry/entry_64.S | 2 -- arch/x86/include/asm/nospec-branch.h | 2 -- 3 files changed, 6 deletions(-) --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -782,7 +782,6 @@ SYM_CODE_START(__switch_to_asm) movl %ebx, PER_CPU_VAR(stack_canary)+stack_canary_offset #endif =20 -#ifdef CONFIG_RETPOLINE /* * When switching from a shallower to a deeper call stack * the RSB may either underflow or use entries populated @@ -791,7 +790,6 @@ SYM_CODE_START(__switch_to_asm) * speculative execution to prevent attack. */ FILL_RETURN_BUFFER %ebx, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW -#endif =20 /* Restore flags or the incoming task to restore AC state. */ popfl --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -249,7 +249,6 @@ SYM_FUNC_START(__switch_to_asm) movq %rbx, PER_CPU_VAR(fixed_percpu_data) + stack_canary_offset #endif =20 -#ifdef CONFIG_RETPOLINE /* * When switching from a shallower to a deeper call stack * the RSB may either underflow or use entries populated @@ -258,7 +257,6 @@ SYM_FUNC_START(__switch_to_asm) * speculative execution to prevent attack. */ FILL_RETURN_BUFFER %r12, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_CTXSW -#endif =20 /* restore callee-saved registers */ popq %r15 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -122,11 +122,9 @@ * monstrosity above, manually. */ .macro FILL_RETURN_BUFFER reg:req nr:req ftr:req -#ifdef CONFIG_RETPOLINE ALTERNATIVE "jmp .Lskip_rsb_\@", "", \ftr __FILL_RETURN_BUFFER(\reg,\nr,%_ASM_SP) .Lskip_rsb_\@: -#endif .endm =20 /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4734EC433EF for ; Sat, 23 Jul 2022 10:11:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238642AbiGWKK7 (ORCPT ); Sat, 23 Jul 2022 06:10:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239051AbiGWKJO (ORCPT ); Sat, 23 Jul 2022 06:09:14 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEE2E7C1B5; Sat, 23 Jul 2022 03:02:22 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4A92361212; Sat, 23 Jul 2022 10:01:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 56433C341C0; Sat, 23 Jul 2022 10:01:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570515; bh=wNMpLzoGPFxhjiSNYi0HVZ8hNKaw3idO+yUyf00eJdQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Dmp8QKo3aabOsXf54FH1TrjeiuJP5vWZUWzluV8MCwqS169nz1c2auMiB3P9xlPBK 2ucp9CaYIUW+VZnYYSeFPYZxF6bX/se52iqeaX70WcmfaPyeHnTZp43t3oyBi5eS8y fhulXngVzHQTzKu8VqkgeOhUMF77Zvp778dc1Cmc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 113/148] x86/speculation: Fix firmware entry SPEC_CTRL handling Date: Sat, 23 Jul 2022 11:55:25 +0200 Message-Id: <20220723095256.025267059@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit e6aa13622ea8283cc699cac5d018cc40a2ba2010 upstream. The firmware entry code may accidentally clear STIBP or SSBD. Fix that. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -286,18 +286,16 @@ extern u64 spec_ctrl_current(void); */ #define firmware_restrict_branch_speculation_start() \ do { \ - u64 val =3D x86_spec_ctrl_base | SPEC_CTRL_IBRS; \ - \ preempt_disable(); \ - alternative_msr_write(MSR_IA32_SPEC_CTRL, val, \ + alternative_msr_write(MSR_IA32_SPEC_CTRL, \ + spec_ctrl_current() | SPEC_CTRL_IBRS, \ X86_FEATURE_USE_IBRS_FW); \ } while (0) =20 #define firmware_restrict_branch_speculation_end() \ do { \ - u64 val =3D x86_spec_ctrl_base; \ - \ - alternative_msr_write(MSR_IA32_SPEC_CTRL, val, \ + alternative_msr_write(MSR_IA32_SPEC_CTRL, \ + spec_ctrl_current(), \ X86_FEATURE_USE_IBRS_FW); \ preempt_enable(); \ } while (0) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24E88C43334 for ; Sat, 23 Jul 2022 10:11:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238773AbiGWKLF (ORCPT ); Sat, 23 Jul 2022 06:11:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239069AbiGWKJQ (ORCPT ); Sat, 23 Jul 2022 06:09:16 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22CE23ED67; Sat, 23 Jul 2022 03:02:25 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E4064B82C1B; Sat, 23 Jul 2022 10:01:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 57DFDC341CE; Sat, 23 Jul 2022 10:01:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570518; bh=p/gFMJFw+0KXPwGqJWjVQcM1S+G3M3UFlYzAwYKkkCQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C5qLNLiPdy8xSHewWmSzV6hgsN2EzsyhqOMyZp9lj2D2uYlDqTFJHbGz5mZzaSaxU HtXXXpHKhaB81kz8k5eh315f0kUtG1IBWv3mLMn6vxTfTSp8n4pYEkgOjIlEBj4M26 V35N2lq2v4t/FGfgb7uApDXTdhx8Oc+P9dohozGY= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 114/148] x86/speculation: Fix SPEC_CTRL write on SMT state change Date: Sat, 23 Jul 2022 11:55:26 +0200 Message-Id: <20220723095256.314818325@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 56aa4d221f1ee2c3a49b45b800778ec6e0ab73c5 upstream. If the SMT state changes, SSBD might get accidentally disabled. Fix that. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/bugs.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1414,7 +1414,8 @@ static void __init spectre_v2_select_mit =20 static void update_stibp_msr(void * __unused) { - write_spec_ctrl_current(x86_spec_ctrl_base, true); + u64 val =3D spec_ctrl_current() | (x86_spec_ctrl_base & SPEC_CTRL_STIBP); + write_spec_ctrl_current(val, true); } =20 /* Update x86_spec_ctrl_base in case SMT state changed. */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C929C43334 for ; Sat, 23 Jul 2022 10:11:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238793AbiGWKLP (ORCPT ); Sat, 23 Jul 2022 06:11:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239076AbiGWKJR (ORCPT ); Sat, 23 Jul 2022 06:09:17 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DBFDCB763; Sat, 23 Jul 2022 03:02:25 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C2BECB82C1A; Sat, 23 Jul 2022 10:02:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2BE01C341C0; Sat, 23 Jul 2022 10:02:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570521; bh=6XerrlduLrZI2GtrTstCZEHooVumelY789m2NBzA1Sw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=2O30ybCCTVsXEr+yTcPDMU3M+5gUJYwrTCPAKGTN1pjlwhBZ90awT6zlaFC2tih1q 71OAv7NFy4nhmrN3U5eRovMCFvd53uKh7gbtaVd58vNm6847p+OHa77TjZcrot2rBS 4ufhVv92gAiUQLV0mUP0LYY/xYDkWNTl986e4QO4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 115/148] x86/speculation: Use cached host SPEC_CTRL value for guest entry/exit Date: Sat, 23 Jul 2022 11:55:27 +0200 Message-Id: <20220723095256.580892226@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit bbb69e8bee1bd882784947095ffb2bfe0f7c9470 upstream. There's no need to recalculate the host value for every entry/exit. Just use the cached value in spec_ctrl_current(). Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/bugs.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -199,7 +199,7 @@ void __init check_bugs(void) void x86_virt_spec_ctrl(u64 guest_spec_ctrl, u64 guest_virt_spec_ctrl, bool set= guest) { - u64 msrval, guestval, hostval =3D x86_spec_ctrl_base; + u64 msrval, guestval, hostval =3D spec_ctrl_current(); struct thread_info *ti =3D current_thread_info(); =20 /* Is MSR_SPEC_CTRL implemented ? */ @@ -212,15 +212,6 @@ x86_virt_spec_ctrl(u64 guest_spec_ctrl, guestval =3D hostval & ~x86_spec_ctrl_mask; guestval |=3D guest_spec_ctrl & x86_spec_ctrl_mask; =20 - /* SSBD controlled in MSR_SPEC_CTRL */ - if (static_cpu_has(X86_FEATURE_SPEC_CTRL_SSBD) || - static_cpu_has(X86_FEATURE_AMD_SSBD)) - hostval |=3D ssbd_tif_to_spec_ctrl(ti->flags); - - /* Conditional STIBP enabled? */ - if (static_branch_unlikely(&switch_to_cond_stibp)) - hostval |=3D stibp_tif_to_spec_ctrl(ti->flags); - if (hostval !=3D guestval) { msrval =3D setguest ? guestval : hostval; wrmsrl(MSR_IA32_SPEC_CTRL, msrval); @@ -1353,7 +1344,6 @@ static void __init spectre_v2_select_mit pr_err(SPECTRE_V2_EIBRS_EBPF_MSG); =20 if (spectre_v2_in_ibrs_mode(mode)) { - /* Force it so VMEXIT will restore correctly */ x86_spec_ctrl_base |=3D SPEC_CTRL_IBRS; write_spec_ctrl_current(x86_spec_ctrl_base, true); } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F4B0C433EF for ; Sat, 23 Jul 2022 10:11:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239054AbiGWKLV (ORCPT ); Sat, 23 Jul 2022 06:11:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239103AbiGWKJS (ORCPT ); Sat, 23 Jul 2022 06:09:18 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E2A5CB777; Sat, 23 Jul 2022 03:02:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id ABBBCB82B92; Sat, 23 Jul 2022 10:02:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0DE55C341C0; Sat, 23 Jul 2022 10:02:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570524; bh=rn3X5p8IfphIAOw9YW6Id6cWDmF/8gFuGDqC30KUOfA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iCIIWCRi8HNzYifGSyG4VkJGg0/3lNmlmKNaMaVX1RdDK2mxR5SWhEA52FQ3jcrWy 6aqNgU7NXYLC8rlUotClsstfZ7s6BBHzrfcN5cb+vrOMK6vpNRBB3xeEXRrTT5DRUE 64WieqcgsFlPOeDRfAizZEHjWA2vrpwbvsJ2AheQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , Borislav Petkov , Paolo Bonzini , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 116/148] x86/speculation: Remove x86_spec_ctrl_mask Date: Sat, 23 Jul 2022 11:55:28 +0200 Message-Id: <20220723095256.820899670@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit acac5e98ef8d638a411cfa2ee676c87e1973f126 upstream. This mask has been made redundant by kvm_spec_ctrl_test_value(). And it doesn't even work when MSR interception is disabled, as the guest can just write to SPEC_CTRL directly. Signed-off-by: Josh Poimboeuf Signed-off-by: Borislav Petkov Reviewed-by: Paolo Bonzini Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/bugs.c | 31 +------------------------------ 1 file changed, 1 insertion(+), 30 deletions(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -85,12 +85,6 @@ u64 spec_ctrl_current(void) EXPORT_SYMBOL_GPL(spec_ctrl_current); =20 /* - * The vendor and possibly platform specific bits which can be modified in - * x86_spec_ctrl_base. - */ -static u64 __ro_after_init x86_spec_ctrl_mask =3D SPEC_CTRL_IBRS; - -/* * AMD specific MSR info for Speculative Store Bypass control. * x86_amd_ls_cfg_ssbd_mask is initialized in identify_boot_cpu(). */ @@ -138,10 +132,6 @@ void __init check_bugs(void) if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL)) rdmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base); =20 - /* Allow STIBP in MSR_SPEC_CTRL if supported */ - if (boot_cpu_has(X86_FEATURE_STIBP)) - x86_spec_ctrl_mask |=3D SPEC_CTRL_STIBP; - /* Select the proper CPU mitigations before patching alternatives: */ spectre_v1_select_mitigation(); spectre_v2_select_mitigation(); @@ -199,19 +189,10 @@ void __init check_bugs(void) void x86_virt_spec_ctrl(u64 guest_spec_ctrl, u64 guest_virt_spec_ctrl, bool set= guest) { - u64 msrval, guestval, hostval =3D spec_ctrl_current(); + u64 msrval, guestval =3D guest_spec_ctrl, hostval =3D spec_ctrl_current(); struct thread_info *ti =3D current_thread_info(); =20 - /* Is MSR_SPEC_CTRL implemented ? */ if (static_cpu_has(X86_FEATURE_MSR_SPEC_CTRL)) { - /* - * Restrict guest_spec_ctrl to supported values. Clear the - * modifiable bits in the host base value and or the - * modifiable bits from the guest value. - */ - guestval =3D hostval & ~x86_spec_ctrl_mask; - guestval |=3D guest_spec_ctrl & x86_spec_ctrl_mask; - if (hostval !=3D guestval) { msrval =3D setguest ? guestval : hostval; wrmsrl(MSR_IA32_SPEC_CTRL, msrval); @@ -1622,16 +1603,6 @@ static enum ssb_mitigation __init __ssb_ } =20 /* - * If SSBD is controlled by the SPEC_CTRL MSR, then set the proper - * bit in the mask to allow guests to use the mitigation even in the - * case where the host does not enable it. - */ - if (static_cpu_has(X86_FEATURE_SPEC_CTRL_SSBD) || - static_cpu_has(X86_FEATURE_AMD_SSBD)) { - x86_spec_ctrl_mask |=3D SPEC_CTRL_SSBD; - } - - /* * We have three CPU feature flags that are in play here: * - X86_BUG_SPEC_STORE_BYPASS - CPU is susceptible. * - X86_FEATURE_SSBD - CPU is able to turn off speculative store bypass From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D0EEC43334 for ; Sat, 23 Jul 2022 10:10:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239096AbiGWKKQ (ORCPT ); Sat, 23 Jul 2022 06:10:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238903AbiGWKI7 (ORCPT ); Sat, 23 Jul 2022 06:08:59 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22106CA750; Sat, 23 Jul 2022 03:02:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C357D60C81; Sat, 23 Jul 2022 10:02:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2868C341C0; Sat, 23 Jul 2022 10:02:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570527; bh=TxC15aCWJTESKfDiZYzcZB3KhHflPWx08diIkyKVH6A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XYr/jmT7mpmed0gzfDvGHX3cYayCQmUe33mG6nTf7N2SeEXhiJYM78/iXzunyeoyZ 3KmaPNEYscPBGiS6OgIQgPQziWl67HPb0GlUSR9xdnu/3RcjG3XJ9ZTI1txutzcK1d w+jXwE1i3oTi3K8RosY1VwnJftisj4Xn/Vjsqrf4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 117/148] objtool: Re-add UNWIND_HINT_{SAVE_RESTORE} Date: Sat, 23 Jul 2022 11:55:29 +0200 Message-Id: <20220723095257.114909578@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 8faea26e611189e933ea2281975ff4dc7c1106b6 upstream. Commit c536ed2fffd5 ("objtool: Remove SAVE/RESTORE hints") removed the save/restore unwind hints because they were no longer needed. Now they're going to be needed again so re-add them. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/unwind_hints.h | 12 +++++++++- include/linux/objtool.h | 6 +++-- tools/include/linux/objtool.h | 6 +++-- tools/objtool/check.c | 40 +++++++++++++++++++++++++++++++= +++++ tools/objtool/check.h | 1=20 5 files changed, 59 insertions(+), 6 deletions(-) --- a/arch/x86/include/asm/unwind_hints.h +++ b/arch/x86/include/asm/unwind_hints.h @@ -8,11 +8,11 @@ #ifdef __ASSEMBLY__ =20 .macro UNWIND_HINT_EMPTY - UNWIND_HINT sp_reg=3DORC_REG_UNDEFINED type=3DUNWIND_HINT_TYPE_CALL end= =3D1 + UNWIND_HINT type=3DUNWIND_HINT_TYPE_CALL end=3D1 .endm =20 .macro UNWIND_HINT_ENTRY - UNWIND_HINT sp_reg=3DORC_REG_UNDEFINED type=3DUNWIND_HINT_TYPE_ENTRY end= =3D1 + UNWIND_HINT type=3DUNWIND_HINT_TYPE_ENTRY end=3D1 .endm =20 .macro UNWIND_HINT_REGS base=3D%rsp offset=3D0 indirect=3D0 extra=3D1 part= ial=3D0 @@ -56,6 +56,14 @@ UNWIND_HINT sp_reg=3DORC_REG_SP sp_offset=3D8 type=3DUNWIND_HINT_TYPE_FUNC .endm =20 +.macro UNWIND_HINT_SAVE + UNWIND_HINT type=3DUNWIND_HINT_TYPE_SAVE +.endm + +.macro UNWIND_HINT_RESTORE + UNWIND_HINT type=3DUNWIND_HINT_TYPE_RESTORE +.endm + #endif /* __ASSEMBLY__ */ =20 #endif /* _ASM_X86_UNWIND_HINTS_H */ --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -40,6 +40,8 @@ struct unwind_hint { #define UNWIND_HINT_TYPE_REGS_PARTIAL 2 #define UNWIND_HINT_TYPE_FUNC 3 #define UNWIND_HINT_TYPE_ENTRY 4 +#define UNWIND_HINT_TYPE_SAVE 5 +#define UNWIND_HINT_TYPE_RESTORE 6 =20 #ifdef CONFIG_STACK_VALIDATION =20 @@ -102,7 +104,7 @@ struct unwind_hint { * the debuginfo as necessary. It will also warn if it sees any * inconsistencies. */ -.macro UNWIND_HINT sp_reg:req sp_offset=3D0 type:req end=3D0 +.macro UNWIND_HINT type:req sp_reg=3D0 sp_offset=3D0 end=3D0 .Lunwind_hint_ip_\@: .pushsection .discard.unwind_hints /* struct unwind_hint */ @@ -126,7 +128,7 @@ struct unwind_hint { #define STACK_FRAME_NON_STANDARD(func) #else #define ANNOTATE_INTRA_FUNCTION_CALL -.macro UNWIND_HINT sp_reg:req sp_offset=3D0 type:req end=3D0 +.macro UNWIND_HINT type:req sp_reg=3D0 sp_offset=3D0 end=3D0 .endm #endif =20 --- a/tools/include/linux/objtool.h +++ b/tools/include/linux/objtool.h @@ -40,6 +40,8 @@ struct unwind_hint { #define UNWIND_HINT_TYPE_REGS_PARTIAL 2 #define UNWIND_HINT_TYPE_FUNC 3 #define UNWIND_HINT_TYPE_ENTRY 4 +#define UNWIND_HINT_TYPE_SAVE 5 +#define UNWIND_HINT_TYPE_RESTORE 6 =20 #ifdef CONFIG_STACK_VALIDATION =20 @@ -102,7 +104,7 @@ struct unwind_hint { * the debuginfo as necessary. It will also warn if it sees any * inconsistencies. */ -.macro UNWIND_HINT sp_reg:req sp_offset=3D0 type:req end=3D0 +.macro UNWIND_HINT type:req sp_reg=3D0 sp_offset=3D0 end=3D0 .Lunwind_hint_ip_\@: .pushsection .discard.unwind_hints /* struct unwind_hint */ @@ -126,7 +128,7 @@ struct unwind_hint { #define STACK_FRAME_NON_STANDARD(func) #else #define ANNOTATE_INTRA_FUNCTION_CALL -.macro UNWIND_HINT sp_reg:req sp_offset=3D0 type:req end=3D0 +.macro UNWIND_HINT type:req sp_reg=3D0 sp_offset=3D0 end=3D0 .endm #endif =20 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1752,6 +1752,17 @@ static int read_unwind_hints(struct objt =20 insn->hint =3D true; =20 + if (hint->type =3D=3D UNWIND_HINT_TYPE_SAVE) { + insn->hint =3D false; + insn->save =3D true; + continue; + } + + if (hint->type =3D=3D UNWIND_HINT_TYPE_RESTORE) { + insn->restore =3D true; + continue; + } + if (hint->type =3D=3D UNWIND_HINT_TYPE_REGS_PARTIAL) { struct symbol *sym =3D find_symbol_by_offset(insn->sec, insn->offset); =20 @@ -2847,6 +2858,35 @@ static int validate_branch(struct objtoo state.instr +=3D insn->instr; =20 if (insn->hint) { + if (insn->restore) { + struct instruction *save_insn, *i; + + i =3D insn; + save_insn =3D NULL; + + sym_for_each_insn_continue_reverse(file, func, i) { + if (i->save) { + save_insn =3D i; + break; + } + } + + if (!save_insn) { + WARN_FUNC("no corresponding CFI save for CFI restore", + sec, insn->offset); + return 1; + } + + if (!save_insn->visited) { + WARN_FUNC("objtool isn't smart enough to handle this CFI save/restore= combo", + sec, insn->offset); + return 1; + } + + insn->cfi =3D save_insn->cfi; + nr_cfi_reused++; + } + state.cfi =3D *insn->cfi; } else { /* XXX track if we actually changed state.cfi */ --- a/tools/objtool/check.h +++ b/tools/objtool/check.h @@ -47,6 +47,7 @@ struct instruction { unsigned long immediate; bool dead_end, ignore, ignore_alts; bool hint; + bool save, restore; bool retpoline_safe; bool entry; s8 instr; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17EB8CCA483 for ; Sat, 23 Jul 2022 10:10:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239198AbiGWKKZ (ORCPT ); Sat, 23 Jul 2022 06:10:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238925AbiGWKJF (ORCPT ); Sat, 23 Jul 2022 06:09:05 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E002CCA766; Sat, 23 Jul 2022 03:02:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7380EB82C22; Sat, 23 Jul 2022 10:02:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B6796C341C0; Sat, 23 Jul 2022 10:02:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570530; bh=uSCrARfkJXjP+FlPoukAD3jkic9RSZrM1HUIDh5lzKI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MiLgZR8icW7ygDGNM8ecXdJdvYflSUYhypB2LJQhmUUU9cUfgrlMyskcIFdsMhhme u3yZKXZEjhNFJP/TkuT0ABJF3DzVHYlWBk/Fu0HqwLpUUQ/6hIWF8ebWgSr2pMrbiZ +jSaVLeG5RSkwqmMcTECgLB865L4vqnkAf/F1dDE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 118/148] KVM: VMX: Flatten __vmx_vcpu_run() Date: Sat, 23 Jul 2022 11:55:30 +0200 Message-Id: <20220723095257.391891551@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 8bd200d23ec42d66ccd517a72dd0b9cc6132d2fd upstream. Move the vmx_vm{enter,exit}() functionality into __vmx_vcpu_run(). This will make it easier to do the spec_ctrl handling before the first RET. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov [cascardo: remove ENDBR] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/vmx/vmenter.S | 118 +++++++++++++++++-----------------------= ----- 1 file changed, 45 insertions(+), 73 deletions(-) --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -31,68 +31,6 @@ .section .noinstr.text, "ax" =20 /** - * vmx_vmenter - VM-Enter the current loaded VMCS - * - * %RFLAGS.ZF: !VMCS.LAUNCHED, i.e. controls VMLAUNCH vs. VMRESUME - * - * Returns: - * %RFLAGS.CF is set on VM-Fail Invalid - * %RFLAGS.ZF is set on VM-Fail Valid - * %RFLAGS.{CF,ZF} are cleared on VM-Success, i.e. VM-Exit - * - * Note that VMRESUME/VMLAUNCH fall-through and return directly if - * they VM-Fail, whereas a successful VM-Enter + VM-Exit will jump - * to vmx_vmexit. - */ -SYM_FUNC_START_LOCAL(vmx_vmenter) - /* EFLAGS.ZF is set if VMCS.LAUNCHED =3D=3D 0 */ - je 2f - -1: vmresume - RET - -2: vmlaunch - RET - -3: cmpb $0, kvm_rebooting - je 4f - RET -4: ud2 - - _ASM_EXTABLE(1b, 3b) - _ASM_EXTABLE(2b, 3b) - -SYM_FUNC_END(vmx_vmenter) - -/** - * vmx_vmexit - Handle a VMX VM-Exit - * - * Returns: - * %RFLAGS.{CF,ZF} are cleared on VM-Success, i.e. VM-Exit - * - * This is vmx_vmenter's partner in crime. On a VM-Exit, control will jump - * here after hardware loads the host's state, i.e. this is the destination - * referred to by VMCS.HOST_RIP. - */ -SYM_FUNC_START(vmx_vmexit) -#ifdef CONFIG_RETPOLINE - ALTERNATIVE "jmp .Lvmexit_skip_rsb", "", X86_FEATURE_RETPOLINE - /* Preserve guest's RAX, it's used to stuff the RSB. */ - push %_ASM_AX - - /* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */ - FILL_RETURN_BUFFER %_ASM_AX, RSB_CLEAR_LOOPS, X86_FEATURE_RETPOLINE - - /* Clear RFLAGS.CF and RFLAGS.ZF to preserve VM-Exit, i.e. !VM-Fail. */ - or $1, %_ASM_AX - - pop %_ASM_AX -.Lvmexit_skip_rsb: -#endif - RET -SYM_FUNC_END(vmx_vmexit) - -/** * __vmx_vcpu_run - Run a vCPU via a transition to VMX guest mode * @vmx: struct vcpu_vmx * (forwarded to vmx_update_host_rsp) * @regs: unsigned long * (to guest registers) @@ -124,8 +62,7 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Copy @launched to BL, _ASM_ARG3 is volatile. */ mov %_ASM_ARG3B, %bl =20 - /* Adjust RSP to account for the CALL to vmx_vmenter(). */ - lea -WORD_SIZE(%_ASM_SP), %_ASM_ARG2 + lea (%_ASM_SP), %_ASM_ARG2 call vmx_update_host_rsp =20 /* Load @regs to RAX. */ @@ -154,11 +91,36 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX =20 - /* Enter guest mode */ - call vmx_vmenter + /* Check EFLAGS.ZF from 'testb' above */ + je .Lvmlaunch =20 - /* Jump on VM-Fail. */ - jbe 2f + /* + * After a successful VMRESUME/VMLAUNCH, control flow "magically" + * resumes below at 'vmx_vmexit' due to the VMCS HOST_RIP setting. + * So this isn't a typical function and objtool needs to be told to + * save the unwind state here and restore it below. + */ + UNWIND_HINT_SAVE + +/* + * If VMRESUME/VMLAUNCH and corresponding vmexit succeed, execution resume= s at + * the 'vmx_vmexit' label below. + */ +.Lvmresume: + vmresume + jmp .Lvmfail + +.Lvmlaunch: + vmlaunch + jmp .Lvmfail + + _ASM_EXTABLE(.Lvmresume, .Lfixup) + _ASM_EXTABLE(.Lvmlaunch, .Lfixup) + +SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) + + /* Restore unwind state from before the VMRESUME/VMLAUNCH. */ + UNWIND_HINT_RESTORE =20 /* Temporarily save guest's RAX. */ push %_ASM_AX @@ -185,9 +147,13 @@ SYM_FUNC_START(__vmx_vcpu_run) mov %r15, VCPU_R15(%_ASM_AX) #endif =20 + /* IMPORTANT: RSB must be stuffed before the first return. */ + FILL_RETURN_BUFFER %_ASM_BX, RSB_CLEAR_LOOPS, X86_FEATURE_RETPOLINE + /* Clear RAX to indicate VM-Exit (as opposed to VM-Fail). */ xor %eax, %eax =20 +.Lclear_regs: /* * Clear all general purpose registers except RSP and RAX to prevent * speculative use of the guest's values, even those that are reloaded @@ -197,7 +163,7 @@ SYM_FUNC_START(__vmx_vcpu_run) * free. RSP and RAX are exempt as RSP is restored by hardware during * VM-Exit and RAX is explicitly loaded with 0 or 1 to return VM-Fail. */ -1: xor %ecx, %ecx + xor %ecx, %ecx xor %edx, %edx xor %ebx, %ebx xor %ebp, %ebp @@ -216,8 +182,8 @@ SYM_FUNC_START(__vmx_vcpu_run) =20 /* "POP" @regs. */ add $WORD_SIZE, %_ASM_SP - pop %_ASM_BX =20 + pop %_ASM_BX #ifdef CONFIG_X86_64 pop %r12 pop %r13 @@ -230,9 +196,15 @@ SYM_FUNC_START(__vmx_vcpu_run) pop %_ASM_BP RET =20 - /* VM-Fail. Out-of-line to avoid a taken Jcc after VM-Exit. */ -2: mov $1, %eax - jmp 1b +.Lfixup: + cmpb $0, kvm_rebooting + jne .Lvmfail + ud2 +.Lvmfail: + /* VM-Fail: set return value to 1 */ + mov $1, %eax + jmp .Lclear_regs + SYM_FUNC_END(__vmx_vcpu_run) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B77FDC43334 for ; Sat, 23 Jul 2022 10:11:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239156AbiGWKLZ (ORCPT ); Sat, 23 Jul 2022 06:11:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239163AbiGWKJV (ORCPT ); Sat, 23 Jul 2022 06:09:21 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E715FCBD09; Sat, 23 Jul 2022 03:02:33 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8BE0B611BD; Sat, 23 Jul 2022 10:02:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B1A3C341C0; Sat, 23 Jul 2022 10:02:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570533; bh=N3qLSRnfhNQQ+z7m0DDtKuNmKRtrSS1K8O0XNISvmOc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=D+2szLaPqLGpxa/ObTAdxq0Utrw84gQIMXBouoqOT4CKMeVhjjHE7VPgJoSrQR5zz MZ+pmC/1KtHjIj45Nnv8fASJhJzrbpKDxKQ+avXFINrvp78bmGlAp0KD50b3vKIb64 sO0GWeUuppoo7pmj8rZYjFRgCuBSW+lt8TSaWE0A= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 119/148] KVM: VMX: Convert launched argument to flags Date: Sat, 23 Jul 2022 11:55:31 +0200 Message-Id: <20220723095257.638599760@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit bb06650634d3552c0f8557e9d16aa1a408040e28 upstream. Convert __vmx_vcpu_run()'s 'launched' argument to 'flags', in preparation for doing SPEC_CTRL handling immediately after vmexit, which will need another flag. This is much easier than adding a fourth argument, because this code supports both 32-bit and 64-bit, and the fourth argument on 32-bit would have to be pushed on the stack. Note that __vmx_vcpu_run_flags() is called outside of the noinstr critical section because it will soon start calling potentially traceable functions. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/vmx/run_flags.h | 7 +++++++ arch/x86/kvm/vmx/vmenter.S | 9 +++++---- arch/x86/kvm/vmx/vmx.c | 17 ++++++++++++++--- arch/x86/kvm/vmx/vmx.h | 5 ++++- 5 files changed, 31 insertions(+), 9 deletions(-) create mode 100644 arch/x86/kvm/vmx/run_flags.h --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3077,7 +3077,7 @@ static int nested_vmx_check_vmentry_hw(s } =20 vm_fail =3D __vmx_vcpu_run(vmx, (unsigned long *)&vcpu->arch.regs, - vmx->loaded_vmcs->launched); + __vmx_vcpu_run_flags(vmx)); =20 if (vmx->msr_autoload.host.nr) vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr); --- /dev/null +++ b/arch/x86/kvm/vmx/run_flags.h @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMX_RUN_FLAGS_H +#define __KVM_X86_VMX_RUN_FLAGS_H + +#define VMX_RUN_VMRESUME (1 << 0) + +#endif /* __KVM_X86_VMX_RUN_FLAGS_H */ --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -5,6 +5,7 @@ #include #include #include +#include "run_flags.h" =20 #define WORD_SIZE (BITS_PER_LONG / 8) =20 @@ -34,7 +35,7 @@ * __vmx_vcpu_run - Run a vCPU via a transition to VMX guest mode * @vmx: struct vcpu_vmx * (forwarded to vmx_update_host_rsp) * @regs: unsigned long * (to guest registers) - * @launched: %true if the VMCS has been launched + * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH * * Returns: * 0 on VM-Exit, 1 on VM-Fail @@ -59,7 +60,7 @@ SYM_FUNC_START(__vmx_vcpu_run) */ push %_ASM_ARG2 =20 - /* Copy @launched to BL, _ASM_ARG3 is volatile. */ + /* Copy @flags to BL, _ASM_ARG3 is volatile. */ mov %_ASM_ARG3B, %bl =20 lea (%_ASM_SP), %_ASM_ARG2 @@ -69,7 +70,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov (%_ASM_SP), %_ASM_AX =20 /* Check if vmlaunch or vmresume is needed */ - testb %bl, %bl + testb $VMX_RUN_VMRESUME, %bl =20 /* Load guest registers. Don't clobber flags. */ mov VCPU_RCX(%_ASM_AX), %_ASM_CX @@ -92,7 +93,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov VCPU_RAX(%_ASM_AX), %_ASM_AX =20 /* Check EFLAGS.ZF from 'testb' above */ - je .Lvmlaunch + jz .Lvmlaunch =20 /* * After a successful VMRESUME/VMLAUNCH, control flow "magically" --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -936,6 +936,16 @@ static bool msr_write_intercepted(struct return true; } =20 +unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx) +{ + unsigned int flags =3D 0; + + if (vmx->loaded_vmcs->launched) + flags |=3D VMX_RUN_VMRESUME; + + return flags; +} + static void clear_atomic_switch_msr_special(struct vcpu_vmx *vmx, unsigned long entry, unsigned long exit) { @@ -6688,7 +6698,8 @@ static fastpath_t vmx_exit_handlers_fast } =20 static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, - struct vcpu_vmx *vmx) + struct vcpu_vmx *vmx, + unsigned long flags) { /* * VMENTER enables interrupts (host state), but the kernel state is @@ -6725,7 +6736,7 @@ static noinstr void vmx_vcpu_enter_exit( native_write_cr2(vcpu->arch.cr2); =20 vmx->fail =3D __vmx_vcpu_run(vmx, (unsigned long *)&vcpu->arch.regs, - vmx->loaded_vmcs->launched); + flags); =20 vcpu->arch.cr2 =3D native_read_cr2(); =20 @@ -6824,7 +6835,7 @@ reenter_guest: x86_spec_ctrl_set_guest(vmx->spec_ctrl, 0); =20 /* The actual VMENTER/EXIT is in the .noinstr.text section. */ - vmx_vcpu_enter_exit(vcpu, vmx); + vmx_vcpu_enter_exit(vcpu, vmx, __vmx_vcpu_run_flags(vmx)); =20 /* * We do not use IBRS in the kernel. If this vCPU has used the --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -13,6 +13,7 @@ #include "vmcs.h" #include "vmx_ops.h" #include "cpuid.h" +#include "run_flags.h" =20 extern const u32 vmx_msr_index[]; =20 @@ -365,7 +366,9 @@ void vmx_set_virtual_apic_mode(struct kv struct vmx_uret_msr *vmx_find_uret_msr(struct vcpu_vmx *vmx, u32 msr); void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu); void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp); -bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, bool launch= ed); +unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx); +bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, + unsigned int flags); int vmx_find_loadstore_msr_slot(struct vmx_msrs *m, u32 msr); void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2175FC43334 for ; Sat, 23 Jul 2022 10:10:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238453AbiGWKKd (ORCPT ); Sat, 23 Jul 2022 06:10:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238940AbiGWKJH (ORCPT ); Sat, 23 Jul 2022 06:09:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 543F6CA76F; Sat, 23 Jul 2022 03:02:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A9016611CD; Sat, 23 Jul 2022 10:01:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC407C341CB; Sat, 23 Jul 2022 10:01:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570504; bh=GIEThlDtnvK8qlEc+Re+qg9RuqDWS0iY9Tzeocl1874=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aJ/+9fcuCA2hUmytcae3HT4nrIgZxc8oo9zX8dv4y5bPpMwBP+TZwKxdNn2MUI4qT eBrWdApWdUklTfliQYiOlFy99dzvlD2ri1OrnGln2/uv9x7EIKp9paHgpomHTkdFlr eKQt3ai+xewfJ8e4dltprmj2F5P6JJSM3MkpKUSg= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 120/148] KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS Date: Sat, 23 Jul 2022 11:55:32 +0200 Message-Id: <20220723095257.919600394@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit fc02735b14fff8c6678b521d324ade27b1a3d4cf upstream. On eIBRS systems, the returns in the vmexit return path from __vmx_vcpu_run() to vmx_vcpu_run() are exposed to RSB poisoning attacks. Fix that by moving the post-vmexit spec_ctrl handling to immediately after the vmexit. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 1=20 arch/x86/kernel/cpu/bugs.c | 4 ++ arch/x86/kvm/vmx/run_flags.h | 1=20 arch/x86/kvm/vmx/vmenter.S | 49 +++++++++++++++++++++++++++---= ----- arch/x86/kvm/vmx/vmx.c | 48 ++++++++++++++++++++----------= ---- arch/x86/kvm/vmx/vmx.h | 1=20 6 files changed, 73 insertions(+), 31 deletions(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -275,6 +275,7 @@ static inline void indirect_branch_predi =20 /* The Intel SPEC CTRL MSR base value cache */ extern u64 x86_spec_ctrl_base; +extern u64 x86_spec_ctrl_current; extern void write_spec_ctrl_current(u64 val, bool force); extern u64 spec_ctrl_current(void); =20 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -186,6 +186,10 @@ void __init check_bugs(void) #endif } =20 +/* + * NOTE: For VMX, this function is not called in the vmexit path. + * It uses vmx_spec_ctrl_restore_host() instead. + */ void x86_virt_spec_ctrl(u64 guest_spec_ctrl, u64 guest_virt_spec_ctrl, bool set= guest) { --- a/arch/x86/kvm/vmx/run_flags.h +++ b/arch/x86/kvm/vmx/run_flags.h @@ -3,5 +3,6 @@ #define __KVM_X86_VMX_RUN_FLAGS_H =20 #define VMX_RUN_VMRESUME (1 << 0) +#define VMX_RUN_SAVE_SPEC_CTRL (1 << 1) =20 #endif /* __KVM_X86_VMX_RUN_FLAGS_H */ --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -33,9 +33,10 @@ =20 /** * __vmx_vcpu_run - Run a vCPU via a transition to VMX guest mode - * @vmx: struct vcpu_vmx * (forwarded to vmx_update_host_rsp) + * @vmx: struct vcpu_vmx * * @regs: unsigned long * (to guest registers) - * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH + * @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH + * VMX_RUN_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl * * Returns: * 0 on VM-Exit, 1 on VM-Fail @@ -54,6 +55,12 @@ SYM_FUNC_START(__vmx_vcpu_run) #endif push %_ASM_BX =20 + /* Save @vmx for SPEC_CTRL handling */ + push %_ASM_ARG1 + + /* Save @flags for SPEC_CTRL handling */ + push %_ASM_ARG3 + /* * Save @regs, _ASM_ARG2 may be modified by vmx_update_host_rsp() and * @regs is needed after VM-Exit to save the guest's register values. @@ -148,25 +155,23 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL mov %r15, VCPU_R15(%_ASM_AX) #endif =20 - /* IMPORTANT: RSB must be stuffed before the first return. */ - FILL_RETURN_BUFFER %_ASM_BX, RSB_CLEAR_LOOPS, X86_FEATURE_RETPOLINE - - /* Clear RAX to indicate VM-Exit (as opposed to VM-Fail). */ - xor %eax, %eax + /* Clear return value to indicate VM-Exit (as opposed to VM-Fail). */ + xor %ebx, %ebx =20 .Lclear_regs: /* - * Clear all general purpose registers except RSP and RAX to prevent + * Clear all general purpose registers except RSP and RBX to prevent * speculative use of the guest's values, even those that are reloaded * via the stack. In theory, an L1 cache miss when restoring registers * could lead to speculative execution with the guest's values. * Zeroing XORs are dirt cheap, i.e. the extra paranoia is essentially * free. RSP and RAX are exempt as RSP is restored by hardware during - * VM-Exit and RAX is explicitly loaded with 0 or 1 to return VM-Fail. + * VM-Exit and RBX is explicitly loaded with 0 or 1 to hold the return + * value. */ + xor %eax, %eax xor %ecx, %ecx xor %edx, %edx - xor %ebx, %ebx xor %ebp, %ebp xor %esi, %esi xor %edi, %edi @@ -184,6 +189,28 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL /* "POP" @regs. */ add $WORD_SIZE, %_ASM_SP =20 + /* + * IMPORTANT: RSB filling and SPEC_CTRL handling must be done before + * the first unbalanced RET after vmexit! + * + * For retpoline, RSB filling is needed to prevent poisoned RSB entries + * and (in some cases) RSB underflow. + * + * eIBRS has its own protection against poisoned RSB, so it doesn't + * need the RSB filling sequence. But it does need to be enabled + * before the first unbalanced RET. + */ + + FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RETPOLINE + + pop %_ASM_ARG2 /* @flags */ + pop %_ASM_ARG1 /* @vmx */ + + call vmx_spec_ctrl_restore_host + + /* Put return value in AX */ + mov %_ASM_BX, %_ASM_AX + pop %_ASM_BX #ifdef CONFIG_X86_64 pop %r12 @@ -203,7 +230,7 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL ud2 .Lvmfail: /* VM-Fail: set return value to 1 */ - mov $1, %eax + mov $1, %_ASM_BX jmp .Lclear_regs =20 SYM_FUNC_END(__vmx_vcpu_run) --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -943,6 +943,14 @@ unsigned int __vmx_vcpu_run_flags(struct if (vmx->loaded_vmcs->launched) flags |=3D VMX_RUN_VMRESUME; =20 + /* + * If writes to the SPEC_CTRL MSR aren't intercepted, the guest is free + * to change it directly without causing a vmexit. In that case read + * it after vmexit and store it in vmx->spec_ctrl. + */ + if (unlikely(!msr_write_intercepted(vmx, MSR_IA32_SPEC_CTRL))) + flags |=3D VMX_RUN_SAVE_SPEC_CTRL; + return flags; } =20 @@ -6685,6 +6693,26 @@ void noinstr vmx_update_host_rsp(struct } } =20 +void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, + unsigned int flags) +{ + u64 hostval =3D this_cpu_read(x86_spec_ctrl_current); + + if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL)) + return; + + if (flags & VMX_RUN_SAVE_SPEC_CTRL) + vmx->spec_ctrl =3D __rdmsr(MSR_IA32_SPEC_CTRL); + + /* + * If the guest/host SPEC_CTRL values differ, restore the host value. + */ + if (vmx->spec_ctrl !=3D hostval) + native_wrmsrl(MSR_IA32_SPEC_CTRL, hostval); + + barrier_nospec(); +} + static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) { switch (to_vmx(vcpu)->exit_reason.basic) { @@ -6837,26 +6865,6 @@ reenter_guest: /* The actual VMENTER/EXIT is in the .noinstr.text section. */ vmx_vcpu_enter_exit(vcpu, vmx, __vmx_vcpu_run_flags(vmx)); =20 - /* - * We do not use IBRS in the kernel. If this vCPU has used the - * SPEC_CTRL MSR it may have left it on; save the value and - * turn it off. This is much more efficient than blindly adding - * it to the atomic save/restore list. Especially as the former - * (Saving guest MSRs on vmexit) doesn't even exist in KVM. - * - * For non-nested case: - * If the L01 MSR bitmap does not intercept the MSR, then we need to - * save it. - * - * For nested case: - * If the L02 MSR bitmap does not intercept the MSR, then we need to - * save it. - */ - if (unlikely(!msr_write_intercepted(vmx, MSR_IA32_SPEC_CTRL))) - vmx->spec_ctrl =3D native_read_msr(MSR_IA32_SPEC_CTRL); - - x86_spec_ctrl_restore_host(vmx->spec_ctrl, 0); - /* All fields are clean at this point */ if (static_branch_unlikely(&enable_evmcs)) current_evmcs->hv_clean_fields |=3D --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -366,6 +366,7 @@ void vmx_set_virtual_apic_mode(struct kv struct vmx_uret_msr *vmx_find_uret_msr(struct vcpu_vmx *vmx, u32 msr); void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu); void vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp); +void vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx, unsigned int flags); unsigned int __vmx_vcpu_run_flags(struct vcpu_vmx *vmx); bool __vmx_vcpu_run(struct vcpu_vmx *vmx, unsigned long *regs, unsigned int flags); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B63CECCA48C for ; Sat, 23 Jul 2022 10:10:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239014AbiGWKKJ (ORCPT ); Sat, 23 Jul 2022 06:10:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238239AbiGWKIm (ORCPT ); Sat, 23 Jul 2022 06:08:42 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FC7EC5D77; Sat, 23 Jul 2022 03:01:47 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8F78161265; Sat, 23 Jul 2022 10:01:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 98550C341C0; Sat, 23 Jul 2022 10:01:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570507; bh=J9Pf+qzW800PVoLP/QpqWlIKLOEjtqTEYoBrXWnkZow=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HS3EqPRsi6wiSsidc1u3cZHYAZS/k6BWOvIY1j2u5bUoyHfbvQ9pYyWbMLv1/0hjZ NuXRoBG4ZtZPsLbDPUx7Ha41nYR46mD6wya3Sl+3gfMTioAExzVoj5saUJXj/XX10p uiBoXQKpgV7kFX4C/398B4M2YanE9Khd8PiqOePc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 121/148] KVM: VMX: Fix IBRS handling after vmexit Date: Sat, 23 Jul 2022 11:55:33 +0200 Message-Id: <20220723095258.204130737@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit bea7e31a5caccb6fe8ed989c065072354f0ecb52 upstream. For legacy IBRS to work, the IBRS bit needs to be always re-written after vmexit, even if it's already on. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/vmx/vmx.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6706,8 +6706,13 @@ void noinstr vmx_spec_ctrl_restore_host( =20 /* * If the guest/host SPEC_CTRL values differ, restore the host value. + * + * For legacy IBRS, the IBRS bit always needs to be written after + * transitioning from a less privileged predictor mode, regardless of + * whether the guest/host values differ. */ - if (vmx->spec_ctrl !=3D hostval) + if (cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS) || + vmx->spec_ctrl !=3D hostval) native_wrmsrl(MSR_IA32_SPEC_CTRL, hostval); =20 barrier_nospec(); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0EE1CCA48C for ; Sat, 23 Jul 2022 10:09:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238680AbiGWKJu (ORCPT ); Sat, 23 Jul 2022 06:09:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238500AbiGWKIe (ORCPT ); Sat, 23 Jul 2022 06:08:34 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F9631EAF3; Sat, 23 Jul 2022 03:01:53 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5FCC5B82C22; Sat, 23 Jul 2022 10:01:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B061DC341C7; Sat, 23 Jul 2022 10:01:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570510; bh=DtFkrCHxAPSTMdz898VV0Dv6t+3H+rDdUVp5aAzqCKc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BQkS83jLr5V6S6qmM4C8xebFYHghcD09DCO2/cKKAZbIoe7MsAFd51FycEwpxoZob 5w8+tiMiguTXQjqlb+vAEA/nyye8ew088mI5DkLHRDaylq/C0d5nDsNdLGUvbxhbnY z23bE3+CGokSAzjIEghQtpCf6EkYtdyhG31V3XAU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Josh Poimboeuf , "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 122/148] x86/speculation: Fill RSB on vmexit for IBRS Date: Sat, 23 Jul 2022 11:55:34 +0200 Message-Id: <20220723095258.498105036@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 9756bba28470722dacb79ffce554336dd1f6a6cd upstream. Prevent RSB underflow/poisoning attacks with RSB. While at it, add a bunch of comments to attempt to document the current state of tribal knowledge about RSB attacks and what exactly is being mitigated. Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/cpufeatures.h | 2 - arch/x86/kernel/cpu/bugs.c | 63 ++++++++++++++++++++++++++++++++= ++--- arch/x86/kvm/vmx/vmenter.S | 6 +-- 3 files changed, 62 insertions(+), 9 deletions(-) --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -204,7 +204,7 @@ #define X86_FEATURE_SME ( 7*32+10) /* AMD Secure Memory Encryption */ #define X86_FEATURE_PTI ( 7*32+11) /* Kernel Page Table Isolation enable= d */ #define X86_FEATURE_KERNEL_IBRS ( 7*32+12) /* "" Set/clear IBRS on kernel= entry/exit */ -/* FREE! ( 7*32+13) */ +#define X86_FEATURE_RSB_VMEXIT ( 7*32+13) /* "" Fill RSB on VM-Exit */ #define X86_FEATURE_INTEL_PPIN ( 7*32+14) /* Intel Processor Inventory Nu= mber */ #define X86_FEATURE_CDP_L2 ( 7*32+15) /* Code and Data Prioritization L2 = */ #define X86_FEATURE_MSR_SPEC_CTRL ( 7*32+16) /* "" MSR SPEC_CTRL is implem= ented */ --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1357,17 +1357,70 @@ static void __init spectre_v2_select_mit pr_info("%s\n", spectre_v2_strings[mode]); =20 /* - * If spectre v2 protection has been enabled, unconditionally fill - * RSB during a context switch; this protects against two independent - * issues: + * If Spectre v2 protection has been enabled, fill the RSB during a + * context switch. In general there are two types of RSB attacks + * across context switches, for which the CALLs/RETs may be unbalanced. * - * - RSB underflow (and switch to BTB) on Skylake+ - * - SpectreRSB variant of spectre v2 on X86_BUG_SPECTRE_V2 CPUs + * 1) RSB underflow + * + * Some Intel parts have "bottomless RSB". When the RSB is empty, + * speculated return targets may come from the branch predictor, + * which could have a user-poisoned BTB or BHB entry. + * + * AMD has it even worse: *all* returns are speculated from the BTB, + * regardless of the state of the RSB. + * + * When IBRS or eIBRS is enabled, the "user -> kernel" attack + * scenario is mitigated by the IBRS branch prediction isolation + * properties, so the RSB buffer filling wouldn't be necessary to + * protect against this type of attack. + * + * The "user -> user" attack scenario is mitigated by RSB filling. + * + * 2) Poisoned RSB entry + * + * If the 'next' in-kernel return stack is shorter than 'prev', + * 'next' could be tricked into speculating with a user-poisoned RSB + * entry. + * + * The "user -> kernel" attack scenario is mitigated by SMEP and + * eIBRS. + * + * The "user -> user" scenario, also known as SpectreBHB, requires + * RSB clearing. + * + * So to mitigate all cases, unconditionally fill RSB on context + * switches. + * + * FIXME: Is this pointless for retbleed-affected AMD? */ setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW); pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context switc= h\n"); =20 /* + * Similar to context switches, there are two types of RSB attacks + * after vmexit: + * + * 1) RSB underflow + * + * 2) Poisoned RSB entry + * + * When retpoline is enabled, both are mitigated by filling/clearing + * the RSB. + * + * When IBRS is enabled, while #1 would be mitigated by the IBRS branch + * prediction isolation protections, RSB still needs to be cleared + * because of #2. Note that SMEP provides no protection here, unlike + * user-space-poisoned RSB entries. + * + * eIBRS, on the other hand, has RSB-poisoning protections, so it + * doesn't need RSB clearing after vmexit. + */ + if (boot_cpu_has(X86_FEATURE_RETPOLINE) || + boot_cpu_has(X86_FEATURE_KERNEL_IBRS)) + setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT); + + /* * Retpoline protects the kernel, but doesn't protect firmware. IBRS * and Enhanced IBRS protect firmware too, so enable IBRS around * firmware calls only when IBRS / Enhanced IBRS aren't otherwise --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -193,15 +193,15 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL * IMPORTANT: RSB filling and SPEC_CTRL handling must be done before * the first unbalanced RET after vmexit! * - * For retpoline, RSB filling is needed to prevent poisoned RSB entries - * and (in some cases) RSB underflow. + * For retpoline or IBRS, RSB filling is needed to prevent poisoned RSB + * entries and (in some cases) RSB underflow. * * eIBRS has its own protection against poisoned RSB, so it doesn't * need the RSB filling sequence. But it does need to be enabled * before the first unbalanced RET. */ =20 - FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RETPOLINE + FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT =20 pop %_ASM_ARG2 /* @flags */ pop %_ASM_ARG1 /* @vmx */ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3AA1C433EF for ; Sat, 23 Jul 2022 10:11:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239340AbiGWKL4 (ORCPT ); Sat, 23 Jul 2022 06:11:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239513AbiGWKJl (ORCPT ); Sat, 23 Jul 2022 06:09:41 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6888CCE50F; Sat, 23 Jul 2022 03:02:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3F2D7612C8; Sat, 23 Jul 2022 10:02:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4D78CC341C0; Sat, 23 Jul 2022 10:02:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570573; bh=2wzI7Jtu17HjUe9r2RGyDwlJF8K15SFARMRYJoBHEEU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JW4O7BxplOaIGRbBU8Mz+1N603yLpW5a8p1VBuqHyegXHAUFlJG0hY6N19VGA5N4b 8vvYBokM5J67ubCCe4VyZpTLnLHKP4J5Me+Z40yw61f26htV7TaK1vVuAa4u6502Ns POR2E9mdc9+TAD3th/NjK0Lc5pXWHUrClK9So6yk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Dave Hansen , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 123/148] x86/common: Stamp out the stepping madness Date: Sat, 23 Jul 2022 11:55:35 +0200 Message-Id: <20220723095258.768160237@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 7a05bc95ed1c5a59e47aaade9fb4083c27de9e62 upstream. The whole MMIO/RETBLEED enumeration went overboard on steppings. Get rid of all that and simply use ANY. If a future stepping of these models would not be affected, it had better set the relevant ARCH_CAP_$FOO_NO bit in IA32_ARCH_CAPABILITIES. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Acked-by: Dave Hansen Signed-off-by: Borislav Petkov Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/common.c | 37 ++++++++++++++++--------------------- 1 file changed, 16 insertions(+), 21 deletions(-) --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1119,32 +1119,27 @@ static const struct x86_cpu_id cpu_vuln_ VULNBL_INTEL_STEPPINGS(HASWELL, X86_STEPPING_ANY, SRBDS), VULNBL_INTEL_STEPPINGS(HASWELL_L, X86_STEPPING_ANY, SRBDS), VULNBL_INTEL_STEPPINGS(HASWELL_G, X86_STEPPING_ANY, SRBDS), - VULNBL_INTEL_STEPPINGS(HASWELL_X, BIT(2) | BIT(4), MMIO), - VULNBL_INTEL_STEPPINGS(BROADWELL_D, X86_STEPPINGS(0x3, 0x5), MMIO), + VULNBL_INTEL_STEPPINGS(HASWELL_X, X86_STEPPING_ANY, MMIO), + VULNBL_INTEL_STEPPINGS(BROADWELL_D, X86_STEPPING_ANY, MMIO), VULNBL_INTEL_STEPPINGS(BROADWELL_G, X86_STEPPING_ANY, SRBDS), VULNBL_INTEL_STEPPINGS(BROADWELL_X, X86_STEPPING_ANY, MMIO), VULNBL_INTEL_STEPPINGS(BROADWELL, X86_STEPPING_ANY, SRBDS), - VULNBL_INTEL_STEPPINGS(SKYLAKE_L, X86_STEPPINGS(0x3, 0x3), SRBDS | MMIO |= RETBLEED), - VULNBL_INTEL_STEPPINGS(SKYLAKE_L, X86_STEPPING_ANY, SRBDS), - VULNBL_INTEL_STEPPINGS(SKYLAKE_X, BIT(3) | BIT(4) | BIT(6) | - BIT(7) | BIT(0xB), MMIO | RETBLEED), - VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPINGS(0x3, 0x3), SRBDS | MMIO | = RETBLEED), - VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPING_ANY, SRBDS), - VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPINGS(0x9, 0xC), SRBDS | MMIO = | RETBLEED), - VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPINGS(0x0, 0x8), SRBDS), - VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPINGS(0x9, 0xD), SRBDS | MMIO | = RETBLEED), - VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPINGS(0x0, 0x8), SRBDS), - VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPINGS(0x5, 0x5), MMIO | MMIO_SB= DS | RETBLEED), - VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPINGS(0x1, 0x1), MMIO), - VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPINGS(0x4, 0x6), MMIO), - VULNBL_INTEL_STEPPINGS(COMETLAKE, BIT(2) | BIT(3) | BIT(5), MMIO | MMIO_S= BDS | RETBLEED), - VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO_= SBDS | RETBLEED), + VULNBL_INTEL_STEPPINGS(SKYLAKE_L, X86_STEPPING_ANY, SRBDS | MMIO | RETBL= EED), + VULNBL_INTEL_STEPPINGS(SKYLAKE_X, X86_STEPPING_ANY, MMIO | RETBLEED), + VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPING_ANY, SRBDS | MMIO | RETBLE= ED), + VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPING_ANY, SRBDS | MMIO | RETB= LEED), + VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPING_ANY, SRBDS | MMIO | RETBLE= ED), + VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | R= ETBLEED), + VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPING_ANY, MMIO), + VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPING_ANY, MMIO), + VULNBL_INTEL_STEPPINGS(COMETLAKE, X86_STEPPING_ANY, MMIO | MMIO_SBDS | R= ETBLEED), VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x0, 0x0), MMIO | RETBL= EED), - VULNBL_INTEL_STEPPINGS(LAKEFIELD, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO_SB= DS | RETBLEED), - VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPINGS(0x1, 0x1), MMIO | RETBLE= ED), - VULNBL_INTEL_STEPPINGS(ATOM_TREMONT, X86_STEPPINGS(0x1, 0x1), MMIO | MMIO= _SBDS), + VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS |= RETBLEED), + VULNBL_INTEL_STEPPINGS(LAKEFIELD, X86_STEPPING_ANY, MMIO | MMIO_SBDS | R= ETBLEED), + VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPING_ANY, MMIO | RETBLEED), + VULNBL_INTEL_STEPPINGS(ATOM_TREMONT, X86_STEPPING_ANY, MMIO | MMIO_SBDS), VULNBL_INTEL_STEPPINGS(ATOM_TREMONT_D, X86_STEPPING_ANY, MMIO), - VULNBL_INTEL_STEPPINGS(ATOM_TREMONT_L, X86_STEPPINGS(0x0, 0x0), MMIO | MM= IO_SBDS), + VULNBL_INTEL_STEPPINGS(ATOM_TREMONT_L, X86_STEPPING_ANY, MMIO | MMIO_SBD= S), =20 VULNBL_AMD(0x15, RETBLEED), VULNBL_AMD(0x16, RETBLEED), From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97A5DC433EF for ; Sat, 23 Jul 2022 10:10:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238991AbiGWKKw (ORCPT ); Sat, 23 Jul 2022 06:10:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239021AbiGWKJN (ORCPT ); Sat, 23 Jul 2022 06:09:13 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45B617E323; Sat, 23 Jul 2022 03:02:22 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BCE37612B4; Sat, 23 Jul 2022 10:02:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C8D24C341C0; Sat, 23 Jul 2022 10:02:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570539; bh=27eIWJulyqSRKoMB+Qg9COgotQ0OI5jj3NKghFqH9oY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RIHe02dZaE2Iw0upVG5h/TgCE4bX7otebXU44OBCjUgRuN3Q7/BLSaD/6FYJtTj2C Fi1RMf2zR5NqlfgpSfOJTkpM26youbVFOw0aUR9wfuWVs1mrChTFVh4XrtEj/FvTv0 Y7JK/5AFNqzNbfYAdQfJH/ftA5NCyUTh/ffP9U3s= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Andrew Cooper , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 124/148] x86/cpu/amd: Enumerate BTC_NO Date: Sat, 23 Jul 2022 11:55:36 +0200 Message-Id: <20220723095259.044246744@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Andrew Cooper commit 26aae8ccbc1972233afd08fb3f368947c0314265 upstream. BTC_NO indicates that hardware is not susceptible to Branch Type Confusion. Zen3 CPUs don't suffer BTC. Hypervisors are expected to synthesise BTC_NO when it is appropriate given the migration pool, to prevent kernels using heuristics. [ bp: Massage. ] Signed-off-by: Andrew Cooper Signed-off-by: Borislav Petkov [cascardo: no X86_FEATURE_BRS] [cascardo: no X86_FEATURE_CPPC] Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/kernel/cpu/amd.c | 21 +++++++++++++++------ arch/x86/kernel/cpu/common.c | 6 ++++-- 3 files changed, 20 insertions(+), 8 deletions(-) --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -316,6 +316,7 @@ #define X86_FEATURE_AMD_SSBD (13*32+24) /* "" Speculative Store Bypass Di= sable */ #define X86_FEATURE_VIRT_SSBD (13*32+25) /* Virtualized Speculative Store= Bypass Disable */ #define X86_FEATURE_AMD_SSB_NO (13*32+26) /* "" Speculative Store Bypass = is fixed in hardware. */ +#define X86_FEATURE_BTC_NO (13*32+29) /* "" Not vulnerable to Branch Type= Confusion */ =20 /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 1= 4 */ #define X86_FEATURE_DTHERM (14*32+ 0) /* Digital Thermal Sensor */ --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -942,12 +942,21 @@ static void init_amd_zn(struct cpuinfo_x node_reclaim_distance =3D 32; #endif =20 - /* - * Fix erratum 1076: CPB feature bit not being set in CPUID. - * Always set it, except when running under a hypervisor. - */ - if (!cpu_has(c, X86_FEATURE_HYPERVISOR) && !cpu_has(c, X86_FEATURE_CPB)) - set_cpu_cap(c, X86_FEATURE_CPB); + /* Fix up CPUID bits, but only if not virtualised. */ + if (!cpu_has(c, X86_FEATURE_HYPERVISOR)) { + + /* Erratum 1076: CPB feature bit not being set in CPUID. */ + if (!cpu_has(c, X86_FEATURE_CPB)) + set_cpu_cap(c, X86_FEATURE_CPB); + + /* + * Zen3 (Fam19 model < 0x10) parts are not susceptible to + * Branch Type Confusion, but predate the allocation of the + * BTC_NO bit. + */ + if (c->x86 =3D=3D 0x19 && !cpu_has(c, X86_FEATURE_BTC_NO)) + set_cpu_cap(c, X86_FEATURE_BTC_NO); + } } =20 static void init_amd(struct cpuinfo_x86 *c) --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1246,8 +1246,10 @@ static void __init cpu_set_bug_bits(stru !arch_cap_mmio_immune(ia32_cap)) setup_force_cpu_bug(X86_BUG_MMIO_STALE_DATA); =20 - if ((cpu_matches(cpu_vuln_blacklist, RETBLEED) || (ia32_cap & ARCH_CAP_RS= BA))) - setup_force_cpu_bug(X86_BUG_RETBLEED); + if (!cpu_has(c, X86_FEATURE_BTC_NO)) { + if (cpu_matches(cpu_vuln_blacklist, RETBLEED) || (ia32_cap & ARCH_CAP_RS= BA)) + setup_force_cpu_bug(X86_BUG_RETBLEED); + } =20 if (cpu_matches(cpu_vuln_whitelist, NO_MELTDOWN)) return; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F101C433EF for ; Sat, 23 Jul 2022 10:11:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239247AbiGWKLc (ORCPT ); Sat, 23 Jul 2022 06:11:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239287AbiGWKJc (ORCPT ); Sat, 23 Jul 2022 06:09:32 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1E8865D44; Sat, 23 Jul 2022 03:02:39 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 65E2860C81; Sat, 23 Jul 2022 10:02:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67525C341C7; Sat, 23 Jul 2022 10:02:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570550; bh=TAV/wnTx091VIu0LMF/c2tWYw4c6o5UNxdxx+9tq7ws=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DiDjNep+hx2+gvJcZ9KXCkUH+gWxLfWQaKfNTadTY7CKyTSb97zt+CtlfqKUDb8H4 MxHrcZh555GTi9LU+nj1sSe+gnqcS8wAzFuiu4myVHk5YDg3ciuUDlgvJv47OWWxaV UYwJ5RB4YpdIwaCONWOLg+5fahkT25myogjbJ7Mw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Thadeu Lima de Souza Cascardo , Ben Hutchings Subject: [PATCH 5.10 125/148] x86/retbleed: Add fine grained Kconfig knobs Date: Sat, 23 Jul 2022 11:55:37 +0200 Message-Id: <20220723095259.304261366@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit f43b9876e857c739d407bc56df288b0ebe1a9164 upstream. Do fine-grained Kconfig for all the various retbleed parts. NOTE: if your compiler doesn't support return thunks this will silently 'upgrade' your mitigation to IBPB, you might not like this. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov [cascardo: there is no CONFIG_OBJTOOL] [cascardo: objtool calling and option parsing has changed] Signed-off-by: Thadeu Lima de Souza Cascardo [bwh: Backported to 5.10: - In scripts/Makefile.build, add the objtool option with an ifdef block, same as for other options - Adjust filename, context] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Makefile | 8 +- arch/x86/Kconfig | 106 +++++++++++++++++++++++---= ----- arch/x86/entry/calling.h | 4 + arch/x86/include/asm/disabled-features.h | 18 ++++- arch/x86/include/asm/linkage.h | 4 - arch/x86/include/asm/nospec-branch.h | 10 ++ arch/x86/include/asm/static_call.h | 2=20 arch/x86/kernel/alternative.c | 5 + arch/x86/kernel/cpu/amd.c | 2=20 arch/x86/kernel/cpu/bugs.c | 42 +++++++----- arch/x86/kernel/static_call.c | 2=20 arch/x86/kvm/emulate.c | 4 - arch/x86/lib/retpoline.S | 4 + scripts/Makefile.build | 3=20 scripts/link-vmlinux.sh | 2=20 security/Kconfig | 11 --- tools/objtool/builtin-check.c | 3=20 tools/objtool/builtin.h | 2=20 tools/objtool/check.c | 9 ++ 19 files changed, 172 insertions(+), 69 deletions(-) --- a/Makefile +++ b/Makefile @@ -672,14 +672,18 @@ endif =20 ifdef CONFIG_CC_IS_GCC RETPOLINE_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-extern -m= indirect-branch-register) -RETPOLINE_CFLAGS +=3D $(call cc-option,-mfunction-return=3Dthunk-extern) RETPOLINE_VDSO_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-inli= ne -mindirect-branch-register) endif ifdef CONFIG_CC_IS_CLANG RETPOLINE_CFLAGS :=3D -mretpoline-external-thunk RETPOLINE_VDSO_CFLAGS :=3D -mretpoline -RETPOLINE_CFLAGS +=3D $(call cc-option,-mfunction-return=3Dthunk-extern) endif + +ifdef CONFIG_RETHUNK +RETHUNK_CFLAGS :=3D -mfunction-return=3Dthunk-extern +RETPOLINE_CFLAGS +=3D $(RETHUNK_CFLAGS) +endif + export RETPOLINE_CFLAGS export RETPOLINE_VDSO_CFLAGS =20 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -453,30 +453,6 @@ config GOLDFISH def_bool y depends on X86_GOLDFISH =20 -config RETPOLINE - bool "Avoid speculative indirect branches in kernel" - default y - help - Compile kernel with the retpoline compiler options to guard against - kernel-to-user data leaks by avoiding speculative indirect - branches. Requires a compiler with -mindirect-branch=3Dthunk-extern - support for full protection. The kernel may run slower. - -config CC_HAS_SLS - def_bool $(cc-option,-mharden-sls=3Dall) - -config CC_HAS_RETURN_THUNK - def_bool $(cc-option,-mfunction-return=3Dthunk-extern) - -config SLS - bool "Mitigate Straight-Line-Speculation" - depends on CC_HAS_SLS && X86_64 - default n - help - Compile the kernel with straight-line-speculation options to guard - against straight line speculation. The kernel image might be slightly - larger. - config X86_CPU_RESCTRL bool "x86 CPU resource control support" depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD) @@ -2430,6 +2406,88 @@ source "kernel/livepatch/Kconfig" =20 endmenu =20 +config CC_HAS_SLS + def_bool $(cc-option,-mharden-sls=3Dall) + +config CC_HAS_RETURN_THUNK + def_bool $(cc-option,-mfunction-return=3Dthunk-extern) + +menuconfig SPECULATION_MITIGATIONS + bool "Mitigations for speculative execution vulnerabilities" + default y + help + Say Y here to enable options which enable mitigations for + speculative execution hardware vulnerabilities. + + If you say N, all mitigations will be disabled. You really + should know what you are doing to say so. + +if SPECULATION_MITIGATIONS + +config PAGE_TABLE_ISOLATION + bool "Remove the kernel mapping in user mode" + default y + depends on (X86_64 || X86_PAE) + help + This feature reduces the number of hardware side channels by + ensuring that the majority of kernel addresses are not mapped + into userspace. + + See Documentation/x86/pti.rst for more details. + +config RETPOLINE + bool "Avoid speculative indirect branches in kernel" + default y + help + Compile kernel with the retpoline compiler options to guard against + kernel-to-user data leaks by avoiding speculative indirect + branches. Requires a compiler with -mindirect-branch=3Dthunk-extern + support for full protection. The kernel may run slower. + +config RETHUNK + bool "Enable return-thunks" + depends on RETPOLINE && CC_HAS_RETURN_THUNK + default y + help + Compile the kernel with the return-thunks compiler option to guard + against kernel-to-user data leaks by avoiding return speculation. + Requires a compiler with -mfunction-return=3Dthunk-extern + support for full protection. The kernel may run slower. + +config CPU_UNRET_ENTRY + bool "Enable UNRET on kernel entry" + depends on CPU_SUP_AMD && RETHUNK + default y + help + Compile the kernel with support for the retbleed=3Dunret mitigation. + +config CPU_IBPB_ENTRY + bool "Enable IBPB on kernel entry" + depends on CPU_SUP_AMD + default y + help + Compile the kernel with support for the retbleed=3Dibpb mitigation. + +config CPU_IBRS_ENTRY + bool "Enable IBRS on kernel entry" + depends on CPU_SUP_INTEL + default y + help + Compile the kernel with support for the spectre_v2=3Dibrs mitigation. + This mitigates both spectre_v2 and retbleed at great cost to + performance. + +config SLS + bool "Mitigate Straight-Line-Speculation" + depends on CC_HAS_SLS && X86_64 + default n + help + Compile the kernel with straight-line-speculation options to guard + against straight line speculation. The kernel image might be slightly + larger. + +endif + config ARCH_HAS_ADD_PAGES def_bool y depends on X86_64 && ARCH_ENABLE_MEMORY_HOTPLUG --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -323,6 +323,7 @@ For 32-bit we have the following convent * Assumes x86_spec_ctrl_{base,current} to have SPEC_CTRL_IBRS set. */ .macro IBRS_ENTER save_reg +#ifdef CONFIG_CPU_IBRS_ENTRY ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_KERNEL_IBRS movl $MSR_IA32_SPEC_CTRL, %ecx =20 @@ -343,6 +344,7 @@ For 32-bit we have the following convent shr $32, %rdx wrmsr .Lend_\@: +#endif .endm =20 /* @@ -350,6 +352,7 @@ For 32-bit we have the following convent * regs. Must be called after the last RET. */ .macro IBRS_EXIT save_reg +#ifdef CONFIG_CPU_IBRS_ENTRY ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_KERNEL_IBRS movl $MSR_IA32_SPEC_CTRL, %ecx =20 @@ -364,6 +367,7 @@ For 32-bit we have the following convent shr $32, %rdx wrmsr .Lend_\@: +#endif .endm =20 /* --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -60,9 +60,19 @@ # define DISABLE_RETPOLINE 0 #else # define DISABLE_RETPOLINE ((1 << (X86_FEATURE_RETPOLINE & 31)) | \ - (1 << (X86_FEATURE_RETPOLINE_LFENCE & 31)) | \ - (1 << (X86_FEATURE_RETHUNK & 31)) | \ - (1 << (X86_FEATURE_UNRET & 31))) + (1 << (X86_FEATURE_RETPOLINE_LFENCE & 31))) +#endif + +#ifdef CONFIG_RETHUNK +# define DISABLE_RETHUNK 0 +#else +# define DISABLE_RETHUNK (1 << (X86_FEATURE_RETHUNK & 31)) +#endif + +#ifdef CONFIG_CPU_UNRET_ENTRY +# define DISABLE_UNRET 0 +#else +# define DISABLE_UNRET (1 << (X86_FEATURE_UNRET & 31)) #endif =20 /* Force disable because it's broken beyond repair */ @@ -82,7 +92,7 @@ #define DISABLED_MASK8 0 #define DISABLED_MASK9 (DISABLE_SMAP) #define DISABLED_MASK10 0 -#define DISABLED_MASK11 (DISABLE_RETPOLINE) +#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET) #define DISABLED_MASK12 0 #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -18,7 +18,7 @@ #define __ALIGN_STR __stringify(__ALIGN) #endif =20 -#if defined(CONFIG_RETPOLINE) && !defined(__DISABLE_EXPORTS) && !defined(B= UILD_VDSO) +#if defined(CONFIG_RETHUNK) && !defined(__DISABLE_EXPORTS) && !defined(BUI= LD_VDSO) #define RET jmp __x86_return_thunk #else /* CONFIG_RETPOLINE */ #ifdef CONFIG_SLS @@ -30,7 +30,7 @@ =20 #else /* __ASSEMBLY__ */ =20 -#if defined(CONFIG_RETPOLINE) && !defined(__DISABLE_EXPORTS) && !defined(B= UILD_VDSO) +#if defined(CONFIG_RETHUNK) && !defined(__DISABLE_EXPORTS) && !defined(BUI= LD_VDSO) #define ASM_RET "jmp __x86_return_thunk\n\t" #else /* CONFIG_RETPOLINE */ #ifdef CONFIG_SLS --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -127,6 +127,12 @@ .Lskip_rsb_\@: .endm =20 +#ifdef CONFIG_CPU_UNRET_ENTRY +#define CALL_ZEN_UNTRAIN_RET "call zen_untrain_ret" +#else +#define CALL_ZEN_UNTRAIN_RET "" +#endif + /* * Mitigate RETBleed for AMD/Hygon Zen uarch. Requires KERNEL CR3 because = the * return thunk isn't mapped into the userspace tables (then again, AMD @@ -139,10 +145,10 @@ * where we have a stack but before any RET instruction. */ .macro UNTRAIN_RET -#ifdef CONFIG_RETPOLINE +#if defined(CONFIG_CPU_UNRET_ENTRY) || defined(CONFIG_CPU_IBPB_ENTRY) ANNOTATE_UNRET_END ALTERNATIVE_2 "", \ - "call zen_untrain_ret", X86_FEATURE_UNRET, \ + CALL_ZEN_UNTRAIN_RET, X86_FEATURE_UNRET, \ "call entry_ibpb", X86_FEATURE_ENTRY_IBPB #endif .endm --- a/arch/x86/include/asm/static_call.h +++ b/arch/x86/include/asm/static_call.h @@ -44,7 +44,7 @@ #define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \ __ARCH_DEFINE_STATIC_CALL_TRAMP(name, ".byte 0xe9; .long " #func " - (. += 4)") =20 -#ifdef CONFIG_RETPOLINE +#ifdef CONFIG_RETHUNK #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \ __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "jmp __x86_return_thunk") #else --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -662,6 +662,7 @@ void __init_or_module noinline apply_ret } } =20 +#ifdef CONFIG_RETHUNK /* * Rewrite the compiler generated return thunk tail-calls. * @@ -723,6 +724,10 @@ void __init_or_module noinline apply_ret } } } +#else +void __init_or_module noinline apply_returns(s32 *start, s32 *end) { } +#endif /* CONFIG_RETHUNK */ + #else /* !RETPOLINES || !CONFIG_STACK_VALIDATION */ =20 void __init_or_module noinline apply_retpolines(s32 *start, s32 *end) { } --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -916,6 +916,7 @@ static void init_amd_bd(struct cpuinfo_x =20 void init_spectral_chicken(struct cpuinfo_x86 *c) { +#ifdef CONFIG_CPU_UNRET_ENTRY u64 value; =20 /* @@ -932,6 +933,7 @@ void init_spectral_chicken(struct cpuinf wrmsrl_safe(MSR_ZEN2_SPECTRAL_CHICKEN, value); } } +#endif } =20 static void init_amd_zn(struct cpuinfo_x86 *c) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -798,7 +798,6 @@ static int __init retbleed_parse_cmdline early_param("retbleed", retbleed_parse_cmdline); =20 #define RETBLEED_UNTRAIN_MSG "WARNING: BTB untrained return thunk mitigati= on is only effective on AMD/Hygon!\n" -#define RETBLEED_COMPILER_MSG "WARNING: kernel not compiled with RETPOLINE= or -mfunction-return capable compiler; falling back to IBPB!\n" #define RETBLEED_INTEL_MSG "WARNING: Spectre v2 mitigation leaves CPU vuln= erable to RETBleed attacks, data leaks possible!\n" =20 static void __init retbleed_select_mitigation(void) @@ -813,18 +812,33 @@ static void __init retbleed_select_mitig return; =20 case RETBLEED_CMD_UNRET: - retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + if (IS_ENABLED(CONFIG_CPU_UNRET_ENTRY)) { + retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + } else { + pr_err("WARNING: kernel not compiled with CPU_UNRET_ENTRY.\n"); + goto do_cmd_auto; + } break; =20 case RETBLEED_CMD_IBPB: - retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; + if (IS_ENABLED(CONFIG_CPU_IBPB_ENTRY)) { + retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; + } else { + pr_err("WARNING: kernel not compiled with CPU_IBPB_ENTRY.\n"); + goto do_cmd_auto; + } break; =20 +do_cmd_auto: case RETBLEED_CMD_AUTO: default: if (boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_AMD || - boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) - retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) { + if (IS_ENABLED(CONFIG_CPU_UNRET_ENTRY)) + retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; + else if (IS_ENABLED(CONFIG_CPU_IBPB_ENTRY)) + retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; + } =20 /* * The Intel mitigation (IBRS or eIBRS) was already selected in @@ -837,14 +851,6 @@ static void __init retbleed_select_mitig =20 switch (retbleed_mitigation) { case RETBLEED_MITIGATION_UNRET: - - if (!IS_ENABLED(CONFIG_RETPOLINE) || - !IS_ENABLED(CONFIG_CC_HAS_RETURN_THUNK)) { - pr_err(RETBLEED_COMPILER_MSG); - retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; - goto retbleed_force_ibpb; - } - setup_force_cpu_cap(X86_FEATURE_RETHUNK); setup_force_cpu_cap(X86_FEATURE_UNRET); =20 @@ -856,7 +862,6 @@ static void __init retbleed_select_mitig break; =20 case RETBLEED_MITIGATION_IBPB: -retbleed_force_ibpb: setup_force_cpu_cap(X86_FEATURE_ENTRY_IBPB); mitigate_smt =3D true; break; @@ -1227,6 +1232,12 @@ static enum spectre_v2_mitigation_cmd __ return SPECTRE_V2_CMD_AUTO; } =20 + if (cmd =3D=3D SPECTRE_V2_CMD_IBRS && !IS_ENABLED(CONFIG_CPU_IBRS_ENTRY))= { + pr_err("%s selected but not compiled in. Switching to AUTO select\n", + mitigation_options[i].option); + return SPECTRE_V2_CMD_AUTO; + } + if (cmd =3D=3D SPECTRE_V2_CMD_IBRS && boot_cpu_data.x86_vendor !=3D X86_V= ENDOR_INTEL) { pr_err("%s selected but not Intel CPU. Switching to AUTO select\n", mitigation_options[i].option); @@ -1284,7 +1295,8 @@ static void __init spectre_v2_select_mit break; } =20 - if (boot_cpu_has_bug(X86_BUG_RETBLEED) && + if (IS_ENABLED(CONFIG_CPU_IBRS_ENTRY) && + boot_cpu_has_bug(X86_BUG_RETBLEED) && retbleed_cmd !=3D RETBLEED_CMD_OFF && boot_cpu_has(X86_FEATURE_IBRS) && boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_INTEL) { --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -108,7 +108,7 @@ void arch_static_call_transform(void *si } EXPORT_SYMBOL_GPL(arch_static_call_transform); =20 -#ifdef CONFIG_RETPOLINE +#ifdef CONFIG_RETHUNK /* * This is called by apply_returns() to fix up static call trampolines, * specifically ARCH_DEFINE_STATIC_CALL_NULL_TRAMP which is recorded as --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -435,10 +435,10 @@ static int fastop(struct x86_emulate_ctx * Depending on .config the SETcc functions look like: * * SETcc %al [3 bytes] - * RET | JMP __x86_return_thunk [1,5 bytes; CONFIG_RETPOLINE] + * RET | JMP __x86_return_thunk [1,5 bytes; CONFIG_RETHUNK] * INT3 [1 byte; CONFIG_SLS] */ -#define RET_LENGTH (1 + (4 * IS_ENABLED(CONFIG_RETPOLINE)) + \ +#define RET_LENGTH (1 + (4 * IS_ENABLED(CONFIG_RETHUNK)) + \ IS_ENABLED(CONFIG_SLS)) #define SETCC_LENGTH (3 + RET_LENGTH) #define SETCC_ALIGN (4 << ((SETCC_LENGTH > 4) & 1) << ((SETCC_LENGTH > 8) = & 1)) --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -71,6 +71,8 @@ SYM_CODE_END(__x86_indirect_thunk_array) * This function name is magical and is used by -mfunction-return=3Dthunk-= extern * for the compiler to generate JMPs to it. */ +#ifdef CONFIG_RETHUNK + .section .text.__x86.return_thunk =20 /* @@ -135,3 +137,5 @@ SYM_FUNC_END(zen_untrain_ret) __EXPORT_THUNK(zen_untrain_ret) =20 EXPORT_SYMBOL(__x86_return_thunk) + +#endif /* CONFIG_RETHUNK */ --- a/scripts/Makefile.build +++ b/scripts/Makefile.build @@ -227,6 +227,9 @@ endif ifdef CONFIG_RETPOLINE objtool_args +=3D --retpoline endif +ifdef CONFIG_RETHUNK + objtool_args +=3D --rethunk +endif ifdef CONFIG_X86_SMAP objtool_args +=3D --uaccess endif --- a/scripts/link-vmlinux.sh +++ b/scripts/link-vmlinux.sh @@ -65,7 +65,7 @@ objtool_link() =20 if [ -n "${CONFIG_VMLINUX_VALIDATION}" ]; then objtoolopt=3D"check" - if [ -n "${CONFIG_RETPOLINE}" ]; then + if [ -n "${CONFIG_CPU_UNRET_ENTRY}" ]; then objtoolopt=3D"${objtoolopt} --unret" fi if [ -z "${CONFIG_FRAME_POINTER}" ]; then --- a/security/Kconfig +++ b/security/Kconfig @@ -54,17 +54,6 @@ config SECURITY_NETWORK implement socket and networking access controls. If you are unsure how to answer this question, answer N. =20 -config PAGE_TABLE_ISOLATION - bool "Remove the kernel mapping in user mode" - default y - depends on (X86_64 || X86_PAE) && !UML - help - This feature reduces the number of hardware side channels by - ensuring that the majority of kernel addresses are not mapped - into userspace. - - See Documentation/x86/pti.rst for more details. - config SECURITY_INFINIBAND bool "Infiniband Security Hooks" depends on SECURITY && INFINIBAND --- a/tools/objtool/builtin-check.c +++ b/tools/objtool/builtin-check.c @@ -19,7 +19,7 @@ #include "objtool.h" =20 bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, stats, - validate_dup, vmlinux, sls, unret; + validate_dup, vmlinux, sls, unret, rethunk; =20 static const char * const check_usage[] =3D { "objtool check [] file.o", @@ -30,6 +30,7 @@ const struct option check_options[] =3D { OPT_BOOLEAN('f', "no-fp", &no_fp, "Skip frame pointer validation"), OPT_BOOLEAN('u', "no-unreachable", &no_unreachable, "Skip 'unreachable in= struction' warnings"), OPT_BOOLEAN('r', "retpoline", &retpoline, "Validate retpoline assumptions= "), + OPT_BOOLEAN(0, "rethunk", &rethunk, "validate and annotate rethunk usag= e"), OPT_BOOLEAN(0, "unret", &unret, "validate entry unret placement"), OPT_BOOLEAN('m', "module", &module, "Indicates the object will be part of= a kernel module"), OPT_BOOLEAN('b', "backtrace", &backtrace, "unwind on error"), --- a/tools/objtool/builtin.h +++ b/tools/objtool/builtin.h @@ -9,7 +9,7 @@ =20 extern const struct option check_options[]; extern bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess, = stats, - validate_dup, vmlinux, sls, unret; + validate_dup, vmlinux, sls, unret, rethunk; =20 extern int cmd_check(int argc, const char **argv); extern int cmd_orc(int argc, const char **argv); --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -3262,8 +3262,11 @@ static int validate_retpoline(struct obj continue; =20 if (insn->type =3D=3D INSN_RETURN) { - WARN_FUNC("'naked' return found in RETPOLINE build", - insn->sec, insn->offset); + if (rethunk) { + WARN_FUNC("'naked' return found in RETHUNK build", + insn->sec, insn->offset); + } else + continue; } else { WARN_FUNC("indirect %s found in RETPOLINE build", insn->sec, insn->offset, @@ -3533,7 +3536,9 @@ int check(struct objtool_file *file) if (ret < 0) goto out; warnings +=3D ret; + } =20 + if (rethunk) { ret =3D create_return_sites_sections(file); if (ret < 0) goto out; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DEE6C433EF for ; Sat, 23 Jul 2022 10:12:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239435AbiGWKMX (ORCPT ); Sat, 23 Jul 2022 06:12:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239188AbiGWKJX (ORCPT ); Sat, 23 Jul 2022 06:09:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A95CDCBD11; Sat, 23 Jul 2022 03:02:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 55DFA61274; Sat, 23 Jul 2022 10:02:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FC1EC341C0; Sat, 23 Jul 2022 10:02:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570553; bh=o4/srVdodmfnh4pywnFMEK+V/2gNOqg0RGT2S3PRrWE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RTTBsgfo0Pkl7aJ0TN341qKH8Myk0hcAtWOVrDZXySyHOvbAv0vwC77XO9Uv6SKBG jC8x5RGZ70V+5huEeBiLvHmBsBAUB0cWLGLDNmgcd4VOL1IQ0HVaoueeS+4seJqlkb rdQ82DqlM1493eqeuEPWhKnyPlS3j9zjXS9hVstU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Pawan Gupta , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 126/148] x86/bugs: Add Cannon lake to RETBleed affected CPU list Date: Sat, 23 Jul 2022 11:55:38 +0200 Message-Id: <20220723095259.574769949@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Pawan Gupta commit f54d45372c6ac9c993451de5e51312485f7d10bc upstream. Cannon lake is also affected by RETBleed, add it to the list. Fixes: 6ad0ad2bf8a6 ("x86/bugs: Report Intel retbleed vulnerability") Signed-off-by: Pawan Gupta Signed-off-by: Borislav Petkov Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/common.c | 1 + 1 file changed, 1 insertion(+) --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1129,6 +1129,7 @@ static const struct x86_cpu_id cpu_vuln_ VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPING_ANY, SRBDS | MMIO | RETBLE= ED), VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPING_ANY, SRBDS | MMIO | RETB= LEED), VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPING_ANY, SRBDS | MMIO | RETBLE= ED), + VULNBL_INTEL_STEPPINGS(CANNONLAKE_L, X86_STEPPING_ANY, RETBLEED), VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | R= ETBLEED), VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPING_ANY, MMIO), VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPING_ANY, MMIO), From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2409C433EF for ; Sat, 23 Jul 2022 10:11:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239276AbiGWKLj (ORCPT ); Sat, 23 Jul 2022 06:11:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239341AbiGWKJe (ORCPT ); Sat, 23 Jul 2022 06:09:34 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1709CCB74A; Sat, 23 Jul 2022 03:02:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 25EEF61212; Sat, 23 Jul 2022 10:02:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 307DDC341C0; Sat, 23 Jul 2022 10:02:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570556; bh=MwuU+xseDp9KXWFIrzDY8KulQryfkv/g26Hg1/eETvA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pLtwvxjw3ATFyH96Gh6I2MUbkPBwX89HSvoFmUkPijF6w9pMeNvuwbyNIEftdX2LE m7K46n4GN2TNeZX4CyW2bbXNRt+b6CYqCpno9u96UW9rXx7AT4Scs7179TC5GflHVp E3Kda3NPF9fHQlguZnIV+qypbXxZOqBSzetDCcJk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Thadeu Lima de Souza Cascardo , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 127/148] x86/bugs: Do not enable IBPB-on-entry when IBPB is not supported Date: Sat, 23 Jul 2022 11:55:39 +0200 Message-Id: <20220723095259.851395210@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thadeu Lima de Souza Cascardo commit 2259da159fbe5dba8ac00b560cf00b6a6537fa18 upstream. There are some VM configurations which have Skylake model but do not support IBPB. In those cases, when using retbleed=3Dibpb, userspace is going to be killed and kernel is going to panic. If the CPU does not support IBPB, warn and proceed with the auto option. Al= so, do not fallback to IBPB on AMD/Hygon systems if it is not supported. Fixes: 3ebc17006888 ("x86/bugs: Add retbleed=3Dibpb") Signed-off-by: Thadeu Lima de Souza Cascardo Signed-off-by: Borislav Petkov Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/bugs.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -821,7 +821,10 @@ static void __init retbleed_select_mitig break; =20 case RETBLEED_CMD_IBPB: - if (IS_ENABLED(CONFIG_CPU_IBPB_ENTRY)) { + if (!boot_cpu_has(X86_FEATURE_IBPB)) { + pr_err("WARNING: CPU does not support IBPB.\n"); + goto do_cmd_auto; + } else if (IS_ENABLED(CONFIG_CPU_IBPB_ENTRY)) { retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; } else { pr_err("WARNING: kernel not compiled with CPU_IBPB_ENTRY.\n"); @@ -836,7 +839,7 @@ do_cmd_auto: boot_cpu_data.x86_vendor =3D=3D X86_VENDOR_HYGON) { if (IS_ENABLED(CONFIG_CPU_UNRET_ENTRY)) retbleed_mitigation =3D RETBLEED_MITIGATION_UNRET; - else if (IS_ENABLED(CONFIG_CPU_IBPB_ENTRY)) + else if (IS_ENABLED(CONFIG_CPU_IBPB_ENTRY) && boot_cpu_has(X86_FEATURE_= IBPB)) retbleed_mitigation =3D RETBLEED_MITIGATION_IBPB; } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC9C8C433EF for ; Sat, 23 Jul 2022 10:12:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238656AbiGWKMa (ORCPT ); Sat, 23 Jul 2022 06:12:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239373AbiGWKJf (ORCPT ); Sat, 23 Jul 2022 06:09:35 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 954C58149C; Sat, 23 Jul 2022 03:02:43 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B0BE2B82C1A; Sat, 23 Jul 2022 10:02:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23109C341C7; Sat, 23 Jul 2022 10:02:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570559; bh=6ZJiZorj7WVCzB1nyaRhoBZ6f67OnAAunAmOUTsjtKA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EFNio/uUjUMiCyS+KCVhcEHeYEoE84p7CMbRtl7qkYkwWS60lHpcwLsOlMNb+PmQ5 8VH3ADEUXtI7Zq96YvDPYGe4pqlbQJ0nbPN+e/Dd5eCfI2nuNaim4RA3nMSIUPkKYC 1C9d7fL5GsXhUEYxpbVN6D1Q+sBDRK14FuDg35x8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Edward Tran , Awais Tanveer , Ankur Arora , Konrad Rzeszutek Wilk , Alexandre Chartre , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 128/148] x86/kexec: Disable RET on kexec Date: Sat, 23 Jul 2022 11:55:40 +0200 Message-Id: <20220723095300.164230742@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Konrad Rzeszutek Wilk commit 697977d8415d61f3acbc4ee6d564c9dcf0309507 upstream. All the invocations unroll to __x86_return_thunk and this file must be PIC independent. This fixes kexec on 64-bit AMD boxes. [ bp: Fix 32-bit build. ] Reported-by: Edward Tran Reported-by: Awais Tanveer Suggested-by: Ankur Arora Signed-off-by: Konrad Rzeszutek Wilk Signed-off-by: Alexandre Chartre Signed-off-by: Borislav Petkov Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/relocate_kernel_32.S | 25 +++++++++++++++++++------ arch/x86/kernel/relocate_kernel_64.S | 23 +++++++++++++++++------ 2 files changed, 36 insertions(+), 12 deletions(-) --- a/arch/x86/kernel/relocate_kernel_32.S +++ b/arch/x86/kernel/relocate_kernel_32.S @@ -7,10 +7,12 @@ #include #include #include +#include #include =20 /* - * Must be relocatable PIC code callable as a C function + * Must be relocatable PIC code callable as a C function, in particular + * there must be a plain RET and not jump to return thunk. */ =20 #define PTR(x) (x << 2) @@ -91,7 +93,9 @@ SYM_CODE_START_NOALIGN(relocate_kernel) movl %edi, %eax addl $(identity_mapped - relocate_kernel), %eax pushl %eax - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(relocate_kernel) =20 SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) @@ -159,12 +163,15 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma xorl %edx, %edx xorl %esi, %esi xorl %ebp, %ebp - RET + ANNOTATE_UNRET_SAFE + ret + int3 1: popl %edx movl CP_PA_SWAP_PAGE(%edi), %esp addl $PAGE_SIZE, %esp 2: + ANNOTATE_RETPOLINE_SAFE call *%edx =20 /* get the re-entry point of the peer system */ @@ -190,7 +197,9 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma movl %edi, %eax addl $(virtual_mapped - relocate_kernel), %eax pushl %eax - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(identity_mapped) =20 SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped) @@ -208,7 +217,9 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_map popl %edi popl %esi popl %ebx - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(virtual_mapped) =20 /* Do the copies */ @@ -271,7 +282,9 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages) popl %edi popl %ebx popl %ebp - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(swap_pages) =20 .globl kexec_control_code_size --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -13,7 +13,8 @@ #include =20 /* - * Must be relocatable PIC code callable as a C function + * Must be relocatable PIC code callable as a C function, in particular + * there must be a plain RET and not jump to return thunk. */ =20 #define PTR(x) (x << 3) @@ -104,7 +105,9 @@ SYM_CODE_START_NOALIGN(relocate_kernel) /* jump to identity mapped page */ addq $(identity_mapped - relocate_kernel), %r8 pushq %r8 - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(relocate_kernel) =20 SYM_CODE_START_LOCAL_NOALIGN(identity_mapped) @@ -191,7 +194,9 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma xorl %r14d, %r14d xorl %r15d, %r15d =20 - RET + ANNOTATE_UNRET_SAFE + ret + int3 =20 1: popq %rdx @@ -210,7 +215,9 @@ SYM_CODE_START_LOCAL_NOALIGN(identity_ma call swap_pages movq $virtual_mapped, %rax pushq %rax - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(identity_mapped) =20 SYM_CODE_START_LOCAL_NOALIGN(virtual_mapped) @@ -231,7 +238,9 @@ SYM_CODE_START_LOCAL_NOALIGN(virtual_map popq %r12 popq %rbp popq %rbx - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(virtual_mapped) =20 /* Do the copies */ @@ -288,7 +297,9 @@ SYM_CODE_START_LOCAL_NOALIGN(swap_pages) lea PAGE_SIZE(%rax), %rsi jmp 0b 3: - RET + ANNOTATE_UNRET_SAFE + ret + int3 SYM_CODE_END(swap_pages) =20 .globl kexec_control_code_size From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A1ABC43334 for ; Sat, 23 Jul 2022 10:11:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239290AbiGWKLo (ORCPT ); Sat, 23 Jul 2022 06:11:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239372AbiGWKJf (ORCPT ); Sat, 23 Jul 2022 06:09:35 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9531662A54; Sat, 23 Jul 2022 03:02:43 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CB74E611CD; Sat, 23 Jul 2022 10:02:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D4789C341C7; Sat, 23 Jul 2022 10:02:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570562; bh=qPIjBdiN/MqGqMuMKa8Tb9yC09kSZgBbUyIlb/5RA2Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fGCuVRHRsLGkO9dl2pZM7XINg7lZtHQIk6IzlCdxDq+BG0+IORm3d1tal8rfLjZHw yTxXszWmSM+laVSXsldzflQB0N3uhO6nCSrqsSEzN5Py8JN2V6CRINUEtXxJZH9sLn vhcPjVEV/v6Cu/oMloDwyekpnh1fwiu9bUvjaw/Y= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Pawan Gupta , Borislav Petkov , Ben Hutchings Subject: [PATCH 5.10 129/148] x86/speculation: Disable RRSBA behavior Date: Sat, 23 Jul 2022 11:55:41 +0200 Message-Id: <20220723095300.478177770@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Pawan Gupta commit 4ad3278df6fe2b0852b00d5757fc2ccd8e92c26e upstream. Some Intel processors may use alternate predictors for RETs on RSB-underflow. This condition may be vulnerable to Branch History Injection (BHI) and intramode-BTI. Kernel earlier added spectre_v2 mitigation modes (eIBRS+Retpolines, eIBRS+LFENCE, Retpolines) which protect indirect CALLs and JMPs against such attacks. However, on RSB-underflow, RET target prediction may fallback to alternate predictors. As a result, RET's predicted target may get influenced by branch history. A new MSR_IA32_SPEC_CTRL bit (RRSBA_DIS_S) controls this fallback behavior when in kernel mode. When set, RETs will not take predictions from alternate predictors, hence mitigating RETs as well. Support for this is enumerated by CPUID.7.2.EDX[RRSBA_CTRL] (bit2). For spectre v2 mitigation, when a user selects a mitigation that protects indirect CALLs and JMPs against BHI and intramode-BTI, set RRSBA_DIS_S also to protect RETs for RSB-underflow case. Signed-off-by: Pawan Gupta Signed-off-by: Borislav Petkov [bwh: Backported to 5.15: adjust context in scattered.c] Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/msr-index.h | 9 +++++++++ arch/x86/kernel/cpu/bugs.c | 26 ++++++++++++++++++++++++++ arch/x86/kernel/cpu/scattered.c | 1 + tools/arch/x86/include/asm/msr-index.h | 9 +++++++++ 5 files changed, 46 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -293,7 +293,7 @@ /* FREE! (11*32+ 8) */ /* FREE! (11*32+ 9) */ #define X86_FEATURE_ENTRY_IBPB (11*32+10) /* "" Issue an IBPB on kernel e= ntry */ -/* FREE! (11*32+11) */ +#define X86_FEATURE_RRSBA_CTRL (11*32+11) /* "" RET prediction control */ #define X86_FEATURE_RETPOLINE (11*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ #define X86_FEATURE_RETPOLINE_LFENCE (11*32+13) /* "" Use LFENCE for Spect= re variant 2 */ #define X86_FEATURE_RETHUNK (11*32+14) /* "" Use REturn THUNK */ --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -51,6 +51,8 @@ #define SPEC_CTRL_STIBP BIT(SPEC_CTRL_STIBP_SHIFT) /* STIBP mask */ #define SPEC_CTRL_SSBD_SHIFT 2 /* Speculative Store Bypass Disable bit= */ #define SPEC_CTRL_SSBD BIT(SPEC_CTRL_SSBD_SHIFT) /* Speculative Store By= pass Disable */ +#define SPEC_CTRL_RRSBA_DIS_S_SHIFT 6 /* Disable RRSBA behavior */ +#define SPEC_CTRL_RRSBA_DIS_S BIT(SPEC_CTRL_RRSBA_DIS_S_SHIFT) =20 #define MSR_IA32_PRED_CMD 0x00000049 /* Prediction Command */ #define PRED_CMD_IBPB BIT(0) /* Indirect Branch Prediction Barrier */ @@ -139,6 +141,13 @@ * bit available to control VERW * behavior. */ +#define ARCH_CAP_RRSBA BIT(19) /* + * Indicates RET may use predictors + * other than the RSB. With eIBRS + * enabled predictions in kernel mode + * are restricted to targets in + * kernel. + */ =20 #define MSR_IA32_FLUSH_CMD 0x0000010b #define L1D_FLUSH BIT(0) /* --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1274,6 +1274,22 @@ static enum spectre_v2_mitigation __init return SPECTRE_V2_RETPOLINE; } =20 +/* Disable in-kernel use of non-RSB RET predictors */ +static void __init spec_ctrl_disable_kernel_rrsba(void) +{ + u64 ia32_cap; + + if (!boot_cpu_has(X86_FEATURE_RRSBA_CTRL)) + return; + + ia32_cap =3D x86_read_arch_cap_msr(); + + if (ia32_cap & ARCH_CAP_RRSBA) { + x86_spec_ctrl_base |=3D SPEC_CTRL_RRSBA_DIS_S; + write_spec_ctrl_current(x86_spec_ctrl_base, true); + } +} + static void __init spectre_v2_select_mitigation(void) { enum spectre_v2_mitigation_cmd cmd =3D spectre_v2_parse_cmdline(); @@ -1368,6 +1384,16 @@ static void __init spectre_v2_select_mit break; } =20 + /* + * Disable alternate RSB predictions in kernel when indirect CALLs and + * JMPs gets protection against BHI and Intramode-BTI, but RET + * prediction from a non-RSB predictor is still a risk. + */ + if (mode =3D=3D SPECTRE_V2_EIBRS_LFENCE || + mode =3D=3D SPECTRE_V2_EIBRS_RETPOLINE || + mode =3D=3D SPECTRE_V2_RETPOLINE) + spec_ctrl_disable_kernel_rrsba(); + spectre_v2_enabled =3D mode; pr_info("%s\n", spectre_v2_strings[mode]); =20 --- a/arch/x86/kernel/cpu/scattered.c +++ b/arch/x86/kernel/cpu/scattered.c @@ -26,6 +26,7 @@ struct cpuid_bit { static const struct cpuid_bit cpuid_bits[] =3D { { X86_FEATURE_APERFMPERF, CPUID_ECX, 0, 0x00000006, 0 }, { X86_FEATURE_EPB, CPUID_ECX, 3, 0x00000006, 0 }, + { X86_FEATURE_RRSBA_CTRL, CPUID_EDX, 2, 0x00000007, 2 }, { X86_FEATURE_CQM_LLC, CPUID_EDX, 1, 0x0000000f, 0 }, { X86_FEATURE_CQM_OCCUP_LLC, CPUID_EDX, 0, 0x0000000f, 1 }, { X86_FEATURE_CQM_MBM_TOTAL, CPUID_EDX, 1, 0x0000000f, 1 }, --- a/tools/arch/x86/include/asm/msr-index.h +++ b/tools/arch/x86/include/asm/msr-index.h @@ -51,6 +51,8 @@ #define SPEC_CTRL_STIBP BIT(SPEC_CTRL_STIBP_SHIFT) /* STIBP mask */ #define SPEC_CTRL_SSBD_SHIFT 2 /* Speculative Store Bypass Disable bit= */ #define SPEC_CTRL_SSBD BIT(SPEC_CTRL_SSBD_SHIFT) /* Speculative Store By= pass Disable */ +#define SPEC_CTRL_RRSBA_DIS_S_SHIFT 6 /* Disable RRSBA behavior */ +#define SPEC_CTRL_RRSBA_DIS_S BIT(SPEC_CTRL_RRSBA_DIS_S_SHIFT) =20 #define MSR_IA32_PRED_CMD 0x00000049 /* Prediction Command */ #define PRED_CMD_IBPB BIT(0) /* Indirect Branch Prediction Barrier */ @@ -138,6 +140,13 @@ * bit available to control VERW * behavior. */ +#define ARCH_CAP_RRSBA BIT(19) /* + * Indicates RET may use predictors + * other than the RSB. With eIBRS + * enabled predictions in kernel mode + * are restricted to targets in + * kernel. + */ =20 #define MSR_IA32_FLUSH_CMD 0x0000010b #define L1D_FLUSH BIT(0) /* From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 900A9C43334 for ; Sat, 23 Jul 2022 10:11:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239300AbiGWKLq (ORCPT ); Sat, 23 Jul 2022 06:11:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239415AbiGWKJh (ORCPT ); Sat, 23 Jul 2022 06:09:37 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1ACACBD35; Sat, 23 Jul 2022 03:02:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A4D1A611BD; Sat, 23 Jul 2022 10:02:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B2717C341C0; Sat, 23 Jul 2022 10:02:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570565; bh=UJnJ0k+B5Cwy8yaML0Wp60Wf0/bRoEx/BajCYGRGqcU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=au7I7DGM62262qjhugNmz8baT9jTIjM2Ifh7PoxJyc3eOol0epijW/OUIuD4OZKe2 RbIqVLqKMEir5kLQgkkIpLwA0Uj13FnQyzIkwLopm8hJpM0Et46FI5qYTFR8G2NS09 GNSHYRdTeFk8PVIH1FqwFyfOHE9pkeIz19hTzSmQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Thomas Gleixner , Borislav Petkov Subject: [PATCH 5.10 130/148] x86/static_call: Serialize __static_call_fixup() properly Date: Sat, 23 Jul 2022 11:55:42 +0200 Message-Id: <20220723095300.761581244@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thomas Gleixner commit c27c753ea6fd1237f4f96abf8b623d7bab505513 upstream. __static_call_fixup() invokes __static_call_transform() without holding text_mutex, which causes lockdep to complain in text_poke_bp(). Adding the proper locking cures that, but as this is either used during early boot or during module finalizing, it's not required to use text_poke_bp(). Add an argument to __static_call_transform() which tells it to use text_poke_early() for it. Fixes: ee88d363d156 ("x86,static_call: Use alternative RET encoding") Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/static_call.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -20,7 +20,8 @@ static const u8 tramp_ud[] =3D { 0x0f, 0xb =20 static const u8 retinsn[] =3D { RET_INSN_OPCODE, 0xcc, 0xcc, 0xcc, 0xcc }; =20 -static void __ref __static_call_transform(void *insn, enum insn_type type,= void *func) +static void __ref __static_call_transform(void *insn, enum insn_type type, + void *func, bool modinit) { int size =3D CALL_INSN_SIZE; const void *code; @@ -49,7 +50,7 @@ static void __ref __static_call_transfor if (memcmp(insn, code, size) =3D=3D 0) return; =20 - if (unlikely(system_state =3D=3D SYSTEM_BOOTING)) + if (system_state =3D=3D SYSTEM_BOOTING || modinit) return text_poke_early(insn, code, size); =20 text_poke_bp(insn, code, size, NULL); @@ -96,12 +97,12 @@ void arch_static_call_transform(void *si =20 if (tramp) { __static_call_validate(tramp, true); - __static_call_transform(tramp, __sc_insn(!func, true), func); + __static_call_transform(tramp, __sc_insn(!func, true), func, false); } =20 if (IS_ENABLED(CONFIG_HAVE_STATIC_CALL_INLINE) && site) { __static_call_validate(site, tail); - __static_call_transform(site, __sc_insn(!func, tail), func); + __static_call_transform(site, __sc_insn(!func, tail), func, false); } =20 mutex_unlock(&text_mutex); @@ -127,8 +128,10 @@ bool __static_call_fixup(void *tramp, u8 return false; } =20 + mutex_lock(&text_mutex); if (op =3D=3D RET_INSN_OPCODE || dest =3D=3D &__x86_return_thunk) - __static_call_transform(tramp, RET, NULL); + __static_call_transform(tramp, RET, NULL, true); + mutex_unlock(&text_mutex); =20 return true; } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90B51C43334 for ; Sat, 23 Jul 2022 10:12:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239358AbiGWKMA (ORCPT ); Sat, 23 Jul 2022 06:12:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239469AbiGWKJk (ORCPT ); Sat, 23 Jul 2022 06:09:40 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5A428049C; Sat, 23 Jul 2022 03:02:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4CE70B82C1F; Sat, 23 Jul 2022 10:02:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 97B67C341C0; Sat, 23 Jul 2022 10:02:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570568; bh=lqnSkCXXiV+5UZ7xk1zTCH+Z/JTRzCyyNJlN1tBUpOg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E1hp6y/iDqo9fN9wiyrOHzM2+x7ptinJqwDL60ewCbyYG+KMpn6VpziqqKy/R+sN5 CvHnoqNzSKckDeZTTqfvf4j4myAzLNxX/7EWxjcODGAcfpeOW4IA35NGfq8yPh9WCY RaQz/ykCGqIvW13tXWuteKS3mwT7YTwP1tOagNv8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Ian Rogers , Stephen Rothwell , Borislav Petkov , Florian Fainelli Subject: [PATCH 5.10 131/148] tools/insn: Restore the relative include paths for cross building Date: Sat, 23 Jul 2022 11:55:43 +0200 Message-Id: <20220723095301.021493138@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Borislav Petkov commit 0705ef64d1ff52b817e278ca6e28095585ff31e1 upstream. Building perf on ppc causes: In file included from util/intel-pt-decoder/intel-pt-insn-decoder.c:15: util/intel-pt-decoder/../../../arch/x86/lib/insn.c:14:10: fatal error: as= m/inat.h: No such file or directory 14 | #include /*__ignore_sync_check__ */ | ^~~~~~~~~~~~ Restore the relative include paths so that the compiler can find the headers. Fixes: 93281c4a9657 ("x86/insn: Add an insn_decode() API") Reported-by: Ian Rogers Reported-by: Stephen Rothwell Signed-off-by: Borislav Petkov Tested-by: Ian Rogers Tested-by: Stephen Rothwell Link: https://lkml.kernel.org/r/20210317150858.02b1bbc8@canb.auug.org.au Cc: Florian Fainelli Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/arch/x86/lib/insn.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/tools/arch/x86/lib/insn.c +++ b/tools/arch/x86/lib/insn.c @@ -10,13 +10,13 @@ #else #include #endif -#include /* __ignore_sync_check__ */ -#include /* __ignore_sync_check__ */ +#include "../include/asm/inat.h" /* __ignore_sync_check__ */ +#include "../include/asm/insn.h" /* __ignore_sync_check__ */ =20 #include #include =20 -#include /* __ignore_sync_check__ */ +#include "../include/asm/emulate_prefix.h" /* __ignore_sync_check__ */ =20 /* Verify next sizeof(t) bytes can be on the same instruction */ #define validate_next(t, insn, n) \ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06890C433EF for ; Sat, 23 Jul 2022 10:11:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239314AbiGWKLx (ORCPT ); Sat, 23 Jul 2022 06:11:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239506AbiGWKJl (ORCPT ); Sat, 23 Jul 2022 06:09:41 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9952CE50B; Sat, 23 Jul 2022 03:02:53 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 0F3CAB82B92; Sat, 23 Jul 2022 10:02:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6CDE8C341C0; Sat, 23 Jul 2022 10:02:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570570; bh=oUt1hCweBt6JC7gg1LyKIiRwNXtpWgJiF7uTPwGsvCg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZbcxIRZSnrX1Y17dDnDijJOy6K6w8WExwDQivR1WPxofNrkrUPDkYo9YHmLAQ3nAS 2jNij8v3AIaEn6/9Y6QKMZhRG2/jfXufAuUDmk49WcTGRc5YginU6wecSH+8qetlvg aORkTXlK+Nb+pZXo4pyOo4XtPyAq8iCML6FXXTGQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Paolo Bonzini , Ben Hutchings Subject: [PATCH 5.10 132/148] x86, kvm: use proper ASM macros for kvm_vcpu_is_preempted Date: Sat, 23 Jul 2022 11:55:44 +0200 Message-Id: <20220723095301.320884513@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Greg Kroah-Hartman commit edbaf6e5e93acda96aae23ba134ef3c1466da3b5 upstream. The build rightfully complains about: arch/x86/kernel/kvm.o: warning: objtool: __raw_callee_save___kvm_vcpu_is_p= reempted()+0x12: missing int3 after ret because the ASM_RET call is not being used correctly in kvm_vcpu_is_preempt= ed(). This was hand-fixed-up in the kvm merge commit a4cfff3f0f8c ("Merge branch 'kvm-older-features' into HEAD") which of course can not be backported to stable kernels, so just fix this up directly instead. Cc: Paolo Bonzini Cc: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/kvm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -953,7 +953,7 @@ asm( "movq __per_cpu_offset(,%rdi,8), %rax;" "cmpb $0, " __stringify(KVM_STEAL_TIME_preempted) "+steal_time(%rax);" "setne %al;" -"ret;" +ASM_RET ".size __raw_callee_save___kvm_vcpu_is_preempted, .-__raw_callee_save___kv= m_vcpu_is_preempted;" ".popsection"); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C718CCA483 for ; Sat, 23 Jul 2022 10:10:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239224AbiGWKKy (ORCPT ); Sat, 23 Jul 2022 06:10:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239052AbiGWKJO (ORCPT ); Sat, 23 Jul 2022 06:09:14 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEE6BCB744; Sat, 23 Jul 2022 03:02:22 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A69476125F; Sat, 23 Jul 2022 10:02:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4257C341C7; Sat, 23 Jul 2022 10:02:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570542; bh=ogXzme9Nd+4tRHQ/niqYL5axx8kpl4DBKD2E0Na7Ndo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pKRghvFw8iBx8dE2MMGfD5wlnyqcJ+UnhBKQX3i0NTDuxBVqYFGdBIXPYaq7YZqdN 28RPNcLJiG0uCYGUtPCdtE/M8UoESaFCBF0DEpjMuziEaJ37nUOZWAFFw4GzMM6LVl vJx7ZrYw4qlE+E5Oou8ijY48T3rtGZ4Ilfhh7Ye0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Ben Hutchings Subject: [PATCH 5.10 133/148] x86/xen: Fix initialisation in hypercall_page after rethunk Date: Sat, 23 Jul 2022 11:55:45 +0200 Message-Id: <20220723095301.607582870@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Ben Hutchings The hypercall_page is special and the RETs there should not be changed into rethunk calls (but can have SLS mitigation). Change the initial instructions to ret + int3 padding, as was done in upstream commit 5b2fc51576ef "x86/ibt,xen: Sprinkle the ENDBR". Signed-off-by: Ben Hutchings Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/xen/xen-head.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -69,9 +69,9 @@ SYM_CODE_END(asm_cpu_bringup_and_idle) SYM_CODE_START(hypercall_page) .rept (PAGE_SIZE / 32) UNWIND_HINT_FUNC - .skip 31, 0x90 ANNOTATE_UNRET_SAFE - RET + ret + .skip 31, 0xcc .endr =20 #define HYPERCALL(n) \ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66E18C433EF for ; Sat, 23 Jul 2022 10:11:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238844AbiGWKLM (ORCPT ); Sat, 23 Jul 2022 06:11:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239091AbiGWKJS (ORCPT ); Sat, 23 Jul 2022 06:09:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67509CB76D; Sat, 23 Jul 2022 03:02:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 80488612C6; Sat, 23 Jul 2022 10:02:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8C69BC341C0; Sat, 23 Jul 2022 10:02:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570544; bh=3FfEQtHfSY7pNsV/mu/qnN2Se9oRRp0rmZZQYox1Mes=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LRdrMoO76sh8OoakglL8nvqFwp1qhWetNntXNvzYLzHllww77t1tki3FL8o7MKXFE Sl7ej41F+CwdDtBES/vMPibYak72kPlM4f/00QrmKIEQ88E2MFySpu/oMJv9vHBg1W JzJJ/T2+32vkMpSCw7B1f5ZEqyx+VsYchqr9qY58= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Steven Rostedt (VMware)" , Josh Poimboeuf Subject: [PATCH 5.10 134/148] x86/ftrace: Add UNWIND_HINT_FUNC annotation for ftrace_stub Date: Sat, 23 Jul 2022 11:55:46 +0200 Message-Id: <20220723095301.923053195@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Josh Poimboeuf commit 18660698a3d30868524cefb60dcd4e0e297f71bb upstream. Prevent an unreachable objtool warning after the sibling call detection gets improved. ftrace_stub() is basically a function, annotate it as such. Acked-by: Steven Rostedt (VMware) Signed-off-by: Josh Poimboeuf Link: https://lore.kernel.org/r/6845e1b2fb0723a95740c6674e548ba38c5ea489.16= 11263461.git.jpoimboe@redhat.com Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/ftrace_64.S | 1 + 1 file changed, 1 insertion(+) --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -173,6 +173,7 @@ SYM_INNER_LABEL(ftrace_graph_call, SYM_L * It is also used to copy the RET for trampolines. */ SYM_INNER_LABEL_ALIGN(ftrace_stub, SYM_L_WEAK) + UNWIND_HINT_FUNC RET SYM_FUNC_END(ftrace_epilogue) From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98F3ECCA483 for ; Sat, 23 Jul 2022 10:11:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239237AbiGWKL2 (ORCPT ); Sat, 23 Jul 2022 06:11:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239286AbiGWKJc (ORCPT ); Sat, 23 Jul 2022 06:09:32 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D8427D1C7; Sat, 23 Jul 2022 03:02:40 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 9D525CE0B68; Sat, 23 Jul 2022 10:02:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8EFDFC341CE; Sat, 23 Jul 2022 10:02:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570547; bh=/dc3mGi7bQmupPffWgNjejnw/63egPiFRy77Qic0Qag=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iqAIXbuU37kZSee7jovqxJoGBoCgS2VdKhvwTr+1vQN9JB2xvSyv7IpYeFYKRJzA8 4E84Qoi+a/nWrW4lN0rpAhHpz9+lPP/OdXJ7+GXfaORZOXP5syh57G1tGl3Z80mzUA izEnokMm41c7V9DsbyLK9xs6bxoyut+YmbLBP4p4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Jiri Slaby , Borislav Petkov Subject: [PATCH 5.10 135/148] x86/asm/32: Fix ANNOTATE_UNRET_SAFE use on 32-bit Date: Sat, 23 Jul 2022 11:55:47 +0200 Message-Id: <20220723095302.194859866@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Jiri Slaby commit 3131ef39fb03bbde237d0b8260445898f3dfda5b upstream. The build on x86_32 currently fails after commit 9bb2ec608a20 (objtool: Update Retpoline validation) with: arch/x86/kernel/../../x86/xen/xen-head.S:35: Error: no such instruction: = `annotate_unret_safe' ANNOTATE_UNRET_SAFE is defined in nospec-branch.h. And head_32.S is missing this include. Fix this. Fixes: 9bb2ec608a20 ("objtool: Update Retpoline validation") Signed-off-by: Jiri Slaby Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/63e23f80-033f-f64e-7522-2816debbc367@kernel= .org Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/head_32.S | 1 + 1 file changed, 1 insertion(+) --- a/arch/x86/kernel/head_32.S +++ b/arch/x86/kernel/head_32.S @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12EA4C43334 for ; Sat, 23 Jul 2022 10:13:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239318AbiGWKNm (ORCPT ); Sat, 23 Jul 2022 06:13:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239252AbiGWKNR (ORCPT ); Sat, 23 Jul 2022 06:13:17 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 974DE8AB3D; Sat, 23 Jul 2022 03:03:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B912BB82C1A; Sat, 23 Jul 2022 10:03:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10494C341C0; Sat, 23 Jul 2022 10:03:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570608; bh=iGIXm+MJ2pyHxJsG/SZdynCWcn2aikgM28KIxvJ1Ohg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CklgUZSWx3AKnrGehuhHcwRT+ZpPQtyBhaAKD4cn2zWku5wNIPMlJIeVhCaMrisHz vAGrLcCuBGnqDY9VmrfSXGRFvS7PHRWKswWlxbujIMpZ5eOaBximjk60iIh1yYIx0X 8bCYzU3Cfuksrfih4DBpfNeO+XzobhoI999GdHHQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, kernel test robot , Nathan Chancellor , Linus Torvalds Subject: [PATCH 5.10 136/148] x86/speculation: Use DECLARE_PER_CPU for x86_spec_ctrl_current Date: Sat, 23 Jul 2022 11:55:48 +0200 Message-Id: <20220723095302.474417251@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Nathan Chancellor commit db886979683a8360ced9b24ab1125ad0c4d2cf76 upstream. Clang warns: arch/x86/kernel/cpu/bugs.c:58:21: error: section attribute is specified o= n redeclared variable [-Werror,-Wsection] DEFINE_PER_CPU(u64, x86_spec_ctrl_current); ^ arch/x86/include/asm/nospec-branch.h:283:12: note: previous declaration i= s here extern u64 x86_spec_ctrl_current; ^ 1 error generated. The declaration should be using DECLARE_PER_CPU instead so all attributes stay in sync. Cc: stable@vger.kernel.org Fixes: fc02735b14ff ("KVM: VMX: Prevent guest RSB poisoning attacks with eI= BRS") Reported-by: kernel test robot Signed-off-by: Nathan Chancellor Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/include/asm/nospec-branch.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -11,6 +11,7 @@ #include #include #include +#include =20 #define RETPOLINE_THUNK_SIZE 32 =20 @@ -281,7 +282,7 @@ static inline void indirect_branch_predi =20 /* The Intel SPEC CTRL MSR base value cache */ extern u64 x86_spec_ctrl_base; -extern u64 x86_spec_ctrl_current; +DECLARE_PER_CPU(u64, x86_spec_ctrl_current); extern void write_spec_ctrl_current(u64 val, bool force); extern u64 spec_ctrl_current(void); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A118C433EF for ; Sat, 23 Jul 2022 10:12:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239082AbiGWKMD (ORCPT ); Sat, 23 Jul 2022 06:12:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239552AbiGWKJn (ORCPT ); Sat, 23 Jul 2022 06:09:43 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80C79CBD33; Sat, 23 Jul 2022 03:02:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1C9636125F; Sat, 23 Jul 2022 10:02:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2846AC341C0; Sat, 23 Jul 2022 10:02:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570576; bh=ht/zw+tJsdLI5zMA2BjnKf2FhyrjDAtiyH10+522ZUY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rSthXR2uXHc9yyX+5HKUWQw2wH7yrEUZ1Sm1uY7M4FujYqHR2OHzxexP/Gs/6EEIW 4e1H93MDCBHpH781abIp3TxDFmB2aLfuVlPkx77T5UTTXBDuQLLZvPtY8G2VqYoIpW LoSRb6D3ZPY6874CE0b58cGZfd4O9RMxoSpOGWWQ= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Guenter Roeck , Thadeu Lima de Souza Cascardo , "Peter Zijlstra (Intel)" , Borislav Petkov , Josh Poimboeuf , Linus Torvalds Subject: [PATCH 5.10 137/148] efi/x86: use naked RET on mixed mode call wrapper Date: Sat, 23 Jul 2022 11:55:49 +0200 Message-Id: <20220723095302.760417661@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thadeu Lima de Souza Cascardo commit 51a6fa0732d6be6a44e0032752ad2ac10d67c796 upstream. When running with return thunks enabled under 32-bit EFI, the system crashes with: kernel tried to execute NX-protected page - exploit attempt? (uid: 0) BUG: unable to handle page fault for address: 000000005bc02900 #PF: supervisor instruction fetch in kernel mode #PF: error_code(0x0011) - permissions violation PGD 18f7063 P4D 18f7063 PUD 18ff063 PMD 190e063 PTE 800000005bc02063 Oops: 0011 [#1] PREEMPT SMP PTI CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.19.0-rc6+ #166 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 RIP: 0010:0x5bc02900 Code: Unable to access opcode bytes at RIP 0x5bc028d6. RSP: 0018:ffffffffb3203e10 EFLAGS: 00010046 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000048 RDX: 000000000190dfac RSI: 0000000000001710 RDI: 000000007eae823b RBP: ffffffffb3203e70 R08: 0000000001970000 R09: ffffffffb3203e28 R10: 747563657865206c R11: 6c6977203a696665 R12: 0000000000001710 R13: 0000000000000030 R14: 0000000001970000 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff8e013ca00000(0000) knlGS:0000000000000= 000 CS: 0010 DS: 0018 ES: 0018 CR0: 0000000080050033 CR2: 000000005bc02900 CR3: 0000000001930000 CR4: 00000000000006f0 Call Trace: ? efi_set_virtual_address_map+0x9c/0x175 efi_enter_virtual_mode+0x4a6/0x53e start_kernel+0x67c/0x71e x86_64_start_reservations+0x24/0x2a x86_64_start_kernel+0xe9/0xf4 secondary_startup_64_no_verify+0xe5/0xeb That's because it cannot jump to the return thunk from the 32-bit code. Using a naked RET and marking it as safe allows the system to proceed booting. Fixes: aa3d480315ba ("x86: Use return-thunk in asm code") Reported-by: Guenter Roeck Signed-off-by: Thadeu Lima de Souza Cascardo Cc: Peter Zijlstra (Intel) Cc: Borislav Petkov Cc: Josh Poimboeuf Cc: Tested-by: Guenter Roeck Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/platform/efi/efi_thunk_64.S | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) --- a/arch/x86/platform/efi/efi_thunk_64.S +++ b/arch/x86/platform/efi/efi_thunk_64.S @@ -22,6 +22,7 @@ #include #include #include +#include =20 .text .code64 @@ -63,7 +64,9 @@ SYM_CODE_START(__efi64_thunk) 1: movq 24(%rsp), %rsp pop %rbx pop %rbp - RET + ANNOTATE_UNRET_SAFE + ret + int3 =20 .code32 2: pushl $__KERNEL_CS From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E258FC43334 for ; Sat, 23 Jul 2022 10:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239124AbiGWKMM (ORCPT ); Sat, 23 Jul 2022 06:12:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238467AbiGWKJs (ORCPT ); Sat, 23 Jul 2022 06:09:48 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59902CE53F; Sat, 23 Jul 2022 03:03:06 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B686F61212; Sat, 23 Jul 2022 10:03:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C07F8C341C7; Sat, 23 Jul 2022 10:03:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570585; bh=A83+z+ZuNOVBikqbMgrr1T/7RH5ymmPgzhuG6vkGXkc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WTSBbiQ4p80lAdoLIDLeTvs+zys9DwlBF56pbVd26cL3B8RzurBX9k64CSLapntRK mPHJ2K3ZuE+4YugF0YIjuH7A5/IBWMCUZn0F+k7YtbwLIaP8b3B3bdQrgFxfGw1Oei RWBUs/ojak3gDsI8DFseohmD6gNbg4f6h/nz1vzs= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Thadeu Lima de Souza Cascardo , Borislav Petkov , Josh Poimboeuf , Paolo Bonzini , Linux Kernel Functional Testing , Jack Wang , Peter Zijlstra Subject: [PATCH 5.10 138/148] x86/kvm: fix FASTOP_SIZE when return thunks are enabled Date: Sat, 23 Jul 2022 11:55:50 +0200 Message-Id: <20220723095303.038128243@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Thadeu Lima de Souza Cascardo commit 84e7051c0bc1f2a13101553959b3a9d9a8e24939 upstream. The return thunk call makes the fastop functions larger, just like IBT does. Consider a 16-byte FASTOP_SIZE when CONFIG_RETHUNK is enabled. Otherwise, functions will be incorrectly aligned and when computing their position for differently sized operators, they will executed in the middle or end of a function, which may as well be an int3, leading to a crash like: [ 36.091116] int3: 0000 [#1] SMP NOPTI [ 36.091119] CPU: 3 PID: 1371 Comm: qemu-system-x86 Not tainted 5.15.0-41= -generic #44 [ 36.091120] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.1= 5.0-1 04/01/2014 [ 36.091121] RIP: 0010:xaddw_ax_dx+0x9/0x10 [kvm] [ 36.091185] Code: 00 0f bb d0 c3 cc cc cc cc 48 0f bb d0 c3 cc cc cc cc = 0f 1f 80 00 00 00 00 0f c0 d0 c3 cc cc cc cc 66 0f c1 d0 c3 cc cc cc cc <0f= > 1f 80 00 00 00 00 0f c1 d0 c3 cc cc cc cc 48 0f c1 d0 c3 cc cc [ 36.091186] RSP: 0018:ffffb1f541143c98 EFLAGS: 00000202 [ 36.091188] RAX: 0000000089abcdef RBX: 0000000000000001 RCX: 00000000000= 00000 [ 36.091188] RDX: 0000000076543210 RSI: ffffffffc073c6d0 RDI: 00000000000= 00200 [ 36.091189] RBP: ffffb1f541143ca0 R08: ffff9f1803350a70 R09: 00000000000= 00002 [ 36.091190] R10: ffff9f1803350a70 R11: 0000000000000000 R12: ffff9f18033= 50a70 [ 36.091190] R13: ffffffffc077fee0 R14: 0000000000000000 R15: 00000000000= 00000 [ 36.091191] FS: 00007efdfce8d640(0000) GS:ffff9f187dd80000(0000) knlGS:= 0000000000000000 [ 36.091192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.091192] CR2: 0000000000000000 CR3: 0000000009b62002 CR4: 00000000007= 72ee0 [ 36.091195] PKRU: 55555554 [ 36.091195] Call Trace: [ 36.091197] [ 36.091198] ? fastop+0x5a/0xa0 [kvm] [ 36.091222] x86_emulate_insn+0x7b8/0xe90 [kvm] [ 36.091244] x86_emulate_instruction+0x2f4/0x630 [kvm] [ 36.091263] ? kvm_arch_vcpu_load+0x7c/0x230 [kvm] [ 36.091283] ? vmx_prepare_switch_to_host+0xf7/0x190 [kvm_intel] [ 36.091290] complete_emulated_mmio+0x297/0x320 [kvm] [ 36.091310] kvm_arch_vcpu_ioctl_run+0x32f/0x550 [kvm] [ 36.091330] kvm_vcpu_ioctl+0x29e/0x6d0 [kvm] [ 36.091344] ? kvm_vcpu_ioctl+0x120/0x6d0 [kvm] [ 36.091357] ? __fget_files+0x86/0xc0 [ 36.091362] ? __fget_files+0x86/0xc0 [ 36.091363] __x64_sys_ioctl+0x92/0xd0 [ 36.091366] do_syscall_64+0x59/0xc0 [ 36.091369] ? syscall_exit_to_user_mode+0x27/0x50 [ 36.091370] ? do_syscall_64+0x69/0xc0 [ 36.091371] ? syscall_exit_to_user_mode+0x27/0x50 [ 36.091372] ? __x64_sys_writev+0x1c/0x30 [ 36.091374] ? do_syscall_64+0x69/0xc0 [ 36.091374] ? exit_to_user_mode_prepare+0x37/0xb0 [ 36.091378] ? syscall_exit_to_user_mode+0x27/0x50 [ 36.091379] ? do_syscall_64+0x69/0xc0 [ 36.091379] ? do_syscall_64+0x69/0xc0 [ 36.091380] ? do_syscall_64+0x69/0xc0 [ 36.091381] ? do_syscall_64+0x69/0xc0 [ 36.091381] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 36.091384] RIP: 0033:0x7efdfe6d1aff [ 36.091390] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 = 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <41= > 89 c0 3d 00 f0 ff ff 77 1f 48 8b 44 24 18 64 48 2b 04 25 28 00 [ 36.091391] RSP: 002b:00007efdfce8c460 EFLAGS: 00000246 ORIG_RAX: 000000= 0000000010 [ 36.091393] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007efdfe6= d1aff [ 36.091393] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 00000000000= 0000c [ 36.091394] RBP: 0000558f1609e220 R08: 0000558f13fb8190 R09: 00000000fff= fffff [ 36.091394] R10: 0000558f16b5e950 R11: 0000000000000246 R12: 00000000000= 00000 [ 36.091394] R13: 0000000000000001 R14: 0000000000000000 R15: 00000000000= 00000 [ 36.091396] [ 36.091397] Modules linked in: isofs nls_iso8859_1 kvm_intel joydev kvm = input_leds serio_raw sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scs= i_dh_alua ipmi_devintf ipmi_msghandler drm msr ip_tables x_tables autofs4 b= trfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_m= emcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multip= ath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel vi= rtio_net net_failover crypto_simd ahci xhci_pci cryptd psmouse virtio_blk l= ibahci xhci_pci_renesas failover [ 36.123271] ---[ end trace db3c0ab5a48fabcc ]--- [ 36.123272] RIP: 0010:xaddw_ax_dx+0x9/0x10 [kvm] [ 36.123319] Code: 00 0f bb d0 c3 cc cc cc cc 48 0f bb d0 c3 cc cc cc cc = 0f 1f 80 00 00 00 00 0f c0 d0 c3 cc cc cc cc 66 0f c1 d0 c3 cc cc cc cc <0f= > 1f 80 00 00 00 00 0f c1 d0 c3 cc cc cc cc 48 0f c1 d0 c3 cc cc [ 36.123320] RSP: 0018:ffffb1f541143c98 EFLAGS: 00000202 [ 36.123321] RAX: 0000000089abcdef RBX: 0000000000000001 RCX: 00000000000= 00000 [ 36.123321] RDX: 0000000076543210 RSI: ffffffffc073c6d0 RDI: 00000000000= 00200 [ 36.123322] RBP: ffffb1f541143ca0 R08: ffff9f1803350a70 R09: 00000000000= 00002 [ 36.123322] R10: ffff9f1803350a70 R11: 0000000000000000 R12: ffff9f18033= 50a70 [ 36.123323] R13: ffffffffc077fee0 R14: 0000000000000000 R15: 00000000000= 00000 [ 36.123323] FS: 00007efdfce8d640(0000) GS:ffff9f187dd80000(0000) knlGS:= 0000000000000000 [ 36.123324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.123325] CR2: 0000000000000000 CR3: 0000000009b62002 CR4: 00000000007= 72ee0 [ 36.123327] PKRU: 55555554 [ 36.123328] Kernel panic - not syncing: Fatal exception in interrupt [ 36.123410] Kernel Offset: 0x1400000 from 0xffffffff81000000 (relocation= range: 0xffffffff80000000-0xffffffffbfffffff) [ 36.135305] ---[ end Kernel panic - not syncing: Fatal exception in inte= rrupt ]--- Fixes: aa3d480315ba ("x86: Use return-thunk in asm code") Signed-off-by: Thadeu Lima de Souza Cascardo Co-developed-by: Peter Zijlstra (Intel) Cc: Borislav Petkov Cc: Josh Poimboeuf Cc: Paolo Bonzini Reported-by: Linux Kernel Functional Testing Message-Id: <20220713171241.184026-1-cascardo@canonical.com> Tested-by: Jack Wang Signed-off-by: Paolo Bonzini Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/emulate.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -188,8 +188,12 @@ #define X8(x...) X4(x), X4(x) #define X16(x...) X8(x), X8(x) =20 -#define NR_FASTOP (ilog2(sizeof(ulong)) + 1) -#define FASTOP_SIZE 8 +#define NR_FASTOP (ilog2(sizeof(ulong)) + 1) +#define RET_LENGTH (1 + (4 * IS_ENABLED(CONFIG_RETHUNK)) + \ + IS_ENABLED(CONFIG_SLS)) +#define FASTOP_LENGTH (ENDBR_INSN_SIZE + 7 + RET_LENGTH) +#define FASTOP_SIZE (8 << ((FASTOP_LENGTH > 8) & 1) << ((FASTOP_LENGTH > 1= 6) & 1)) +static_assert(FASTOP_LENGTH <=3D FASTOP_SIZE); =20 struct opcode { u64 flags : 56; @@ -438,8 +442,6 @@ static int fastop(struct x86_emulate_ctx * RET | JMP __x86_return_thunk [1,5 bytes; CONFIG_RETHUNK] * INT3 [1 byte; CONFIG_SLS] */ -#define RET_LENGTH (1 + (4 * IS_ENABLED(CONFIG_RETHUNK)) + \ - IS_ENABLED(CONFIG_SLS)) #define SETCC_LENGTH (3 + RET_LENGTH) #define SETCC_ALIGN (4 << ((SETCC_LENGTH > 4) & 1) << ((SETCC_LENGTH > 8) = & 1)) static_assert(SETCC_LENGTH <=3D SETCC_ALIGN); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72EE9C433EF for ; Sat, 23 Jul 2022 10:12:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239420AbiGWKMO (ORCPT ); Sat, 23 Jul 2022 06:12:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238772AbiGWKJv (ORCPT ); Sat, 23 Jul 2022 06:09:51 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DB5B3F337; Sat, 23 Jul 2022 03:03:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8BBDD60C81; Sat, 23 Jul 2022 10:03:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9986DC341CA; Sat, 23 Jul 2022 10:03:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570588; bh=nuzqddq499/Cz35y7gS4UjYy2QLCf/Ts4K7ar6xCcXM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pclEJzr+od9lDiT/HmO5VstbxRxzPYS66nc3ekFMJeNRDfvtJNVwOfYlWKA3XrJig Z1zRS1LpSxrZQdnTRLwnOPznxMGltiKi9/d9Y8ynQzMI7eXwgrF4yoxaFpvLMU4J21 eDUNSxGLe/DSGiOpvyaoBK+4Y9LbE6Awir4P0ZtI= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Linus Torvalds , Paolo Bonzini Subject: [PATCH 5.10 139/148] KVM: emulate: do not adjust size of fastop and setcc subroutines Date: Sat, 23 Jul 2022 11:55:51 +0200 Message-Id: <20220723095303.318087524@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Paolo Bonzini commit 79629181607e801c0b41b8790ac4ee2eb5d7bc3e upstream. Instead of doing complicated calculations to find the size of the subroutin= es (which are even more complicated because they need to be stringified into an asm statement), just hardcode to 16. It is less dense for a few combinations of IBT/SLS/retbleed, but it has the advantage of being really simple. Cc: stable@vger.kernel.org # 5.15.x: 84e7051c0bc1: x86/kvm: fix FASTOP_SIZE= when return thunks are enabled Cc: stable@vger.kernel.org Suggested-by: Linus Torvalds Signed-off-by: Paolo Bonzini Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/emulate.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -188,13 +188,6 @@ #define X8(x...) X4(x), X4(x) #define X16(x...) X8(x), X8(x) =20 -#define NR_FASTOP (ilog2(sizeof(ulong)) + 1) -#define RET_LENGTH (1 + (4 * IS_ENABLED(CONFIG_RETHUNK)) + \ - IS_ENABLED(CONFIG_SLS)) -#define FASTOP_LENGTH (ENDBR_INSN_SIZE + 7 + RET_LENGTH) -#define FASTOP_SIZE (8 << ((FASTOP_LENGTH > 8) & 1) << ((FASTOP_LENGTH > 1= 6) & 1)) -static_assert(FASTOP_LENGTH <=3D FASTOP_SIZE); - struct opcode { u64 flags : 56; u64 intercept : 8; @@ -308,9 +301,15 @@ static void invalidate_registers(struct * Moreover, they are all exactly FASTOP_SIZE bytes long, so functions for * different operand sizes can be reached by calculation, rather than a ju= mp * table (which would be bigger than the code). + * + * The 16 byte alignment, considering 5 bytes for the RET thunk, 3 for END= BR + * and 1 for the straight line speculation INT3, leaves 7 bytes for the + * body of the function. Currently none is larger than 4. */ static int fastop(struct x86_emulate_ctxt *ctxt, fastop_t fop); =20 +#define FASTOP_SIZE 16 + #define __FOP_FUNC(name) \ ".align " __stringify(FASTOP_SIZE) " \n\t" \ ".type " name ", @function \n\t" \ @@ -442,9 +441,7 @@ static int fastop(struct x86_emulate_ctx * RET | JMP __x86_return_thunk [1,5 bytes; CONFIG_RETHUNK] * INT3 [1 byte; CONFIG_SLS] */ -#define SETCC_LENGTH (3 + RET_LENGTH) -#define SETCC_ALIGN (4 << ((SETCC_LENGTH > 4) & 1) << ((SETCC_LENGTH > 8) = & 1)) -static_assert(SETCC_LENGTH <=3D SETCC_ALIGN); +#define SETCC_ALIGN 16 =20 #define FOP_SETCC(op) \ ".align " __stringify(SETCC_ALIGN) " \n\t" \ From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3685C43334 for ; Sat, 23 Jul 2022 10:12:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238936AbiGWKMf (ORCPT ); Sat, 23 Jul 2022 06:12:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238965AbiGWKKA (ORCPT ); Sat, 23 Jul 2022 06:10:00 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E0C6CFE7E; Sat, 23 Jul 2022 03:03:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 69FA4611BD; Sat, 23 Jul 2022 10:03:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74FAEC341C0; Sat, 23 Jul 2022 10:03:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570590; bh=kqPY9MUO21vFnYIoOXN8BlGKOrZjLsXuoEhrjvrRTEc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SM/CnCCV7dCzHO7qk9gEQtGQfy9A4Gp5K98S5CcYfFW240mq1hgIjoY2gcQaKX+I1 NjZ5/zTzzsXZDCGghLY9oxqd4KasxvINru9+umAFylFJsSKobNxnDdi76DqrK4cikL F7g11qBwmoywmoTvA4GEursJaa+qczroWeT2oUL4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Adrian Hunter , Borislav Petkov , Ian Rogers , Jiri Olsa , Namhyung Kim , Pawan Gupta , Peter Zijlstra , Arnaldo Carvalho de Melo Subject: [PATCH 5.10 140/148] tools arch x86: Sync the msr-index.h copy with the kernel sources Date: Sat, 23 Jul 2022 11:55:52 +0200 Message-Id: <20220723095303.601411287@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Arnaldo Carvalho de Melo commit 91d248c3b903b46a58cbc7e8d38d684d3e4007c2 upstream. To pick up the changes from these csets: 4ad3278df6fe2b08 ("x86/speculation: Disable RRSBA behavior") d7caac991feeef1b ("x86/cpu/amd: Add Spectral Chicken") That cause no changes to tooling: $ tools/perf/trace/beauty/tracepoints/x86_msr.sh > before $ cp arch/x86/include/asm/msr-index.h tools/arch/x86/include/asm/msr-inde= x.h $ tools/perf/trace/beauty/tracepoints/x86_msr.sh > after $ diff -u before after $ Just silences this perf build warning: Warning: Kernel ABI header at 'tools/arch/x86/include/asm/msr-index.h' di= ffers from latest version at 'arch/x86/include/asm/msr-index.h' diff -u tools/arch/x86/include/asm/msr-index.h arch/x86/include/asm/msr-i= ndex.h Cc: Adrian Hunter Cc: Borislav Petkov Cc: Ian Rogers Cc: Jiri Olsa Cc: Namhyung Kim Cc: Pawan Gupta Cc: Peter Zijlstra Link: https://lore.kernel.org/lkml/YtQTm9wsB3hxQWvy@kernel.org Signed-off-by: Arnaldo Carvalho de Melo Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/arch/x86/include/asm/msr-index.h | 4 ++++ 1 file changed, 4 insertions(+) --- a/tools/arch/x86/include/asm/msr-index.h +++ b/tools/arch/x86/include/asm/msr-index.h @@ -93,6 +93,7 @@ #define MSR_IA32_ARCH_CAPABILITIES 0x0000010a #define ARCH_CAP_RDCL_NO BIT(0) /* Not susceptible to Meltdown */ #define ARCH_CAP_IBRS_ALL BIT(1) /* Enhanced IBRS support */ +#define ARCH_CAP_RSBA BIT(2) /* RET may use alternative branch predictor= s */ #define ARCH_CAP_SKIP_VMENTRY_L1DFLUSH BIT(3) /* Skip L1D flush on vmentry= */ #define ARCH_CAP_SSB_NO BIT(4) /* * Not susceptible to Speculative Store Bypass @@ -516,6 +517,9 @@ /* Fam 17h MSRs */ #define MSR_F17H_IRPERF 0xc00000e9 =20 +#define MSR_ZEN2_SPECTRAL_CHICKEN 0xc00110e3 +#define MSR_ZEN2_SPECTRAL_CHICKEN_BIT BIT_ULL(1) + /* Fam 16h MSRs */ #define MSR_F16H_L2I_PERF_CTL 0xc0010230 #define MSR_F16H_L2I_PERF_CTR 0xc0010231 From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3E00C433EF for ; Sat, 23 Jul 2022 10:12:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239482AbiGWKMj (ORCPT ); Sat, 23 Jul 2022 06:12:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239037AbiGWKKK (ORCPT ); Sat, 23 Jul 2022 06:10:10 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E6D3D0660; Sat, 23 Jul 2022 03:03:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4BFCB61263; Sat, 23 Jul 2022 10:03:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 57E4BC341C0; Sat, 23 Jul 2022 10:03:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570593; bh=N4ZoAMH1labCB/9pAJSInhlWqw/PW7AQbISNPRZAbYY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PkFjTiqcoKLm0S/NkJdE+ZOmuKsqsaJwLu/R7jWswvGEu6ovwlh4DQ8WgHlFlxdgT IPbQE1zMb1qXrnHivzSvfNfYVsS++K+e7GIADMs6lmqZ6Y31ujfmds78g4dHGnZCrl MgMfb1e95fN/IX49c8CNOq+9DHpFzrKmpGaQ04uk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Adrian Hunter , Borislav Petkov , Ian Rogers , Jiri Olsa , Namhyung Kim , Arnaldo Carvalho de Melo Subject: [PATCH 5.10 141/148] tools headers cpufeatures: Sync with the kernel sources Date: Sat, 23 Jul 2022 11:55:53 +0200 Message-Id: <20220723095303.842060788@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Arnaldo Carvalho de Melo commit f098addbdb44c8a565367f5162f3ab170ed9404a upstream. To pick the changes from: f43b9876e857c739 ("x86/retbleed: Add fine grained Kconfig knobs") a149180fbcf336e9 ("x86: Add magic AMD return-thunk") 15e67227c49a5783 ("x86: Undo return-thunk damage") 369ae6ffc41a3c11 ("x86/retpoline: Cleanup some #ifdefery") 4ad3278df6fe2b08 x86/speculation: Disable RRSBA behavior 26aae8ccbc197223 x86/cpu/amd: Enumerate BTC_NO 9756bba28470722d x86/speculation: Fill RSB on vmexit for IBRS 3ebc170068885b6f x86/bugs: Add retbleed=3Dibpb 2dbb887e875b1de3 x86/entry: Add kernel IBRS implementation 6b80b59b35557065 x86/bugs: Report AMD retbleed vulnerability a149180fbcf336e9 x86: Add magic AMD return-thunk 15e67227c49a5783 x86: Undo return-thunk damage a883d624aed463c8 x86/cpufeatures: Move RETPOLINE flags to word 11 51802186158c74a0 x86/speculation/mmio: Enumerate Processor MMIO Stale Dat= a bug This only causes these perf files to be rebuilt: CC /tmp/build/perf/bench/mem-memcpy-x86-64-asm.o CC /tmp/build/perf/bench/mem-memset-x86-64-asm.o And addresses this perf build warning: Warning: Kernel ABI header at 'tools/arch/x86/include/asm/cpufeatures.h' = differs from latest version at 'arch/x86/include/asm/cpufeatures.h' diff -u tools/arch/x86/include/asm/cpufeatures.h arch/x86/include/asm/cpu= features.h Warning: Kernel ABI header at 'tools/arch/x86/include/asm/disabled-featur= es.h' differs from latest version at 'arch/x86/include/asm/disabled-feature= s.h' diff -u tools/arch/x86/include/asm/disabled-features.h arch/x86/include/a= sm/disabled-features.h Cc: Adrian Hunter Cc: Borislav Petkov Cc: Ian Rogers Cc: Jiri Olsa Cc: Namhyung Kim Cc: Peter Zijlstra Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/arch/x86/include/asm/cpufeatures.h | 12 ++++++++++-- tools/arch/x86/include/asm/disabled-features.h | 21 ++++++++++++++++++++- 2 files changed, 30 insertions(+), 3 deletions(-) --- a/tools/arch/x86/include/asm/cpufeatures.h +++ b/tools/arch/x86/include/asm/cpufeatures.h @@ -203,8 +203,8 @@ #define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9) /* AMD ProcFeedbackInterface = */ #define X86_FEATURE_SME ( 7*32+10) /* AMD Secure Memory Encryption */ #define X86_FEATURE_PTI ( 7*32+11) /* Kernel Page Table Isolation enable= d */ -#define X86_FEATURE_RETPOLINE ( 7*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ -#define X86_FEATURE_RETPOLINE_LFENCE ( 7*32+13) /* "" Use LFENCEs for Spec= tre variant 2 */ +#define X86_FEATURE_KERNEL_IBRS ( 7*32+12) /* "" Set/clear IBRS on kernel= entry/exit */ +#define X86_FEATURE_RSB_VMEXIT ( 7*32+13) /* "" Fill RSB on VM-Exit */ #define X86_FEATURE_INTEL_PPIN ( 7*32+14) /* Intel Processor Inventory Nu= mber */ #define X86_FEATURE_CDP_L2 ( 7*32+15) /* Code and Data Prioritization L2 = */ #define X86_FEATURE_MSR_SPEC_CTRL ( 7*32+16) /* "" MSR SPEC_CTRL is implem= ented */ @@ -290,6 +290,12 @@ #define X86_FEATURE_FENCE_SWAPGS_KERNEL (11*32+ 5) /* "" LFENCE in kernel = entry SWAPGS path */ #define X86_FEATURE_SPLIT_LOCK_DETECT (11*32+ 6) /* #AC for split lock */ #define X86_FEATURE_PER_THREAD_MBA (11*32+ 7) /* "" Per-thread Memory Band= width Allocation */ +#define X86_FEATURE_ENTRY_IBPB (11*32+10) /* "" Issue an IBPB on kernel e= ntry */ +#define X86_FEATURE_RRSBA_CTRL (11*32+11) /* "" RET prediction control */ +#define X86_FEATURE_RETPOLINE (11*32+12) /* "" Generic Retpoline mitigati= on for Spectre variant 2 */ +#define X86_FEATURE_RETPOLINE_LFENCE (11*32+13) /* "" Use LFENCE for Spect= re variant 2 */ +#define X86_FEATURE_RETHUNK (11*32+14) /* "" Use REturn THUNK */ +#define X86_FEATURE_UNRET (11*32+15) /* "" AMD BTB untrain return */ =20 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instruction= s */ @@ -308,6 +314,7 @@ #define X86_FEATURE_AMD_SSBD (13*32+24) /* "" Speculative Store Bypass Di= sable */ #define X86_FEATURE_VIRT_SSBD (13*32+25) /* Virtualized Speculative Store= Bypass Disable */ #define X86_FEATURE_AMD_SSB_NO (13*32+26) /* "" Speculative Store Bypass = is fixed in hardware. */ +#define X86_FEATURE_BTC_NO (13*32+29) /* "" Not vulnerable to Branch Type= Confusion */ =20 /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 1= 4 */ #define X86_FEATURE_DTHERM (14*32+ 0) /* Digital Thermal Sensor */ @@ -418,5 +425,6 @@ #define X86_BUG_ITLB_MULTIHIT X86_BUG(23) /* CPU may incur MCE during cer= tain page attribute changes */ #define X86_BUG_SRBDS X86_BUG(24) /* CPU may leak RNG bits if not mitiga= ted */ #define X86_BUG_MMIO_STALE_DATA X86_BUG(25) /* CPU is affected by Process= or MMIO Stale Data vulnerabilities */ +#define X86_BUG_RETBLEED X86_BUG(26) /* CPU is affected by RETBleed */ =20 #endif /* _ASM_X86_CPUFEATURES_H */ --- a/tools/arch/x86/include/asm/disabled-features.h +++ b/tools/arch/x86/include/asm/disabled-features.h @@ -56,6 +56,25 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif =20 +#ifdef CONFIG_RETPOLINE +# define DISABLE_RETPOLINE 0 +#else +# define DISABLE_RETPOLINE ((1 << (X86_FEATURE_RETPOLINE & 31)) | \ + (1 << (X86_FEATURE_RETPOLINE_LFENCE & 31))) +#endif + +#ifdef CONFIG_RETHUNK +# define DISABLE_RETHUNK 0 +#else +# define DISABLE_RETHUNK (1 << (X86_FEATURE_RETHUNK & 31)) +#endif + +#ifdef CONFIG_CPU_UNRET_ENTRY +# define DISABLE_UNRET 0 +#else +# define DISABLE_UNRET (1 << (X86_FEATURE_UNRET & 31)) +#endif + #ifdef CONFIG_IOMMU_SUPPORT # define DISABLE_ENQCMD 0 #else @@ -76,7 +95,7 @@ #define DISABLED_MASK8 0 #define DISABLED_MASK9 (DISABLE_SMAP) #define DISABLED_MASK10 0 -#define DISABLED_MASK11 0 +#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET) #define DISABLED_MASK12 0 #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56522C433EF for ; Sat, 23 Jul 2022 10:12:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238854AbiGWKMy (ORCPT ); Sat, 23 Jul 2022 06:12:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238965AbiGWKMg (ORCPT ); Sat, 23 Jul 2022 06:12:36 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3421D2C25; Sat, 23 Jul 2022 03:03:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DBB6CB82C1D; Sat, 23 Jul 2022 10:03:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 382B9C341C0; Sat, 23 Jul 2022 10:03:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570596; bh=l+wIGaEFm0akZXyYx4NKM3m+CvuHaxklMUOikpE88dA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tNIXNqSbu9vFjwBHtCwMAnMuyx1w9DxZCgCAHyx5TJWT3Ps4V8zsyRSMrl/rUcg41 JEz2jTRQM6xeKV4M5N1EocwqOCOtmyAidH73OkPDNi3KCVbNLeOFJKVfM9xRteTK3J aVAAS9nw7NMzHJ3qkMoTf3vrGYd5G9XkELM4Q25k= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Kim Phillips , Borislav Petkov Subject: [PATCH 5.10 142/148] x86/bugs: Remove apostrophe typo Date: Sat, 23 Jul 2022 11:55:54 +0200 Message-Id: <20220723095304.149027452@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Kim Phillips commit bcf163150cd37348a0cb59e95c916a83a9344b0e upstream. Remove a superfluous ' in the mitigation string. Fixes: e8ec1b6e08a2 ("x86/bugs: Enable STIBP for JMP2RET") Signed-off-by: Kim Phillips Signed-off-by: Borislav Petkov Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kernel/cpu/bugs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1137,7 +1137,7 @@ spectre_v2_user_select_mitigation(void) if (retbleed_mitigation =3D=3D RETBLEED_MITIGATION_UNRET) { if (mode !=3D SPECTRE_V2_USER_STRICT && mode !=3D SPECTRE_V2_USER_STRICT_PREFERRED) - pr_info("Selecting STIBP always-on mode to complement retbleed mitigati= on'\n"); + pr_info("Selecting STIBP always-on mode to complement retbleed mitigati= on\n"); mode =3D SPECTRE_V2_USER_STRICT_PREFERRED; } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FCAAC43334 for ; Sat, 23 Jul 2022 10:13:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239529AbiGWKNH (ORCPT ); Sat, 23 Jul 2022 06:13:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54436 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239386AbiGWKMq (ORCPT ); Sat, 23 Jul 2022 06:12:46 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1738D2C36; Sat, 23 Jul 2022 03:03:35 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D6B55B82B92; Sat, 23 Jul 2022 10:03:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2413BC341CA; Sat, 23 Jul 2022 10:03:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570599; bh=GiZoy1kPgadu41SsnSu581+QtHXEg34TCVQwA8lX2S8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RwTPoO/scAItgQ7GgkCZH3OHnnKC/0RDhbbzra/16q3WThGHUKr1rJULSCsiQijT2 vERV47y0ZF5NnCrhoAqyKzxMQWgq+iQRkvDw4mHXco1Z9GYYLXwSWrAjP2lk1AGjSE RytcJJiKFyDVMAR5VZ5Gixld61GQu64dkJFOAsmE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov Subject: [PATCH 5.10 143/148] um: Add missing apply_returns() Date: Sat, 23 Jul 2022 11:55:55 +0200 Message-Id: <20220723095304.427523891@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 564d998106397394b6aad260f219b882b3347e62 upstream. Implement apply_returns() stub for UM, just like all the other patching routines. Fixes: 15e67227c49a ("x86: Undo return-thunk damage") Reported-by: Randy Dunlap Signed-off-by: Borislav Petkov Link: https://lore.kernel.org/r/Ys%2Ft45l%2FgarIrD0u@worktop.programming.ki= cks-ass.net Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/um/kernel/um_arch.c | 4 ++++ 1 file changed, 4 insertions(+) --- a/arch/um/kernel/um_arch.c +++ b/arch/um/kernel/um_arch.c @@ -362,6 +362,10 @@ void apply_retpolines(s32 *start, s32 *e { } =20 +void apply_returns(s32 *start, s32 *end) +{ +} + void apply_alternatives(struct alt_instr *start, struct alt_instr *end) { } From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84A87C433EF for ; Sat, 23 Jul 2022 10:13:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239475AbiGWKNS (ORCPT ); Sat, 23 Jul 2022 06:13:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239448AbiGWKMw (ORCPT ); Sat, 23 Jul 2022 06:12:52 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D226D3657; Sat, 23 Jul 2022 03:03:41 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 06985CE0B68; Sat, 23 Jul 2022 10:03:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB938C341C0; Sat, 23 Jul 2022 10:03:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570602; bh=/rrw6F0rSsfVQqxOLNWBcDP28QbOYy6khI9EDel11R8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZAgPHxQlFmZuqi7ejQ7Wu+vPV19O5ug+d5u6BXY68ukis8GVRmjtb7jkGQrhH+Wat aTnFGUkuHz5LupAATJEkxMOvBskaXSHYHPBk1dXtgOqz4OhVDLLhmLPuDMBhU3PSo1 ZmDXPso14agA8ZESVgT7HdatpQMEXpN/rXfz5o1Y= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, "Peter Zijlstra (Intel)" , Borislav Petkov , Kees Cook , Nick Desaulniers Subject: [PATCH 5.10 144/148] x86: Use -mindirect-branch-cs-prefix for RETPOLINE builds Date: Sat, 23 Jul 2022 11:55:56 +0200 Message-Id: <20220723095304.680881838@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra commit 68cf4f2a72ef8786e6b7af6fd9a89f27ac0f520d upstream. In order to further enable commit: bbe2df3f6b6d ("x86/alternative: Try inline spectre_v2=3Dretpoline,amd") add the new GCC flag -mindirect-branch-cs-prefix: https://gcc.gnu.org/g:2196a681d7810ad8b227bf983f38ba716620545e https://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D102952 https://bugs.llvm.org/show_bug.cgi?id=3D52323 to RETPOLINE=3Dy builds. This should allow fully inlining retpoline,amd for GCC builds. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Reviewed-by: Kees Cook Acked-by: Nick Desaulniers Link: https://lkml.kernel.org/r/20211119165630.276205624@infradead.org Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- Makefile | 1 + 1 file changed, 1 insertion(+) --- a/Makefile +++ b/Makefile @@ -672,6 +672,7 @@ endif =20 ifdef CONFIG_CC_IS_GCC RETPOLINE_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-extern -m= indirect-branch-register) +RETPOLINE_CFLAGS +=3D $(call cc-option,-mindirect-branch-cs-prefix) RETPOLINE_VDSO_CFLAGS :=3D $(call cc-option,-mindirect-branch=3Dthunk-inli= ne -mindirect-branch-register) endif ifdef CONFIG_CC_IS_CLANG From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6547BC433EF for ; Sat, 23 Jul 2022 10:13:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239501AbiGWKNh (ORCPT ); Sat, 23 Jul 2022 06:13:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239535AbiGWKNN (ORCPT ); Sat, 23 Jul 2022 06:13:13 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ABEE8B4A2; Sat, 23 Jul 2022 03:03:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C131DB82C1B; Sat, 23 Jul 2022 10:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0BE96C341C0; Sat, 23 Jul 2022 10:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570605; bh=zMPVrNhp/F3QfRcx1IwmUr2ts61b2A89EFn63wrxbtY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tJc7e3WAQ9MG/JsLo6JpnxCMs5Gwr+Pa/wESpXLKykzDY2lOJnHWeN7OibGB4Jtb9 HwwgrQTj3toxmQ4UPItVA2qFURaAwNm4eEHo2Hbf8hEGJPIQzmogQYmz+skSEec5To o8t3X5PdZ8vyfcy8sV5+ra38rEg+EZX5FmT6UWjk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Paolo Bonzini , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Peter Zijlstra , Josh Poimboeuf , Nathan Chancellor , Linus Torvalds Subject: [PATCH 5.10 145/148] kvm: fix objtool relocation warning Date: Sat, 23 Jul 2022 11:55:57 +0200 Message-Id: <20220723095304.971051168@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Linus Torvalds commit 291073a566b2094c7192872cc0f17ce73d83cb76 upstream. The recent change to make objtool aware of more symbol relocation types (commit 24ff65257375: "objtool: Teach get_alt_entry() about more relocation types") also added another check, and resulted in this objtool warning when building kvm on x86: arch/x86/kvm/emulate.o: warning: objtool: __ex_table+0x4: don't know ho= w to handle reloc symbol type: kvm_fastop_exception The reason seems to be that kvm_fastop_exception() is marked as a global symbol, which causes the relocation to ke kept around for objtool. And at the same time, the kvm_fastop_exception definition (which is done as an inline asm statement) doesn't actually set the type of the global, which then makes objtool unhappy. The minimal fix is to just not mark kvm_fastop_exception as being a global symbol. It's only used in that one compilation unit anyway, so it was always pointless. That's how all the other local exception table labels are done. I'm not entirely happy about the kinds of games that the kvm code plays with doing its own exception handling, and the fact that it confused objtool is most definitely a symptom of the code being a bit too subtle and ad-hoc. But at least this trivial one-liner makes objtool no longer upset about what is going on. Fixes: 24ff65257375 ("objtool: Teach get_alt_entry() about more relocation = types") Link: https://lore.kernel.org/lkml/CAHk-=3DwiZwq-0LknKhXN4M+T8jbxn_2i9mcKpO= +OaBSSq_Eh7tg@mail.gmail.com/ Cc: Borislav Petkov Cc: Paolo Bonzini Cc: Sean Christopherson Cc: Vitaly Kuznetsov Cc: Wanpeng Li Cc: Jim Mattson Cc: Joerg Roedel Cc: Peter Zijlstra Cc: Josh Poimboeuf Cc: Nathan Chancellor Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- arch/x86/kvm/emulate.c | 1 - 1 file changed, 1 deletion(-) --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -452,7 +452,6 @@ static int fastop(struct x86_emulate_ctx ".skip " __stringify(SETCC_ALIGN) " - (.-" #op "), 0xcc \n\t" =20 asm(".pushsection .fixup, \"ax\"\n" - ".global kvm_fastop_exception \n" "kvm_fastop_exception: xor %esi, %esi; " ASM_RET ".popsection"); From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B476DC43334 for ; Sat, 23 Jul 2022 10:12:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239397AbiGWKMK (ORCPT ); Sat, 23 Jul 2022 06:12:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239612AbiGWKJq (ORCPT ); Sat, 23 Jul 2022 06:09:46 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10DC9CE53B; Sat, 23 Jul 2022 03:03:04 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 04951CE0B68; Sat, 23 Jul 2022 10:03:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 073ABC341C7; Sat, 23 Jul 2022 10:02:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570579; bh=BrG/XoHrnxpbzose77TytGGeC3aIeV236WuPjsEdWSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s3WdleQGB9eMgk2iOVc3uRkFR8zKmzqGnabGBT7HcgFycB39JCry16z3UcVP1vHb4 NvJ1MQ50pyz5LWdZMyS2yiBw7FbBxMbVTwcjnS+eG/Th3mWFCQRtKvhaHeCrJQEFF6 hzfjTvoHIgLtj1SERE3z2ZOLzaZccaIKoVszt5Fc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Vasily Gorbik , Ingo Molnar , Peter Zijlstra Subject: [PATCH 5.10 146/148] objtool: Fix elf_create_undef_symbol() endianness Date: Sat, 23 Jul 2022 11:55:58 +0200 Message-Id: <20220723095305.214720457@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Vasily Gorbik commit 46c7405df7de8deb97229eacebcee96d61415f3f upstream. Currently x86 cross-compilation fails on big endian system with: x86_64-cross-ld: init/main.o: invalid string offset 488112128 >=3D 6229 f= or section `.strtab' Mark new ELF data in elf_create_undef_symbol() as symbol, so that libelf does endianness handling correctly. Fixes: 2f2f7e47f052 ("objtool: Add elf_create_undef_symbol()") Signed-off-by: Vasily Gorbik Signed-off-by: Ingo Molnar Acked-by: Peter Zijlstra Link: https://lore.kernel.org/r/patch-1.thread-6c9df9.git-d39264656387.your= -ad-here.call-01620841104-ext-2554@work.hours Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/objtool/elf.c | 1 + 1 file changed, 1 insertion(+) --- a/tools/objtool/elf.c +++ b/tools/objtool/elf.c @@ -961,6 +961,7 @@ static int elf_add_string(struct elf *el data->d_buf =3D str; data->d_size =3D strlen(str) + 1; data->d_align =3D 1; + data->d_type =3D ELF_T_SYM; =20 len =3D strtab->len; strtab->len +=3D data->d_size; From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C449C433EF for ; Sat, 23 Jul 2022 10:12:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239390AbiGWKMH (ORCPT ); Sat, 23 Jul 2022 06:12:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239599AbiGWKJq (ORCPT ); Sat, 23 Jul 2022 06:09:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03582CE534; Sat, 23 Jul 2022 03:03:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D29DD61263; Sat, 23 Jul 2022 10:03:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DB503C341C0; Sat, 23 Jul 2022 10:03:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570582; bh=vZOb0znR5b2+sXNreQRuTLxggurukIZ1dLOaYUIDq9o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0mTHOoyiJcvqBeOzBy1wsd2LpjjIztLOtODOKg4jAILZVUzAHtXQCNrzWtdpPl0Oh mBuFdV5E98SdqrOjFseFJBVpJcK1i9Sv/lvi4oQ9lAi/AJR2Y9NePTdgk6v0IpgGlb PNiiIi0gP/5N0yfX9reNxnupqniZ++5aws/22a8g= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Borislav Petkov , Juergen Gross , Arnaldo Carvalho de Melo Subject: [PATCH 5.10 147/148] tools arch: Update arch/x86/lib/mem{cpy,set}_64.S copies used in perf bench mem memcpy - again Date: Sat, 23 Jul 2022 11:55:59 +0200 Message-Id: <20220723095305.473622561@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Arnaldo Carvalho de Melo commit fb24e308b6310541e70d11a3f19dc40742974b95 upstream. To bring in the change made in this cset: 5e21a3ecad1500e3 ("x86/alternative: Merge include files") This just silences these perf tools build warnings, no change in the tools: Warning: Kernel ABI header at 'tools/arch/x86/lib/memcpy_64.S' differs fr= om latest version at 'arch/x86/lib/memcpy_64.S' diff -u tools/arch/x86/lib/memcpy_64.S arch/x86/lib/memcpy_64.S Warning: Kernel ABI header at 'tools/arch/x86/lib/memset_64.S' differs fr= om latest version at 'arch/x86/lib/memset_64.S' diff -u tools/arch/x86/lib/memset_64.S arch/x86/lib/memset_64.S Cc: Borislav Petkov Cc: Juergen Gross Signed-off-by: Arnaldo Carvalho de Melo Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/arch/x86/lib/memcpy_64.S | 2 +- tools/arch/x86/lib/memset_64.S | 2 +- tools/include/asm/{alternative-asm.h =3D> alternative.h} | 0 tools/arch/x86/lib/memcpy_64.S | 2 +- tools/arch/x86/lib/memset_64.S | 2 +- tools/include/asm/alternative-asm.h | 10 ---------- tools/include/asm/alternative.h | 10 ++++++++++ 4 files changed, 12 insertions(+), 12 deletions(-) rename tools/include/asm/{alternative-asm.h =3D> alternative.h} (100%) --- a/tools/arch/x86/lib/memcpy_64.S +++ b/tools/arch/x86/lib/memcpy_64.S @@ -4,7 +4,7 @@ #include #include #include -#include +#include #include =20 .pushsection .noinstr.text, "ax" --- a/tools/arch/x86/lib/memset_64.S +++ b/tools/arch/x86/lib/memset_64.S @@ -3,7 +3,7 @@ =20 #include #include -#include +#include #include =20 /* --- a/tools/include/asm/alternative-asm.h +++ /dev/null @@ -1,10 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _TOOLS_ASM_ALTERNATIVE_ASM_H -#define _TOOLS_ASM_ALTERNATIVE_ASM_H - -/* Just disable it so we can build arch/x86/lib/memcpy_64.S for perf bench= : */ - -#define altinstruction_entry # -#define ALTERNATIVE_2 # - -#endif --- /dev/null +++ b/tools/include/asm/alternative.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _TOOLS_ASM_ALTERNATIVE_ASM_H +#define _TOOLS_ASM_ALTERNATIVE_ASM_H + +/* Just disable it so we can build arch/x86/lib/memcpy_64.S for perf bench= : */ + +#define altinstruction_entry # +#define ALTERNATIVE_2 # + +#endif From nobody Wed Apr 15 07:00:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B97D5C433EF for ; Sat, 23 Jul 2022 10:13:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239095AbiGWKNq (ORCPT ); Sat, 23 Jul 2022 06:13:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239265AbiGWKNS (ORCPT ); Sat, 23 Jul 2022 06:13:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4C118C59A; Sat, 23 Jul 2022 03:03:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C94C5611CD; Sat, 23 Jul 2022 10:03:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D5669C341C0; Sat, 23 Jul 2022 10:03:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1658570611; bh=Q1TD7F9ns2LiP9OwYh3h/gmbzL6OmOLlOkUZovjoCiI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=xFpX/iLmmynZhwDeReVxBl3gVtlrYK8rAYt6x81vW8zQUj0aLG+HT84aS/lzSeOcD NlmPVp6HRsFduQnmsBzqtFxUAzs+kXemxig9GreHgPNf8txPaLItb9XXZ8kJ7WWfuO w4ztiFZyd/BOiwuNBtTZO/LuXssnWkJBqM9b8f9Y= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Adrian Hunter , Borislav Petkov , Jiri Olsa , Namhyung Kim , Arnaldo Carvalho de Melo Subject: [PATCH 5.10 148/148] tools headers: Remove broken definition of __LITTLE_ENDIAN Date: Sat, 23 Jul 2022 11:56:00 +0200 Message-Id: <20220723095305.747384732@linuxfoundation.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220723095224.302504400@linuxfoundation.org> References: <20220723095224.302504400@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Arnaldo Carvalho de Melo commit fa2c02e5798c17c89cbb3135940086ebe07e5c9f upstream. The linux/kconfig.h file was copied from the kernel but the line where with the generated/autoconf.h include from where the CONFIG_ entries would come from was deleted, as tools/ build system don't create that file, so we ended up always defining just __LITTLE_ENDIAN as CONFIG_CPU_BIG_ENDIAN was nowhere to be found. This in turn ended up breaking the build in some systems where __LITTLE_ENDIAN was already defined, such as the androind NDK. So just ditch that block that depends on the CONFIG_CPU_BIG_ENDIAN define. The kconfig.h file was copied just to get IS_ENABLED() and a 'make -C tools/all' doesn't breaks with this removal. Fixes: 93281c4a96572a34 ("x86/insn: Add an insn_decode() API") Cc: Adrian Hunter Cc: Borislav Petkov Cc: Jiri Olsa Cc: Namhyung Kim Link: http://lore.kernel.org/lkml/YO8hK7lqJcIWuBzx@kernel.org Signed-off-by: Arnaldo Carvalho de Melo Signed-off-by: Greg Kroah-Hartman Tested-by: Florian Fainelli Tested-by: Guenter Roeck Tested-by: Linux Kernel Functional Testing Tested-by: Pavel Machek (CIP) Tested-by: Rudi Heitbaum Tested-by: Sudip Mukherjee --- tools/include/linux/kconfig.h | 6 ------ 1 file changed, 6 deletions(-) --- a/tools/include/linux/kconfig.h +++ b/tools/include/linux/kconfig.h @@ -4,12 +4,6 @@ =20 /* CONFIG_CC_VERSION_TEXT (Do not delete this comment. See help in Kconfig= ) */ =20 -#ifdef CONFIG_CPU_BIG_ENDIAN -#define __BIG_ENDIAN 4321 -#else -#define __LITTLE_ENDIAN 1234 -#endif - #define __ARG_PLACEHOLDER_1 0, #define __take_second_arg(__ignored, val, ...) val