From nobody Thu Nov 28 19:52:08 2024 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54CB718BB93 for ; Fri, 27 Sep 2024 19:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; cv=none; b=gQo8oRWiIEhnCl4O8PFMrCAo+LXvxiet/vmU4dAJG6lQ4s4tIdtCfR7Nr4CHEnwwOoX4yu/3J2xUinBUe2E87OkcMRK/p3Wzigu9tkplXSPV5Sd4IDRha05guOPGTLzFtvqbyt/JYHoRKItoc2B0x0XaMYoKVW8BI7cltd7xzuQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; c=relaxed/simple; bh=SdYo3SJOO9E8y9fJnK8dpXCIWoKBTfNaRJvwZWvNKMs=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=qZrspznmYJNWBl75VYBIpymWubJ4bZ4AS13dDbT+L5lb9q4bYf+9kcRxspVV2eoW1/0tIYrddLHk7eQCvi2mLjj9RuYvfSEJvWWtHHVePpV6iAp6gwpgD5fiYvbFWoqdAn+9xRgSKx1oyA06/Q3XfeKnjzBSQu0R/ALCBkVicv0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=ZQ6DnIAg; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ZQ6DnIAg" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=npS5RDCIOIEGGznuy7YxziiBw+pbWzoev+F3iwc3u00=; b=ZQ6DnIAgR7G7AqqDvPD8Md5+T1 D5t174OSOpna4+Y34g9EVHzzZRRL0SHMk0j+Z/YDUhYPKKHkQbQpJfXE4VqUpCI02NztzmigrWs85 Zj81q41x1JpJ6da11NZ3M6Mgz8KcfaTNZPj5cawTaYg6JsaLwN0Xp5l+eP5qzTHfy/tMQzTmkTJsi iVwyTYpg02vS6YWuWUK1lQ6LhCwYwDoOQbVo/MqnWuQ0D5pr+nR/c8irEsQ9+VsLLUYznpffzCEcc yx2HIDyDSU0CJZ1RTmlmFn47ugUtSHZNuBmWRGbDw+bGM/FYZDgKwFPT/aahmmkuFtVfasUXnJda2 XOVJkDCA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzF-0000000AVch-0YFZ; Fri, 27 Sep 2024 19:50:46 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id AADC0300BAE; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194924.390462812@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:48:57 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 01/14] x86/cfi: Wreck things... References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With the introduction of kCFI the addition of ENDBR to SYM_FUNC_START* no longer suffices to make the function indirectly callable. This now requires the use of SYM_TYPED_FUNC_START. As such, remove the implicit ENDBR from SYM_FUNC_START* and add some explicit annotations to (mostly) fix things up again. This leaves: vmlinux.o: warning: objtool: .export_symbol+0xc8: data relocation to !ENDBR= : entry_ibpb+0x0 vmlinux.o: warning: objtool: .export_symbol+0xe8: data relocation to !ENDBR= : asm_load_gs_index+0x0 vmlinux.o: warning: objtool: .export_symbol+0xf8: data relocation to !ENDBR= : clear_bhb_loop+0x0 vmlinux.o: warning: objtool: .export_symbol+0x16218: data relocation to !EN= DBR: rdmsr_safe_regs+0x0 vmlinux.o: warning: objtool: .export_symbol+0x16228: data relocation to !EN= DBR: wrmsr_safe_regs+0x0 vmlinux.o: warning: objtool: .export_symbol+0x16238: data relocation to !EN= DBR: __sw_hweight32+0x0 vmlinux.o: warning: objtool: .export_symbol+0x16248: data relocation to !EN= DBR: __sw_hweight64+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c450: data relocation to !EN= DBR: clear_page_rep+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c460: data relocation to !EN= DBR: clear_page_orig+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c470: data relocation to !EN= DBR: clear_page_erms+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c480: data relocation to !EN= DBR: rep_stos_alternative+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c4a0: data relocation to !EN= DBR: copy_page+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c4b0: data relocation to !EN= DBR: rep_movs_alternative+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c4c0: data relocation to !EN= DBR: __copy_user_nocache+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c580: data relocation to !EN= DBR: __get_user_1+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c590: data relocation to !EN= DBR: __get_user_2+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c5a0: data relocation to !EN= DBR: __get_user_4+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c5b0: data relocation to !EN= DBR: __get_user_8+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c5c0: data relocation to !EN= DBR: __get_user_nocheck_1+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c5d0: data relocation to !EN= DBR: __get_user_nocheck_2+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c5e0: data relocation to !EN= DBR: __get_user_nocheck_4+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c5f0: data relocation to !EN= DBR: __get_user_nocheck_8+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c620: data relocation to !EN= DBR: memmove+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c630: data relocation to !EN= DBR: memmove+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c640: data relocation to !EN= DBR: memset+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c650: data relocation to !EN= DBR: memset+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c660: data relocation to !EN= DBR: __put_user_1+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c670: data relocation to !EN= DBR: __put_user_nocheck_1+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c680: data relocation to !EN= DBR: __put_user_2+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c690: data relocation to !EN= DBR: __put_user_nocheck_2+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c6a0: data relocation to !EN= DBR: __put_user_4+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c6b0: data relocation to !EN= DBR: __put_user_nocheck_4+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c6c0: data relocation to !EN= DBR: __put_user_8+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c6d0: data relocation to !EN= DBR: __put_user_nocheck_8+0x0 vmlinux.o: warning: objtool: .export_symbol+0x3c9f0: data relocation to !EN= DBR: entry_untrain_ret+0x0 Which states that while these functions are exported and (directly) callable, they cannot be called indirectly. There are two solutions: - exclude the .export_symbol section from validation; effectively saying that having linkable but not indirectly callable exports are fine by default, or - make all of those use SYM_TYPED_FUNC_START to restore the traditional (and expected, but less secure?) behaviour. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/crypto/aesni-intel_asm.S | 2 ++ arch/x86/entry/calling.h | 1 + arch/x86/entry/entry_64.S | 1 + arch/x86/include/asm/linkage.h | 18 ++++++------------ arch/x86/include/asm/paravirt_types.h | 12 +++++++++++- arch/x86/kernel/acpi/madt_playdead.S | 1 + arch/x86/kernel/acpi/wakeup_64.S | 1 + arch/x86/kernel/ftrace_64.S | 4 ++++ arch/x86/kernel/irqflags.S | 1 + arch/x86/kernel/paravirt.c | 14 ++++++++++++-- arch/x86/mm/mem_encrypt_boot.S | 1 + arch/x86/power/hibernate_asm_64.S | 2 ++ arch/x86/xen/xen-asm.S | 5 +++++ 13 files changed, 48 insertions(+), 15 deletions(-) --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -17,6 +17,7 @@ */ =20 #include +#include #include =20 #define STATE1 %xmm0 @@ -1071,6 +1072,7 @@ SYM_FUNC_END(_aesni_inc) * size_t len, u8 *iv) */ SYM_FUNC_START(aesni_ctr_enc) + ANNOTATE_NOENDBR FRAME_BEGIN cmp $16, LEN jb .Lctr_enc_just_ret --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -431,6 +431,7 @@ For 32-bit we have the following convent /* rdi: arg1 ... normal C conventions. rax is saved/restored. */ .macro THUNK name, func SYM_FUNC_START(\name) + ANNOTATE_NOENDBR pushq %rbp movq %rsp, %rbp =20 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -175,6 +175,7 @@ SYM_CODE_END(entry_SYSCALL_64) */ .pushsection .text, "ax" SYM_FUNC_START(__switch_to_asm) + ANNOTATE_NOENDBR /* * Save callee-saved registers * This must match the order in inactive_task_frame --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -119,33 +119,27 @@ =20 /* SYM_FUNC_START -- use for global functions */ #define SYM_FUNC_START(name) \ - SYM_START(name, SYM_L_GLOBAL, SYM_F_ALIGN) \ - ENDBR + SYM_START(name, SYM_L_GLOBAL, SYM_F_ALIGN) =20 /* SYM_FUNC_START_NOALIGN -- use for global functions, w/o alignment */ #define SYM_FUNC_START_NOALIGN(name) \ - SYM_START(name, SYM_L_GLOBAL, SYM_A_NONE) \ - ENDBR + SYM_START(name, SYM_L_GLOBAL, SYM_A_NONE) =20 /* SYM_FUNC_START_LOCAL -- use for local functions */ #define SYM_FUNC_START_LOCAL(name) \ - SYM_START(name, SYM_L_LOCAL, SYM_F_ALIGN) \ - ENDBR + SYM_START(name, SYM_L_LOCAL, SYM_F_ALIGN) =20 /* SYM_FUNC_START_LOCAL_NOALIGN -- use for local functions, w/o alignment = */ #define SYM_FUNC_START_LOCAL_NOALIGN(name) \ - SYM_START(name, SYM_L_LOCAL, SYM_A_NONE) \ - ENDBR + SYM_START(name, SYM_L_LOCAL, SYM_A_NONE) =20 /* SYM_FUNC_START_WEAK -- use for weak functions */ #define SYM_FUNC_START_WEAK(name) \ - SYM_START(name, SYM_L_WEAK, SYM_F_ALIGN) \ - ENDBR + SYM_START(name, SYM_L_WEAK, SYM_F_ALIGN) =20 /* SYM_FUNC_START_WEAK_NOALIGN -- use for weak functions, w/o alignment */ #define SYM_FUNC_START_WEAK_NOALIGN(name) \ - SYM_START(name, SYM_L_WEAK, SYM_A_NONE) \ - ENDBR + SYM_START(name, SYM_L_WEAK, SYM_A_NONE) =20 #endif /* _ASM_X86_LINKAGE_H */ =20 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -246,7 +246,17 @@ extern struct paravirt_patch_template pv =20 int paravirt_disable_iospace(void); =20 -/* This generates an indirect call based on the operation type number. */ +/* + * This generates an indirect call based on the operation type number. + * + * Since alternatives run after enabling CET/IBT -- the latter setting/cle= aring + * capabilities and the former requiring all capabilities being finalized = -- + * these indirect calls are subject to IBT and the paravirt stubs should h= ave + * ENDBR on. + * + * OTOH since this is effectively a __nocfi indirect call, the paravirt st= ubs + * don't need to bother with CFI prefixes. + */ #define PARAVIRT_CALL \ ANNOTATE_RETPOLINE_SAFE \ "call *%[paravirt_opptr];" --- a/arch/x86/kernel/acpi/madt_playdead.S +++ b/arch/x86/kernel/acpi/madt_playdead.S @@ -14,6 +14,7 @@ * rsi: PGD of the identity mapping */ SYM_FUNC_START(asm_acpi_mp_play_dead) + ANNOTATE_NOENDBR /* Turn off global entries. Following CR3 write will flush them. */ movq %cr4, %rdx andq $~(X86_CR4_PGE), %rdx --- a/arch/x86/kernel/acpi/wakeup_64.S +++ b/arch/x86/kernel/acpi/wakeup_64.S @@ -17,6 +17,7 @@ * Hooray, we are in Long 64-bit mode (but still running in low memory) */ SYM_FUNC_START(wakeup_long64) + ANNOTATE_NOENDBR movq saved_magic(%rip), %rax movq $0x123456789abcdef0, %rdx cmpq %rdx, %rax --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -146,12 +146,14 @@ SYM_FUNC_END(ftrace_stub_graph) #ifdef CONFIG_DYNAMIC_FTRACE =20 SYM_FUNC_START(__fentry__) + ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT RET SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) =20 SYM_FUNC_START(ftrace_caller) + ANNOTATE_NOENDBR /* save_mcount_regs fills in first two parameters */ save_mcount_regs =20 @@ -197,6 +199,7 @@ SYM_FUNC_END(ftrace_caller); STACK_FRAME_NON_STANDARD_FP(ftrace_caller) =20 SYM_FUNC_START(ftrace_regs_caller) + ANNOTATE_NOENDBR /* Save the current flags before any operations that can change them */ pushfq =20 @@ -317,6 +320,7 @@ SYM_FUNC_END(ftrace_stub_direct_tramp) #else /* ! CONFIG_DYNAMIC_FTRACE */ =20 SYM_FUNC_START(__fentry__) + ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT =20 cmpq $ftrace_stub, ftrace_trace_function --- a/arch/x86/kernel/irqflags.S +++ b/arch/x86/kernel/irqflags.S @@ -9,6 +9,7 @@ */ .pushsection .noinstr.text, "ax" SYM_FUNC_START(native_save_fl) + ENDBR pushf pop %_ASM_AX RET --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -106,6 +106,16 @@ static noinstr void pv_native_write_cr2( native_write_cr2(val); } =20 +static noinstr unsigned long pv_native_read_cr3(void) +{ + return __native_read_cr3(); +} + +static noinstr void pv_native_write_cr3(unsigned long cr3) +{ + native_write_cr3(cr3); +} + static noinstr unsigned long pv_native_get_debugreg(int regno) { return native_get_debugreg(regno); @@ -199,8 +209,8 @@ struct paravirt_patch_template pv_ops =3D #ifdef CONFIG_PARAVIRT_XXL .mmu.read_cr2 =3D __PV_IS_CALLEE_SAVE(pv_native_read_cr2), .mmu.write_cr2 =3D pv_native_write_cr2, - .mmu.read_cr3 =3D __native_read_cr3, - .mmu.write_cr3 =3D native_write_cr3, + .mmu.read_cr3 =3D pv_native_read_cr3, + .mmu.write_cr3 =3D pv_native_write_cr3, =20 .mmu.pgd_alloc =3D __paravirt_pgd_alloc, .mmu.pgd_free =3D paravirt_nop, --- a/arch/x86/mm/mem_encrypt_boot.S +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -72,6 +72,7 @@ SYM_FUNC_START(sme_encrypt_execute) SYM_FUNC_END(sme_encrypt_execute) =20 SYM_FUNC_START(__enc_copy) + ANNOTATE_NOENDBR /* * Routine used to encrypt memory in place. * This routine must be run outside of the kernel proper since --- a/arch/x86/power/hibernate_asm_64.S +++ b/arch/x86/power/hibernate_asm_64.S @@ -26,6 +26,7 @@ /* code below belongs to the image kernel */ .align PAGE_SIZE SYM_FUNC_START(restore_registers) + ANNOTATE_NOENDBR /* go back to the original page tables */ movq %r9, %cr3 =20 @@ -119,6 +120,7 @@ SYM_FUNC_END(restore_image) =20 /* code below has been relocated to a safe page */ SYM_FUNC_START(core_restore_code) + ANNOTATE_NOENDBR /* switch to temporary page tables */ movq %rax, %cr3 /* flush TLB */ --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -28,6 +28,7 @@ * non-zero. */ SYM_FUNC_START(xen_irq_disable_direct) + ENDBR movb $1, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) RET SYM_FUNC_END(xen_irq_disable_direct) @@ -67,6 +68,7 @@ SYM_FUNC_END(check_events) * then enter the hypervisor to get them handled. */ SYM_FUNC_START(xen_irq_enable_direct) + ENDBR FRAME_BEGIN /* Unmask events */ movb $0, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) @@ -97,6 +99,7 @@ SYM_FUNC_END(xen_irq_enable_direct) * x86 use opposite senses (mask vs enable). */ SYM_FUNC_START(xen_save_fl_direct) + ENDBR testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) setz %ah addb %ah, %ah @@ -104,6 +107,7 @@ SYM_FUNC_START(xen_save_fl_direct) SYM_FUNC_END(xen_save_fl_direct) =20 SYM_FUNC_START(xen_read_cr2) + ENDBR FRAME_BEGIN _ASM_MOV PER_CPU_VAR(xen_vcpu), %_ASM_AX _ASM_MOV XEN_vcpu_info_arch_cr2(%_ASM_AX), %_ASM_AX @@ -112,6 +116,7 @@ SYM_FUNC_START(xen_read_cr2) SYM_FUNC_END(xen_read_cr2); =20 SYM_FUNC_START(xen_read_cr2_direct) + ENDBR FRAME_BEGIN _ASM_MOV PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_arch_cr2), %_ASM_AX FRAME_END From nobody Thu Nov 28 19:52:08 2024 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30AC41C3F3A for ; Fri, 27 Sep 2024 19:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466663; cv=none; b=hvYcm14ux1wqRBabSyAzO98stmraq4pZO3+O0CA4ZVQf4EJKwNtoYYQZBQX7Tzw7oa1xaDRcF/LqEpxF+jfAEOSqSas9Klb6Fh+kGd0V0K4Btl8WscZquhnDpkKbElbVw14zpoCMMIlq+ontyamDb0egGR79s/NXGLYJ7nAEMKc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466663; c=relaxed/simple; bh=VidXdPtbh2bTuFCIDSaznp2IkB1WCS04s8nddvipaSw=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=nB+W3oKbEsq3N0pEpcBqxaK7uBM8pGuVKu1xemXC60yE65oZufP0e+nWR35IGkjF/yzreirsclfPgKo1+ZawJofa1aD8Y8RN68sJa8RwKZtyoIvehF0CA8bmO7CPX/q2/Z3S8sJJVuS7nnMFjPVHjPVwv1RQQUkB1eomjgSspsc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=IHoubrKT; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="IHoubrKT" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=tblBls7liJOC0BRcZvZJjuzCBGu7d6vWrUIadWdBE1c=; b=IHoubrKTrxc7PXlAeVGX/qvMoW 1sytf1UEZNtCUTwh1ae2KGF+/uYsDmss/uNLQq3v+KUApw3NSt6JA157HO6rZRs42EgYi2y04cO2+ PlwCct2kf4y+iPRSstN4GHWbAhqP9Tc/bKCxuMi9t/yKMwJE/0fjGYXO77B9Z4rHBGnXJH35JQ088 SLgMGxfWrrSJJl0WPH7dS57EXGloHn+wo8jh4j4a4TNYqYe+rg1AYFEVYiC4c4CmQ+VIpiSyxfUYN dzFi383uiJHT85ApaT6U5Trlm01THdhRLvsQJ75d60FjIH+edcfa4OrdZmv4xqr+r95b+SgDGSYfe U6DZcjdg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzF-00000002Onl-430Y; Fri, 27 Sep 2024 19:50:46 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id AEE10300C1F; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194924.504708296@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:48:58 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 02/14] x86/boot: Mark start_secondary() with __noendbr References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The handoff between the boot stubs and start_secondary() are before IBT is enabled and is definitely not subject to kCFI. As such, suppress all that f= or this function. Notably when the ENDBR poison would become fatal (ud1 instead of nop) this = will trigger a tripple fault because we haven't set up the IDT to handle #UD yet. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/smpboot.c | 3 ++- include/linux/objtool.h | 13 ++++++++++--- 2 files changed, 12 insertions(+), 4 deletions(-) --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -228,7 +228,7 @@ static void ap_calibrate_delay(void) /* * Activate a secondary processor. */ -static void notrace start_secondary(void *unused) +static void notrace __noendbr start_secondary(void *unused) { /* * Don't put *anything* except direct CPU state initialization @@ -313,6 +313,7 @@ static void notrace start_secondary(void wmb(); cpu_startup_entry(CPUHP_AP_ONLINE_IDLE); } +NOENDBR_SYMBOL(start_secondary); =20 /* * The bootstrap kernel entry code has set these up. Save them for --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -45,12 +45,18 @@ #define STACK_FRAME_NON_STANDARD_FP(func) #endif =20 -#define ANNOTATE_NOENDBR \ - "986: \n\t" \ +#define __ANNOTATE_NOENDBR(label) \ ".pushsection .discard.noendbr\n\t" \ - ".long 986b\n\t" \ + ".long " #label "\n\t" \ ".popsection\n\t" =20 +#define NOENDBR_SYMBOL(func) \ + asm(__ANNOTATE_NOENDBR(func)) + +#define ANNOTATE_NOENDBR \ + "986: \n\t" \ + __ANNOTATE_NOENDBR(986b) + #define ASM_REACHABLE \ "998:\n\t" \ ".pushsection .discard.reachable\n\t" \ @@ -157,6 +163,7 @@ #define STACK_FRAME_NON_STANDARD_FP(func) #define ANNOTATE_NOENDBR #define ASM_REACHABLE +#define NOENDBR_SYMBOL(func) #else #define ANNOTATE_INTRA_FUNCTION_CALL .macro UNWIND_HINT type:req sp_reg=3D0 sp_offset=3D0 signal=3D0 From nobody Thu Nov 28 19:52:08 2024 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 752011C230E for ; Fri, 27 Sep 2024 19:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; cv=none; b=EWH6utuqnpvUmWUH8ipvUI4G+jjwvDhdI5URSppTmc0oOqz+IDuwYKxJ4e9LmUH0NOpx4uV+f3j1CWi/UZ+Lju074Ot47z3N01E26Hi/dtj1BDnNNXoRaWTLIgIh4i1qsktnp/cKUWNZfsAyOWzPUE+OtctAHTO6X8ZiCL1ikPQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; c=relaxed/simple; bh=Xxd+7KjV5+HgGdOk/MOUF4qFidlu+p84pKGG8meR5hM=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=HQQWf3WOtoO23HDoOO2wYjyNry8DyCV/gzM0fVmpI2Siyvzm9MEa18NLUdjq7iGzVwR1z0BqoxpL0jPhVT/G5UIckw2RRBykiv6ZN51DbZl7ojwPmmoRyDrQLyfYte35wAVnkTvaqZA6Kn643ddDbAkV1YVYobUu7KoknGP+P/0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Xn14qP8p; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Xn14qP8p" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ojwF+rKwH0i5yOPUIhXM5uKj2cf0V2HnToDyhfHscac=; b=Xn14qP8patnfXqdGBG/0UwI1KI I9M7TgwjNlrzDshTssQ6+9A9bbbAEkL8tr8Q/MZSynTMEUnQwdb2ObMPVUCVx62fCLe2bP+kT3ArX ZDTA+p3tVaEpcCvb85KjocMBMZX7Xx+s332NugCi+tIM/4zEg5GigoxnF+XfPezxaILcGMogaDV0D iIIznTnz5ALufpgtoO7Z3huRfbjq+rSMoWA4jT9PmZqqjm2LNnGeC4t91xeWMVi4ttoBy16B1jJqZ 1pJz7LUyj4l6eHLjfiECXnve+1dJp+sQKUyhTEP25MB7CUY7nYhCjDSvwMDuqCpgRwGx5vf+AzCA7 6XK2TPDw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzF-0000000AVcg-1Ehg; Fri, 27 Sep 2024 19:50:46 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id B330C300CBA; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194924.617641991@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:48:59 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 03/14] x86/alternative: Simplify callthunk patching References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that paravirt call patching is implemented using alternatives, it is possible to avoid having to patch the alternative sites by including the altinstr_replacement calls in the call_sites list. This means we're now stacking relative adjustments like so: callthunks_patch_builtin_calls(): patches all function calls to target: func() -> func()-10 since the CALL accounting lives in the CALL_PADDING. This explicitly includes .altinstr_replacement alt_replace_call(): patches: x86_BUG() -> target() this patching is done in a relative manner, and will preserve the above adjustment, meaning that with calldepth patching it will do: x86_BUG()-10 -> target()-10 apply_relocation(): does code relocation, and adjusts all RIP-relative instructions to the new location, also in a relative manner. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/alternative.h | 1 - arch/x86/kernel/alternative.c | 8 ++++---- arch/x86/kernel/callthunks.c | 13 ------------- arch/x86/kernel/module.c | 17 ++++++----------- tools/objtool/arch/x86/decode.c | 1 + tools/objtool/check.c | 12 ++---------- 6 files changed, 13 insertions(+), 39 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -109,7 +109,6 @@ struct module; =20 struct callthunk_sites { s32 *call_start, *call_end; - struct alt_instr *alt_start, *alt_end; }; =20 #ifdef CONFIG_CALL_THUNKS --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1701,14 +1701,14 @@ void __init alternative_instructions(voi apply_retpolines(__retpoline_sites, __retpoline_sites_end); apply_returns(__return_sites, __return_sites_end); =20 - apply_alternatives(__alt_instructions, __alt_instructions_end); - /* - * Now all calls are established. Apply the call thunks if - * required. + * Adjust call CALL instructions to point to func()-10, including those + * in .altinstr_replacement. */ callthunks_patch_builtin_calls(); =20 + apply_alternatives(__alt_instructions, __alt_instructions_end); + /* * Seal all functions that do not have their address taken. */ --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -239,21 +239,10 @@ patch_call_sites(s32 *start, s32 *end, c } =20 static __init_or_module void -patch_alt_call_sites(struct alt_instr *start, struct alt_instr *end, - const struct core_text *ct) -{ - struct alt_instr *a; - - for (a =3D start; a < end; a++) - patch_call((void *)&a->instr_offset + a->instr_offset, ct); -} - -static __init_or_module void callthunks_setup(struct callthunk_sites *cs, const struct core_text *ct) { prdbg("Patching call sites %s\n", ct->name); patch_call_sites(cs->call_start, cs->call_end, ct); - patch_alt_call_sites(cs->alt_start, cs->alt_end, ct); prdbg("Patching call sites done%s\n", ct->name); } =20 @@ -262,8 +251,6 @@ void __init callthunks_patch_builtin_cal struct callthunk_sites cs =3D { .call_start =3D __call_sites, .call_end =3D __call_sites_end, - .alt_start =3D __alt_instructions, - .alt_end =3D __alt_instructions_end }; =20 if (!cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -275,12 +275,7 @@ int module_finalize(const Elf_Ehdr *hdr, void *rseg =3D (void *)returns->sh_addr; apply_returns(rseg, rseg + returns->sh_size); } - if (alt) { - /* patch .altinstructions */ - void *aseg =3D (void *)alt->sh_addr; - apply_alternatives(aseg, aseg + alt->sh_size); - } - if (calls || alt) { + if (calls) { struct callthunk_sites cs =3D {}; =20 if (calls) { @@ -288,13 +283,13 @@ int module_finalize(const Elf_Ehdr *hdr, cs.call_end =3D (void *)calls->sh_addr + calls->sh_size; } =20 - if (alt) { - cs.alt_start =3D (void *)alt->sh_addr; - cs.alt_end =3D (void *)alt->sh_addr + alt->sh_size; - } - callthunks_patch_module_calls(&cs, me); } + if (alt) { + /* patch .altinstructions */ + void *aseg =3D (void *)alt->sh_addr; + apply_alternatives(aseg, aseg + alt->sh_size); + } if (ibt_endbr) { void *iseg =3D (void *)ibt_endbr->sh_addr; apply_seal_endbr(iseg, iseg + ibt_endbr->sh_size); --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -845,5 +845,6 @@ bool arch_is_rethunk(struct symbol *sym) bool arch_is_embedded_insn(struct symbol *sym) { return !strcmp(sym->name, "retbleed_return_thunk") || + !strcmp(sym->name, "srso_alias_safe_ret") || !strcmp(sym->name, "srso_safe_ret"); } --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1360,15 +1360,6 @@ static void annotate_call_site(struct ob if (!sym) sym =3D reloc->sym; =20 - /* - * Alternative replacement code is just template code which is - * sometimes copied to the original instruction. For now, don't - * annotate it. (In the future we might consider annotating the - * original instruction if/when it ever makes sense to do so.) - */ - if (!strcmp(insn->sec->name, ".altinstr_replacement")) - return; - if (sym->static_call_tramp) { list_add_tail(&insn->call_node, &file->static_call_list); return; @@ -1426,7 +1417,8 @@ static void annotate_call_site(struct ob return; } =20 - if (insn->type =3D=3D INSN_CALL && !insn->sec->init) + if (insn->type =3D=3D INSN_CALL && !insn->sec->init && + !insn->_call_dest->embedded_insn) list_add_tail(&insn->call_node, &file->call_list); =20 if (!sibling && dead_end_function(file, sym)) From nobody Thu Nov 28 19:52:08 2024 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30A711C3F39 for ; Fri, 27 Sep 2024 19:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466663; cv=none; b=clnwtD2bD7kAfMnWy8VfwUb5YhWIcVhy7cdYHuOYBVFL6dMC8eVxrOugOFeZoU6AJyMIWRblzTreESARJbZpLmhMXJdEtvxwTlnOrF9yuN4DcqmrlJWD+BNmcNBqRsJ332Wx6085k7Z/2f//2yeIChnY0Xi42WJD1SXayEMEsU4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466663; c=relaxed/simple; bh=NVm/025w8faWRZl4j6He/bJwsRYRrGpoESgWPtfeBLY=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=UYhm35g8W+OzzRUb4RY8FkTnaIopnQfQbB3eQjr6y1ZkDA7sf4DtSdwoczLtF7YorBbARKdcW9/pmtM+nGO5IfN+fMxdaadLxB0uAC/hSoKBFpJ2gXOBpBFr26+7i9Nv1Su4Twn6qKPo6JavU5C01Pna+mA/XPPwF+j+GNDQAiU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=ptTL+7Cx; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ptTL+7Cx" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=TGYcIgm/NfsHCXx6/jnmfMThkiwjCScFwzdD18YngiQ=; b=ptTL+7CxYX4kgbtUONjtSSdjpX xteWt78pbrBoBP2DQBEwMOYdfcNPIE+F57bVYpDC2FlIuKHTxAX1gFD+rDQOBKhhayszhY/YApZKO sWp4HygnedMOPW7nlAr3NplETJKLz9zSv7+rqLomr/wujtFZn1kV4pzPasE/KhS4o6a+BXBQpGIq8 ajli2WncZBo+jBQQsNbcHPJ2nRP+DzhqtH1TuRCye4nf6eZk4AQ/RyLEENTZO7K+lWQz+w2Ixd27R mvCWcIJJ65eZcO7XsevGQ7I/ChnamFY9Qi5uznZVnqgE4xxgR7AgmAPOW0GdSI0P2lEkw2k1IY140 jpNg5GbA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzF-00000002Onk-3zdX; Fri, 27 Sep 2024 19:50:46 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id B728D30114C; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194924.730517860@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:00 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 04/14] objtool/x86: Add .tail_call_sites References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to allow rewriting all direct (tail) calls to ENDBR to go +4, also generate a tail-call sites section. This must be different from the .call_sites, because call depth tracking specifically cares about the CALL instruction, but not JMP instructions. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/vmlinux.lds.S | 9 ++++- tools/objtool/check.c | 52 +++++++++++++++++++++++++++= ++--- tools/objtool/include/objtool/objtool.h | 1=20 tools/objtool/objtool.c | 1=20 4 files changed, 58 insertions(+), 5 deletions(-) --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -285,6 +285,7 @@ SECTIONS *(.return_sites) __return_sites_end =3D .; } +#endif =20 . =3D ALIGN(8); .call_sites : AT(ADDR(.call_sites) - LOAD_OFFSET) { @@ -292,7 +293,13 @@ SECTIONS *(.call_sites) __call_sites_end =3D .; } -#endif + + . =3D ALIGN(8); + .tail_call_sites : AT(ADDR(.tail_call_sites) - LOAD_OFFSET) { + __tail_call_sites =3D .; + *(.tail_call_sites) + __tail_call_sites_end =3D .; + } =20 #ifdef CONFIG_X86_KERNEL_IBT . =3D ALIGN(8); --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -893,7 +893,6 @@ static int create_cfi_sections(struct ob =20 sec =3D find_section_by_name(file->elf, ".cfi_sites"); if (sec) { - INIT_LIST_HEAD(&file->call_list); WARN("file already has .cfi_sites section, skipping"); return 0; } @@ -1018,6 +1017,45 @@ static int create_direct_call_sections(s return 0; } =20 +static int create_direct_tail_call_sections(struct objtool_file *file) +{ + struct instruction *insn; + struct section *sec; + int idx; + + sec =3D find_section_by_name(file->elf, ".tail_call_sites"); + if (sec) { + INIT_LIST_HEAD(&file->tail_call_list); + WARN("file already has .tail_call_sites section, skipping"); + return 0; + } + + if (list_empty(&file->tail_call_list)) + return 0; + + idx =3D 0; + list_for_each_entry(insn, &file->tail_call_list, call_node) + idx++; + + sec =3D elf_create_section_pair(file->elf, ".tail_call_sites", + sizeof(unsigned int), idx, idx); + if (!sec) + return -1; + + idx =3D 0; + list_for_each_entry(insn, &file->tail_call_list, call_node) { + + if (!elf_init_reloc_text_sym(file->elf, sec, + idx * sizeof(unsigned int), idx, + insn->sec, insn->offset)) + return -1; + + idx++; + } + + return 0; +} + /* * Warnings shouldn't be reported for ignored functions. */ @@ -1417,9 +1455,12 @@ static void annotate_call_site(struct ob return; } =20 - if (insn->type =3D=3D INSN_CALL && !insn->sec->init && - !insn->_call_dest->embedded_insn) - list_add_tail(&insn->call_node, &file->call_list); + if (!insn->sec->init && !insn->_call_dest->embedded_insn) { + if (insn->type =3D=3D INSN_CALL) + list_add_tail(&insn->call_node, &file->call_list); + else + list_add_tail(&insn->call_node, &file->tail_call_list); + } =20 if (!sibling && dead_end_function(file, sym)) insn->dead_end =3D true; @@ -4802,6 +4843,9 @@ int check(struct objtool_file *file) ret =3D create_direct_call_sections(file); if (ret < 0) goto out; + ret =3D create_direct_tail_call_sections(file); + if (ret < 0) + goto out; warnings +=3D ret; } } --- a/tools/objtool/include/objtool/objtool.h +++ b/tools/objtool/include/objtool/objtool.h @@ -28,6 +28,7 @@ struct objtool_file { struct list_head mcount_loc_list; struct list_head endbr_list; struct list_head call_list; + struct list_head tail_call_list; bool ignore_unreachables, hints, rodata; =20 unsigned int nr_endbr; --- a/tools/objtool/objtool.c +++ b/tools/objtool/objtool.c @@ -106,6 +106,7 @@ struct objtool_file *objtool_open_read(c INIT_LIST_HEAD(&file.mcount_loc_list); INIT_LIST_HEAD(&file.endbr_list); INIT_LIST_HEAD(&file.call_list); + INIT_LIST_HEAD(&file.tail_call_list); file.ignore_unreachables =3D opts.no_unreachable; file.hints =3D false; From nobody Thu Nov 28 19:52:08 2024 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52C4713BAE2 for ; Fri, 27 Sep 2024 19:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; cv=none; b=cuOKMhz+t4MR7C7nrkrXmbwNC+++Tso8nYMeJd9e4BHs/VNXx2oWhzTJNNf0LF/1NwhGYMKqudPiQziVx3vaX77c2BIvv191t/jx4aakAF/IDBuuqt3otDl0A/T98Z/CsJowVkob9nR2i38NILvlK2FgbW/zGvE11D0ZE5YMDIw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; c=relaxed/simple; bh=K8q9IXe8qIpviHcOuGYp515Mdl6owfhbVqnML6iwuaI=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Z3yApyTCvgN/74fiCQhyoP8BA7FYKq52JCOndBum2ilAiuomF62tAz5t4m1QIHJtUKsmjsc7o83wZSH9mOckdBy7ZrLcvMWTM/hsuds+voZzfF3CG6ScBCtz2P3dY3coHwAgJctBcH9Y1IRZTOUh0eWDXkhdTCwJvtOAoacr5Vw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Jl0g8Vz+; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Jl0g8Vz+" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=SSSBP/b3nCsMC3CL687Z1QQwJNwcB3hiCtNug+UhlAU=; b=Jl0g8Vz+g8Bwg/dQ+k/jrkLVDK CQS7h42THZD/ibBZHJn6J2pPFRfWmlPgzW2C2VxuOVbQhGWbcvD+JaCMBuqJXtUdV/QA7DOFhOL2R 9M35gN80mKhCNedZEMlV5H8rmHoM+0SYcs2Zf6Q/Xk2Gf2NuMfZ3hkMpsPL7b8xhVgF9ZOjmzyIaf o0un1xGG59iwfHU5eAAD4Imi399IVeMETTk5i3i7B6qAMPLAHutKCrTe3brkNubYzDZNaK06W23z9 iJf806RiLPiC+3PA0/3HMYyGOtfQzeLoB+tJGljKbpJYHv0JQKRGjidgfhD87eMxfsW43LpXwtUl2 ImzitelQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzG-0000000AVct-0mYP; Fri, 27 Sep 2024 19:50:46 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id BAEE130119B; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194924.839606244@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:01 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 05/14] objtool: Rename the skylake hack to --direct-call References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to use the .call_sites and .tail_call_sites sections beyond the call-depth-tracking skylake thing, give it a more generic name. Specifically it will be used to 'relink' all direct calls to avoid ENDBR. A compiler could only hope to do this using an LTO pass, and even then the dynamic linker would need to get involved. Signed-off-by: Peter Zijlstra (Intel) --- scripts/Makefile.lib | 2 +- tools/objtool/builtin-check.c | 8 ++------ tools/objtool/check.c | 18 +++++++++--------- tools/objtool/include/objtool/builtin.h | 2 +- 4 files changed, 13 insertions(+), 17 deletions(-) --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -267,7 +267,7 @@ objtool :=3D $(objtree)/tools/objtool/objt =20 objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HACK) +=3D --hacks=3Djump_label objtool-args-$(CONFIG_HAVE_NOINSTR_HACK) +=3D --hacks=3Dnoinstr -objtool-args-$(CONFIG_MITIGATION_CALL_DEPTH_TRACKING) +=3D --hacks=3Dskyla= ke +objtool-args-$(CONFIG_MITIGATION_CALL_DEPTH_TRACKING) +=3D --direct-call objtool-args-$(CONFIG_X86_KERNEL_IBT) +=3D --ibt objtool-args-$(CONFIG_FINEIBT) +=3D --cfi objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) +=3D --mcount --- a/tools/objtool/builtin-check.c +++ b/tools/objtool/builtin-check.c @@ -57,17 +57,13 @@ static int parse_hacks(const struct opti found =3D true; } =20 - if (!str || strstr(str, "skylake")) { - opts.hack_skylake =3D true; - found =3D true; - } - return found ? 0 : -1; } =20 static const struct option check_options[] =3D { OPT_GROUP("Actions:"), - OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr,skylake= ", "patch toolchain bugs/limitations", parse_hacks), + OPT_CALLBACK_OPTARG('h', "hacks", NULL, NULL, "jump_label,noinstr", "patc= h toolchain bugs/limitations", parse_hacks), + OPT_BOOLEAN(0, "direct-call", &opts.direct_call, "annotate direct calls= "), OPT_BOOLEAN('i', "ibt", &opts.ibt, "validate and annotate IBT"), OPT_BOOLEAN('m', "mcount", &opts.mcount, "annotate mcount/fentry calls fo= r ftrace"), OPT_BOOLEAN('n', "noinstr", &opts.noinstr, "validate noinstr rules"), --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -4838,16 +4838,16 @@ int check(struct objtool_file *file) if (ret < 0) goto out; warnings +=3D ret; + } =20 - if (opts.hack_skylake) { - ret =3D create_direct_call_sections(file); - if (ret < 0) - goto out; - ret =3D create_direct_tail_call_sections(file); - if (ret < 0) - goto out; - warnings +=3D ret; - } + if (opts.direct_call) { + ret =3D create_direct_call_sections(file); + if (ret < 0) + goto out; + ret =3D create_direct_tail_call_sections(file); + if (ret < 0) + goto out; + warnings +=3D ret; } =20 if (opts.mcount) { --- a/tools/objtool/include/objtool/builtin.h +++ b/tools/objtool/include/objtool/builtin.h @@ -12,7 +12,7 @@ struct opts { bool dump_orc; bool hack_jump_label; bool hack_noinstr; - bool hack_skylake; + bool direct_call; bool ibt; bool mcount; bool noinstr; From nobody Thu Nov 28 19:52:08 2024 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B5311C3F3C for ; Fri, 27 Sep 2024 19:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466664; cv=none; b=BdWVVUmzpYktdzmr4jiXQbmhxcjSA6eDZJ2mlvYmTrowKYNpf86mF9QX7RCf0AP8fkfO2UQLsCfcl+gNMwW78+8THSPUrOjx1/nCDm43NTRF/iAU/Lc6hfvBFYJJ5b6bFrb1c06TzrR/qhkScnxRGnO8uj5QDALbVVhbv+pyptc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466664; c=relaxed/simple; bh=kQaFbC4eVC0UP983bJ9oNVqQfL86oa2qo9PsnHN/zAM=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=KL9JkbGlq6sKqAKK5kluDR4uFN9bRweccNc+rxXlNWBISMaqxK3TE8kZPjuXBkO73SEJ7boBuUBqDrotPLIDHsinc2LwYAi+6Q1IfXLzR7R2MNZPrdb+GIRQQPKvm/DkiLuLteuskEZGYe0F9hin3IhBl6Sd1W9aDlAobZL//6Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=hxcwUHMB; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="hxcwUHMB" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ROsGWBJKd8LF8eUnYQyESejYA9lwOOpnxR5fs87Okc8=; b=hxcwUHMByxq3qxzPft9/txX7I5 67nI2rlZQMU9ZN9a/Hrpox1s73cCepAwmccVqqHdTFsjY9pr2OMDrvvcNpmrMNooVJK8Yb79cc72h 9/lRhSElfLhLHsF11/HMqnpwE367cvrT1p2qbZyA3W/23RBLQU3unzG7PrnEHdoK+4GTNFDxf54Pt wtBu6t1YyEeXC9EXWOVWKCusLJl9HMVTPv4+Hb0Q4BmUcn6I2KEOEeY7fjGtl4jhRzrpy+eAHbNsZ XYIPkGIVQS+h8+F1FLiIKD8Kp7R6P+GkTfd3aFXgLYINGMu6D3J7Zl3TS5lmgjbFqDz4Z4KDavLbM EpV2bjVA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzG-00000002Onn-2eO0; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id BEFC63011E4; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194924.949420892@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:02 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 06/14] x86/traps: Prepare for ENDBR poison UD1 usage References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Prepare for ENDBR poison to become UD1 by making #UD recognise it as a valid BUG address so we get a nice BUG splat when we trip on one. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/bug.h | 6 ++- arch/x86/include/asm/ibt.h | 8 +++- arch/x86/kernel/cet.c | 18 ++++++--- arch/x86/kernel/traps.c | 89 +++++++++++++++++++++++++++++++++-------= ----- 4 files changed, 90 insertions(+), 31 deletions(-) --- a/arch/x86/include/asm/bug.h +++ b/arch/x86/include/asm/bug.h @@ -22,8 +22,10 @@ #define SECOND_BYTE_OPCODE_UD2 0x0b =20 #define BUG_NONE 0xffff -#define BUG_UD1 0xfffe -#define BUG_UD2 0xfffd +#define BUG_UD2 0xfffe +#define BUG_UD1 0xfffd +#define BUG_UD1_UBSAN 0xfffc +#define BUG_UD1_ENDBR 0xfffb =20 #ifdef CONFIG_GENERIC_BUG =20 --- a/arch/x86/include/asm/ibt.h +++ b/arch/x86/include/asm/ibt.h @@ -41,7 +41,7 @@ _ASM_PTR fname "\n\t" \ ".popsection\n\t" =20 -static inline __attribute_const__ u32 gen_endbr(void) +static __always_inline __attribute_const__ u32 gen_endbr(void) { u32 endbr; =20 @@ -56,7 +56,7 @@ static inline __attribute_const__ u32 ge return endbr; } =20 -static inline __attribute_const__ u32 gen_endbr_poison(void) +static __always_inline __attribute_const__ u32 gen_endbr_poison(void) { /* * 4 byte NOP that isn't NOP4 (in fact it is OSP NOP3), such that it @@ -77,6 +77,10 @@ static inline bool is_endbr(u32 val) extern __noendbr u64 ibt_save(bool disable); extern __noendbr void ibt_restore(u64 save); =20 +struct pt_regs; + +extern bool __do_kernel_cp_fault(struct pt_regs *regs); + #else /* __ASSEMBLY__ */ =20 #ifdef CONFIG_X86_64 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -81,6 +81,17 @@ static void do_user_cp_fault(struct pt_r =20 static __ro_after_init bool ibt_fatal =3D true; =20 +bool __do_kernel_cp_fault(struct pt_regs *regs) +{ + pr_err("Missing ENDBR: %pS\n", (void *)instruction_pointer(regs)); + if (!ibt_fatal) { + printk(KERN_DEFAULT CUT_HERE); + __warn(__FILE__, __LINE__, (void *)regs->ip, TAINT_WARN, regs, NULL); + return true; + } + return false; +} + static void do_kernel_cp_fault(struct pt_regs *regs, unsigned long error_c= ode) { if ((error_code & CP_EC) !=3D CP_ENDBR) { @@ -93,12 +104,9 @@ static void do_kernel_cp_fault(struct pt return; } =20 - pr_err("Missing ENDBR: %pS\n", (void *)instruction_pointer(regs)); - if (!ibt_fatal) { - printk(KERN_DEFAULT CUT_HERE); - __warn(__FILE__, __LINE__, (void *)regs->ip, TAINT_WARN, regs, NULL); + if (__do_kernel_cp_fault(regs)) return; - } + BUG(); } =20 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -94,10 +94,18 @@ __always_inline int is_valid_bugaddr(uns =20 /* * Check for UD1 or UD2, accounting for Address Size Override Prefixes. - * If it's a UD1, get the ModRM byte to pass along to UBSan. + * If it's a UD1, further decode to determine its use: + * + * UBSan{0}: 67 0f b9 00 ud1 (%eax),%eax + * UBSan{10}: 67 0f b9 40 10 ud1 0x10(%eax),%eax + * static_call: 0f b9 cc ud1 %esp,%ecx + * ENDBR Poison: 0f b9 48 00 ud1 0(%eax),%edx + * + * Notably UBSAN uses EAX, static_call uses ECX and ENDBR uses EDX. */ -__always_inline int decode_bug(unsigned long addr, u32 *imm) +__always_inline int decode_bug(unsigned long addr, s32 *imm, int *len) { + unsigned long start =3D addr; u8 v; =20 if (addr < TASK_SIZE_MAX) @@ -110,24 +118,41 @@ __always_inline int decode_bug(unsigned return BUG_NONE; =20 v =3D *(u8 *)(addr++); - if (v =3D=3D SECOND_BYTE_OPCODE_UD2) + if (v =3D=3D SECOND_BYTE_OPCODE_UD2) { + *len =3D addr - start; return BUG_UD2; + } =20 - if (!IS_ENABLED(CONFIG_UBSAN_TRAP) || v !=3D SECOND_BYTE_OPCODE_UD1) + if (v !=3D SECOND_BYTE_OPCODE_UD1) return BUG_NONE; =20 - /* Retrieve the immediate (type value) for the UBSAN UD1 */ - v =3D *(u8 *)(addr++); - if (X86_MODRM_RM(v) =3D=3D 4) - addr++; - *imm =3D 0; - if (X86_MODRM_MOD(v) =3D=3D 1) - *imm =3D *(u8 *)addr; - else if (X86_MODRM_MOD(v) =3D=3D 2) - *imm =3D *(u32 *)addr; - else - WARN_ONCE(1, "Unexpected MODRM_MOD: %u\n", X86_MODRM_MOD(v)); + v =3D *(u8 *)(addr++); /* ModRM */ + + /* Decode immediate, if present */ + if (X86_MODRM_MOD(v) !=3D 3) { + if (X86_MODRM_RM(v) =3D=3D 4) + addr++; /* Skip SIB byte */ + + if (X86_MODRM_MOD(v) =3D=3D 1) { + *imm =3D *(s8 *)addr; + addr +=3D 1; + + } else if (X86_MODRM_MOD(v) =3D=3D 2) { + *imm =3D *(s32 *)addr; + addr +=3D 4; + } + } + + /* record instruction length */ + *len =3D addr - start; + + if (X86_MODRM_REG(v) =3D=3D 0) /* EAX */ + return BUG_UD1_UBSAN; + + if (X86_MODRM_REG(v) =3D=3D 2 && /* RDX */ + *len =3D=3D ENDBR_INSN_SIZE) + return BUG_UD1_ENDBR; =20 return BUG_UD1; } @@ -258,8 +283,8 @@ static inline void handle_invalid_op(str static noinstr bool handle_bug(struct pt_regs *regs) { bool handled =3D false; - int ud_type; - u32 imm; + int ud_type, ud_len; + s32 ud_imm; =20 /* * Normally @regs are unpoisoned by irqentry_enter(), but handle_bug() @@ -267,7 +292,7 @@ static noinstr bool handle_bug(struct pt * irqentry_enter(). */ kmsan_unpoison_entry_regs(regs); - ud_type =3D decode_bug(regs->ip, &imm); + ud_type =3D decode_bug(regs->ip, &ud_imm, &ud_len); if (ud_type =3D=3D BUG_NONE) return handled; =20 @@ -281,15 +306,35 @@ static noinstr bool handle_bug(struct pt */ if (regs->flags & X86_EFLAGS_IF) raw_local_irq_enable(); - if (ud_type =3D=3D BUG_UD2) { + + switch (ud_type) { + case BUG_UD2: if (report_bug(regs->ip, regs) =3D=3D BUG_TRAP_TYPE_WARN || handle_cfi_failure(regs) =3D=3D BUG_TRAP_TYPE_WARN) { - regs->ip +=3D LEN_UD2; + regs->ip +=3D ud_len; + handled =3D true; + } + break; + + case BUG_UD1_UBSAN: + if (IS_ENABLED(CONFIG_UBSAN_TRAP)) { + pr_crit("%s at %pS\n", + report_ubsan_failure(regs, ud_imm), + (void *)regs->ip); + } + break; + + case BUG_UD1_ENDBR: + if (__do_kernel_cp_fault(regs)) { + regs->ip +=3D ud_len; handled =3D true; } - } else if (IS_ENABLED(CONFIG_UBSAN_TRAP)) { - pr_crit("%s at %pS\n", report_ubsan_failure(regs, imm), (void *)regs->ip= ); + break; + + default: + break; } + if (regs->flags & X86_EFLAGS_IF) raw_local_irq_disable(); instrumentation_end(); From nobody Thu Nov 28 19:52:08 2024 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA9C51C2DA9 for ; Fri, 27 Sep 2024 19:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466664; cv=none; b=VoYLrZ7Gp8cNj6421WBcVRjhh52WkFdMYRzdDQkNP5hcnupzhkvSBTIwOn+M6oSRgPiUjB+oXNF8ZRpbEauXzebk7m3mDMVcSsMG5Sivw9X/wx7qKjIHF6tjS1xvpUZ+a2ybP/tEK1FvNtjeMmbJ2yp+HhLXqBY6m7tq1MRX2Sg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466664; c=relaxed/simple; bh=6XTpH0cJVK/rYZRKbB+rtFzk/FmVEmfgufGBwZzuFdY=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=NxIwe68MnweFgCzXabHg9EjSIB7QwGDLzHLJxwlrMHZdYxTBOMGGMViFXCKbzJyrKy3A/Mobbk3+ixAHGxUy2PC/GrnobQubOqpAQlh+qVvn1CcKo4Zs6VfbDKtW5M45mb7xE275hc5lnimNGkMuou0VnzXWurSk0dRbixDW8FU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=bHyuwQvQ; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="bHyuwQvQ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=NfcLlfp3hJU+ZP8AcK7eQOnm49ijhTFuRZhXuJusWdk=; b=bHyuwQvQT7o2T0qGfWDXlhlujs aaa3jE9tpaELGtF+iv0LWr8h+2sgtvCsnZ6rwEw9UgxIOqt7mbdF8bYmuecTbBas0ZL0WRPkkrmnt yIwsh6E/PYYjGgKB2fsx24RPKQt3pRfFsw3EcLdvjA5FwdMZHlIZZFUKSzLoXrsimKbqJvBMBlg0g MC3+3CqaEn793NhhzrxZtSdCjy3gDYbs0+rfnmmzQeSZBxIO2Ek+VraLHWDu2qHG3f4JdShp9kl3q FEb4mnf8kyCxvGitSosv01lQv8GujGbcRmRFTUXASFHgs7LxSQ0yj6K+UppxESRHzoOF8h0AcsakS FZiembbg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzG-00000002Ono-2o0X; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C2C7A3021DE; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.069013308@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:03 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, Andrii Nakryiko Subject: [PATCH 07/14] x86/ibt: Clean up is_endbr() References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pretty much every caller of is_endbr() actually wants to test something at = an address and ends up doing get_kernel_nofault(). Fold the lot into a more convenient helper. Note: this effectively reverts commit a8497506cd2c ("bpf: Avoid get_kernel_nofault() to fetch kprobe entry IP") which was entirely the wrong way to go about doing things. The right solution is to optimize get_kernel_nofault() itself, it really doesn't need STAC/CLAC nor the speculation barrier. Using __get_user is a historical hack, not a requirement. Cc: Andrii Nakryiko Signed-off-by: Peter Zijlstra (Intel) Acked-by: Alexei Starovoitov Acked-by: Andrii Nakryiko --- arch/x86/events/core.c | 2 +- arch/x86/include/asm/ibt.h | 5 +++-- arch/x86/kernel/alternative.c | 19 +++++++++++++------ arch/x86/kernel/kprobes/core.c | 11 +---------- arch/x86/net/bpf_jit_comp.c | 4 ++-- kernel/trace/bpf_trace.c | 14 ++------------ 6 files changed, 22 insertions(+), 33 deletions(-) --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -2845,7 +2845,7 @@ static bool is_uprobe_at_func_entry(stru return true; =20 /* endbr64 (64-bit only) */ - if (user_64bit_mode(regs) && is_endbr(*(u32 *)auprobe->insn)) + if (user_64bit_mode(regs) && is_endbr((u32 *)auprobe->insn)) return true; =20 return false; --- a/arch/x86/include/asm/ibt.h +++ b/arch/x86/include/asm/ibt.h @@ -65,7 +65,7 @@ static __always_inline __attribute_const return 0x001f0f66; /* osp nopl (%rax) */ } =20 -static inline bool is_endbr(u32 val) +static inline bool __is_endbr(u32 val) { if (val =3D=3D gen_endbr_poison()) return true; @@ -74,6 +74,7 @@ static inline bool is_endbr(u32 val) return val =3D=3D gen_endbr(); } =20 +extern __noendbr bool is_endbr(u32 *val); extern __noendbr u64 ibt_save(bool disable); extern __noendbr void ibt_restore(u64 save); =20 @@ -102,7 +103,7 @@ extern bool __do_kernel_cp_fault(struct =20 #define __noendbr =20 -static inline bool is_endbr(u32 val) { return false; } +static inline bool is_endbr(u32 *val) { return false; } =20 static inline u64 ibt_save(bool disable) { return 0; } static inline void ibt_restore(u64 save) { } --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -852,16 +852,23 @@ void __init_or_module noinline apply_ret =20 #ifdef CONFIG_X86_KERNEL_IBT =20 +__noendbr bool is_endbr(u32 *val) +{ + u32 endbr; + + if (get_kernel_nofault(endbr, val)) + return false; + + return __is_endbr(endbr); +} + static void poison_cfi(void *addr); =20 static void __init_or_module poison_endbr(void *addr, bool warn) { - u32 endbr, poison =3D gen_endbr_poison(); - - if (WARN_ON_ONCE(get_kernel_nofault(endbr, addr))) - return; + u32 poison =3D gen_endbr_poison(); =20 - if (!is_endbr(endbr)) { + if (!is_endbr(addr)) { WARN_ON_ONCE(warn); return; } @@ -988,7 +995,7 @@ static u32 cfi_seed __ro_after_init; static u32 cfi_rehash(u32 hash) { hash ^=3D cfi_seed; - while (unlikely(is_endbr(hash) || is_endbr(-hash))) { + while (unlikely(__is_endbr(hash) || __is_endbr(-hash))) { bool lsb =3D hash & 1; hash >>=3D 1; if (lsb) --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -373,16 +373,7 @@ static bool can_probe(unsigned long padd kprobe_opcode_t *arch_adjust_kprobe_addr(unsigned long addr, unsigned long= offset, bool *on_func_entry) { - u32 insn; - - /* - * Since 'addr' is not guaranteed to be safe to access, use - * copy_from_kernel_nofault() to read the instruction: - */ - if (copy_from_kernel_nofault(&insn, (void *)addr, sizeof(u32))) - return NULL; - - if (is_endbr(insn)) { + if (is_endbr((u32 *)addr)) { *on_func_entry =3D !offset || offset =3D=3D 4; if (*on_func_entry) offset =3D 4; --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -625,7 +625,7 @@ int bpf_arch_text_poke(void *ip, enum bp * See emit_prologue(), for IBT builds the trampoline hook is preceded * with an ENDBR instruction. */ - if (is_endbr(*(u32 *)ip)) + if (is_endbr(ip)) ip +=3D ENDBR_INSN_SIZE; =20 return __bpf_arch_text_poke(ip, t, old_addr, new_addr); @@ -2971,7 +2971,7 @@ static int __arch_prepare_bpf_trampoline /* skip patched call instruction and point orig_call to actual * body of the kernel function. */ - if (is_endbr(*(u32 *)orig_call)) + if (is_endbr(orig_call)) orig_call +=3D ENDBR_INSN_SIZE; orig_call +=3D X86_PATCH_SIZE; } --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1027,19 +1027,9 @@ static const struct bpf_func_proto bpf_g #ifdef CONFIG_X86_KERNEL_IBT static unsigned long get_entry_ip(unsigned long fentry_ip) { - u32 instr; + if (is_endbr((void *)(fentry_ip - ENDBR_INSN_SIZE))) + return fentry_ip - ENDBR_INSN_SIZE; =20 - /* We want to be extra safe in case entry ip is on the page edge, - * but otherwise we need to avoid get_kernel_nofault()'s overhead. - */ - if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) { - if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE))) - return fentry_ip; - } else { - instr =3D *(u32 *)(fentry_ip - ENDBR_INSN_SIZE); - } - if (is_endbr(instr)) - fentry_ip -=3D ENDBR_INSN_SIZE; return fentry_ip; } #else From nobody Thu Nov 28 19:52:08 2024 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B58C1C3F3D for ; Fri, 27 Sep 2024 19:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466663; cv=none; b=G/opAZu2ueRKrN9NfY0pTguaHXqkmX8fxUiSRXihBN8CB6qqfYmA5HQaNu54WysJaOFaxK27iRu/Okt44DYcbgkGt0PqeQYg1hsTg90sq818Cr8CMm6j2nh1ZY0b9Qih1pSPVH1+gSNC5PNUYCQfO6xCGBMmxX09td0Dt6kQL6Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466663; c=relaxed/simple; bh=C1E04fEG9kmHcFxhirOzJyn20hcQMNssZ7QzcqdfwTM=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=MVzMf8R2hAJyVstOxSfVHG8DviwRE0XlCQSPUXDFqQDffCoHLueO7hHRBWimrzI2xQ9dGy8P3boQTzvAsgwbll20zCYc2uvZ2faYZagngii/7q+vYtdjmn0+cFfkvY8zG+fMCOqeN3q8WF1XO9s7QNYc3dS+uEADImc6E9nmmqY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=S1Qoh504; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="S1Qoh504" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=mrg3n5kuv76jLLAOEDkCz2d0wdxL1/leoBvVupWWebI=; b=S1Qoh504qprUUGxIj6ZrXSo8r6 qO2fgjNDxjQN7IMGHeCbXneLBG5HIS82Ok4V0YMT90VDQjtxCySxnokKbS2D6zF+4P8pFLh3/glEE VxTFXjzRcuVLWaV7df1lDPKtqbVr0GPwAJhfP5EpuLy7oFGWVLG2wNhuqiIpidC4cT6HvjH+LATbS gsaBNJvDMP+s5DcsqiPaRR9vVgOnsnWKTv1Q6sey0ongG7JO3xq9sx8/vyyGDKQyi1ZJLJqtHOMyr 52vqLqX4P6VPKT6lyQpAdRlCw+Lw61EXk5IJFXDEZ6r+J5KFCk5m7Ru9NDDSlm/7b4akdopHqJAh5 cwtCjTAg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzG-00000002Onp-2xjE; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C6A3B308CAD; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.177636534@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:04 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 08/14] x86/ibt: Clean up poison_endbr() References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Basically, get rid of the .warn argument and explicitly don't call the function when we know there isn't an endbr. This makes the calling code clearer. Nota: perhaps don't add functions to .cfi_sites when the function doesn't have endbr -- OTOH why would the compiler emit the prefix if it has already determined there are no indirect callers and has omitted the ENDBR instruction. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/alternative.c | 42 +++++++++++++++++++++++++++++++++++--= ----- 1 file changed, 35 insertions(+), 7 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -864,14 +864,12 @@ __noendbr bool is_endbr(u32 *val) =20 static void poison_cfi(void *addr); =20 -static void __init_or_module poison_endbr(void *addr, bool warn) +static void __init_or_module poison_endbr(void *addr) { u32 poison =3D gen_endbr_poison(); =20 - if (!is_endbr(addr)) { - WARN_ON_ONCE(warn); + if (WARN_ON_ONCE(!is_endbr(addr))) return; - } =20 DPRINTK(ENDBR, "ENDBR at: %pS (%px)", addr, addr); =20 @@ -896,7 +894,7 @@ void __init_or_module noinline apply_sea for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; =20 - poison_endbr(addr, true); + poison_endbr(addr); if (IS_ENABLED(CONFIG_FINEIBT)) poison_cfi(addr - 16); } @@ -1203,6 +1201,14 @@ static int cfi_rewrite_preamble(s32 *sta void *addr =3D (void *)s + *s; u32 hash; =20 + /* + * When the function doesn't start with ENDBR the compiler will + * have determined there are no indirect calls to it and we + * don't need no CFI either. + */ + if (!is_endbr(addr + 16)) + continue; + hash =3D decode_preamble_hash(addr); if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", addr, addr, 5, addr)) @@ -1223,7 +1229,10 @@ static void cfi_rewrite_endbr(s32 *start for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; =20 - poison_endbr(addr+16, false); + if (!is_endbr(addr + 16)) + continue; + + poison_endbr(addr + 16); } } =20 @@ -1356,9 +1365,23 @@ static inline void poison_hash(void *add =20 static void poison_cfi(void *addr) { + /* + * Compilers manage to be inconsistent with ENDBR vs __cfi prefixes, + * some (static) functions for which they can determine the address + * is never taken do not get a __cfi prefix, but *DO* get an ENDBR. + * + * As such, these functions will get sealed, but we need to be careful + * to not unconditionally scribble the previous function. + */ switch (cfi_mode) { case CFI_FINEIBT: /* + * FineIBT prefix should start with an ENDBR. + */ + if (!is_endbr(addr)) + break; + + /* * __cfi_\func: * osp nopl (%rax) * subl $0, %r10d @@ -1366,12 +1389,17 @@ static void poison_cfi(void *addr) * ud2 * 1: nop */ - poison_endbr(addr, false); + poison_endbr(addr); poison_hash(addr + fineibt_preamble_hash); break; =20 case CFI_KCFI: /* + * kCFI prefix should start with a valid hash. + */ + if (!decode_preamble_hash(addr)) + break; + /* * __cfi_\func: * movl $0, %eax * .skip 11, 0x90 From nobody Thu Nov 28 19:52:08 2024 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FFF01C3F3F for ; Fri, 27 Sep 2024 19:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466665; cv=none; b=Z/K3s3336UfM3IH8y+sXF4atk8/tUFufmobnGUs6m/AfmgfUnCY+Dr1jWZ04wPcoUhbMfOsDG198nc0gszZTMWNNS/ixx+e26t1QV7c0JzP6dzJv3wCz++578qi/J1rpAJ9nwDd/SIFCQogHiskIB89LlknKwhNyxfq47RlR7vo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466665; c=relaxed/simple; bh=LQeE+d/8C5Kwujs85E9lBJI2GTyBoJtRrDx/j1NZMCw=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=KaLfT/gHZ+PMbhhl1dceKeO0fVJ0tm7XHTNg7Jw5cisKvQMoMAf53Vn/8iMyw3Pt7SsJlD4bQd1PyY2g3sBorgn4QAdqa0jo4yoxqRyS2gFmw/zx8gVU0thk9LSSR3mA4jZ1EuKVJzOe1uGLOSHNfbrOdsZNdHABLioK60tby1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=OMoaUILg; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OMoaUILg" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=mM3DKa+Drporg9ZpHE6FFapZzeMpklgm+8jriIQrSHY=; b=OMoaUILgRAD8nwxugy76yojmhh xKLCr8DWVvzbggKxBemrkj4ZqbEA0U0TLl3ZRkoRqfMAZo+lKL4fjdxQFZsauPQewyavveGsLgeE3 hIOWHa0jyIZnIJABV0fR3tHElMUsiHrXHpGt6eEW4gmLIhbrjt/mG/nKJoCUYCwgYuRNzHlIxP7oo rZibB8qIpU7Aorx6EG1/ez5jiOHSpHlLhM71JKQXnsQN+vhajLN/sjoWxfkY8fegeYu9jHqEyQdjU 5NvqltzAJ5KjJ6wHf6jqNY8Bxoa1hAhTLxrBZi74oEKqwYwYIntXw0hnhXKc88uhGOyUmxIywjRJj CSPcQbIA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzH-00000002Onw-2Ose; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CA7B3308CB6; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.283644921@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:05 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 09/14] x86/ibt: Implement IBT+ References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" As originally conceived, use UD1 based poison to seal IBT. This is also fat= al on !IBT hardware. This requires all direct (tail) calls avoid ever touching ENDBR. To that purpose rewrite them using the .call_sites and .tail_call_sites sections fr= om objtool --direct-call. Since this is a wee bit tricky, stick this in a 'def_bool y' config option. This again stacks 3 layers of relocation, just like the earlier callthunk p= atch. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/Kconfig | 4 + arch/x86/include/asm/alternative.h | 1=20 arch/x86/include/asm/cfi.h | 14 ++--- arch/x86/include/asm/ibt.h | 9 +++ arch/x86/kernel/alternative.c | 89 ++++++++++++++++++++++++++++++++= ++++- arch/x86/kernel/module.c | 9 +++ arch/x86/kernel/static_call.c | 23 ++++++++- arch/x86/net/bpf_jit_comp.c | 6 ++ scripts/Makefile.lib | 1=20 tools/objtool/check.c | 2=20 10 files changed, 144 insertions(+), 14 deletions(-) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1866,6 +1866,10 @@ config X86_KERNEL_IBT does significantly reduce the number of ENDBR instructions in the kernel image. =20 +config X86_KERNEL_IBT_PLUS + depends on X86_KERNEL_IBT + def_bool y + config X86_INTEL_MEMORY_PROTECTION_KEYS prompt "Memory Protection Keys" def_bool y --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -98,6 +98,7 @@ extern struct alt_instr __alt_instructio extern int alternatives_patched; =20 extern void alternative_instructions(void); +extern void apply_direct_call_offset(s32 *start, s32 *end); extern void apply_alternatives(struct alt_instr *start, struct alt_instr *= end); extern void apply_retpolines(s32 *start, s32 *end); extern void apply_returns(s32 *start, s32 *end); --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -31,12 +31,12 @@ * IBT: * * foo: - * endbr64 + * endbr64 / osp nop3 / ud1 0x0(%eax), %edx * ... code here ... * ret * * direct caller: - * call foo / call foo+4 + * call foo / call foo+4 # must be +4 when IBT+ * * indirect caller: * lea foo(%rip), %r11 @@ -50,12 +50,12 @@ * movl $0x12345678, %eax * # 11 nops when CONFIG_CALL_PADDING * foo: - * endbr64 # when IBT + * endbr64 / osp nop3 / ud1 # when IBT * ... code here ... * ret * * direct call: - * call foo # / call foo+4 when IBT + * call foo / call foo+4 # +4 possible with IBT, mandatory with IBT+ * * indirect call: * lea foo(%rip), %r11 @@ -72,16 +72,16 @@ * __cfi_foo: * endbr64 * subl 0x12345678, %r10d - * jz foo + * jz foo+4 * ud2 * nop * foo: - * osp nop3 # was endbr64 + * osp nop3 / ud1 0x0(%eax), %edx # was endbr64 * ... code here ... * ret * * direct caller: - * call foo / call foo+4 + * call foo / call foo+4 # must be +4 when IBT+ * * indirect caller: * lea foo(%rip), %r11 --- a/arch/x86/include/asm/ibt.h +++ b/arch/x86/include/asm/ibt.h @@ -58,11 +58,20 @@ static __always_inline __attribute_const =20 static __always_inline __attribute_const__ u32 gen_endbr_poison(void) { +#ifdef CONFIG_X86_KERNEL_IBT_PLUS + /* + * When we rewrite direct calls to +4, the endbr at +0 becomes unused, + * poisong it with a UD1 to trip !IBT hardware and to ensure these + * bytes are really unused. + */ + return 0x0048b90f; /* ud1 0x0(%eax), %edx */ +#else /* * 4 byte NOP that isn't NOP4 (in fact it is OSP NOP3), such that it * will be unique to (former) ENDBR sites. */ return 0x001f0f66; /* osp nopl (%rax) */ +#endif } =20 static inline bool __is_endbr(u32 val) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -171,6 +171,8 @@ static void add_nop(u8 *buf, unsigned in *buf =3D INT3_INSN_OPCODE; } =20 +extern s32 __call_sites[], __call_sites_end[]; +extern s32 __tail_call_sites[], __tail_call_sites_end[]; extern s32 __retpoline_sites[], __retpoline_sites_end[]; extern s32 __return_sites[], __return_sites_end[]; extern s32 __cfi_sites[], __cfi_sites_end[]; @@ -423,6 +425,19 @@ static int alt_replace_call(u8 *instr, u if (!target) target =3D bug; =20 +#ifdef CONFIG_X86_KERNEL_IBT_PLUS + /* + * apply_direct_call_offset() would have patched the alternative to + * "CALL BUG_func+4" *if* that function has an ENDBR. And *if* target + * also has an ENDBR this all works out. Except if BUG_func() and target() + * do not agree on the having of ENDBR, then things go sideways. + */ + if (is_endbr(bug)) + bug +=3D ENDBR_INSN_SIZE; + if (is_endbr(target)) + target +=3D ENDBR_INSN_SIZE; +#endif + /* (BUG_func - .) + (target - BUG_func) :=3D target - . */ *(s32 *)(insn_buff + 1) +=3D target - bug; =20 @@ -850,6 +865,64 @@ void __init_or_module noinline apply_ret =20 #endif /* CONFIG_MITIGATION_RETPOLINE && CONFIG_OBJTOOL */ =20 +#ifdef CONFIG_X86_KERNEL_IBT_PLUS +__init_or_module void apply_direct_call_offset(s32 *start, s32 *end) +{ + s32 *s; + + /* + * incompatible with call depth tracking + */ + if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) + return; + + for (s =3D start; s < end; s++) { + void *dest, *addr =3D (void *)s + *s; + struct insn insn; + int ret; + + ret =3D insn_decode_kernel(&insn, addr); + if (WARN_ON_ONCE(ret < 0)) + continue; + + dest =3D addr + insn.length + insn.immediate.value; + if (!is_endbr(dest)) + continue; + + switch (insn.opcode.bytes[0]) { + case CALL_INSN_OPCODE: + case JMP32_INSN_OPCODE: + apply_reloc(4, addr+1, 4); + continue; + + case JMP8_INSN_OPCODE: + case 0x70 ... 0x7f: /* Jcc.d8 */ + apply_reloc(1, addr+1, 4); + continue; + + case 0x0f: + switch (insn.opcode.bytes[1]) { + case 0x80 ... 0x8f: + apply_reloc(4, addr+2, 4); + continue; + + default: + break; + } + break; + + default: + break; + } + + printk("at: %pS, instruction: %*ph\n", addr, insn.length, addr); + BUG(); + } +} +#else +__init_or_module void apply_direct_call_offset(s32 *start, s32 *end) { } +#endif + #ifdef CONFIG_X86_KERNEL_IBT =20 __noendbr bool is_endbr(u32 *val) @@ -1067,6 +1140,7 @@ asm( ".pushsection .rodata \n" "fineibt_preamble_start: \n" " endbr64 \n" " subl $0x12345678, %r10d \n" + "fineibt_preamble_jcc: \n" " je fineibt_preamble_end \n" " ud2 \n" " nop \n" @@ -1075,10 +1149,13 @@ asm( ".pushsection .rodata \n" ); =20 extern u8 fineibt_preamble_start[]; +extern u8 fineibt_preamble_jcc[]; extern u8 fineibt_preamble_end[]; =20 -#define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_sta= rt) -#define fineibt_preamble_hash 7 +#define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_sta= rt) +#define fineibt_preamble_offset (fineibt_preamble_jcc - fineibt_preamble_s= tart) +#define fineibt_preamble_hash (fineibt_preamble_offset - 4) +#define fineibt_preamble_jccd8 (fineibt_preamble_offset + 1) =20 asm( ".pushsection .rodata \n" "fineibt_caller_start: \n" @@ -1217,6 +1294,8 @@ static int cfi_rewrite_preamble(s32 *sta text_poke_early(addr, fineibt_preamble_start, fineibt_preamble_size); WARN_ON(*(u32 *)(addr + fineibt_preamble_hash) !=3D 0x12345678); text_poke_early(addr + fineibt_preamble_hash, &hash, 4); + + *(u8 *)(addr + fineibt_preamble_jccd8) +=3D 4; } =20 return 0; @@ -1726,6 +1805,12 @@ void __init alternative_instructions(voi */ paravirt_set_cap(); =20 + /* + * Adjust all (tail) calls to func()+4 to avoid ENDBR. + */ + apply_direct_call_offset(__call_sites, __call_sites_end); + apply_direct_call_offset(__tail_call_sites, __tail_call_sites_end); + __apply_fineibt(__retpoline_sites, __retpoline_sites_end, __cfi_sites, __cfi_sites_end, true); =20 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -227,7 +227,7 @@ int module_finalize(const Elf_Ehdr *hdr, const Elf_Shdr *s, *alt =3D NULL, *locks =3D NULL, *orc =3D NULL, *orc_ip =3D NULL, *retpolines =3D NULL, *returns =3D NULL, *ibt_endbr =3D NULL, - *calls =3D NULL, *cfi =3D NULL; + *calls =3D NULL, *tails =3D NULL, *cfi =3D NULL; char *secstrings =3D (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; =20 for (s =3D sechdrs; s < sechdrs + hdr->e_shnum; s++) { @@ -245,6 +245,8 @@ int module_finalize(const Elf_Ehdr *hdr, returns =3D s; if (!strcmp(".call_sites", secstrings + s->sh_name)) calls =3D s; + if (!strcmp(".tail_call_sites", secstrings + s->sh_name)) + tails =3D s; if (!strcmp(".cfi_sites", secstrings + s->sh_name)) cfi =3D s; if (!strcmp(".ibt_endbr_seal", secstrings + s->sh_name)) @@ -284,6 +286,11 @@ int module_finalize(const Elf_Ehdr *hdr, } =20 callthunks_patch_module_calls(&cs, me); + apply_direct_call_offset(cs.call_start, cs.call_end); + } + if (tails) { + void *tseg =3D (void *)tails->sh_addr; + apply_direct_call_offset(tseg, tseg + tails->sh_size); } if (alt) { /* patch .altinstructions */ --- a/arch/x86/kernel/static_call.c +++ b/arch/x86/kernel/static_call.c @@ -50,6 +50,23 @@ asm (".global __static_call_return\n\t" "ret; int3\n\t" ".size __static_call_return, . - __static_call_return \n\t"); =20 +static void *translate_call_dest(void *dest, bool call) +{ + if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) { + if (!call) + return dest; + + return callthunks_translate_call_dest(dest); + } + + if (IS_ENABLED(CONFIG_X86_KERNEL_IBT_PLUS)) { + if (is_endbr(dest)) + dest +=3D 4; + } + + return dest; +} + static void __ref __static_call_transform(void *insn, enum insn_type type, void *func, bool modinit) { @@ -63,7 +80,7 @@ static void __ref __static_call_transfor =20 switch (type) { case CALL: - func =3D callthunks_translate_call_dest(func); + func =3D translate_call_dest(func, true); code =3D text_gen_insn(CALL_INSN_OPCODE, insn, func); if (func =3D=3D &__static_call_return0) { emulate =3D code; @@ -77,6 +94,7 @@ static void __ref __static_call_transfor break; =20 case JMP: + func =3D translate_call_dest(func, false); code =3D text_gen_insn(JMP32_INSN_OPCODE, insn, func); break; =20 @@ -92,7 +110,8 @@ static void __ref __static_call_transfor func =3D __static_call_return; if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) func =3D x86_return_thunk; - } + + } else func =3D translate_call_dest(func, false); =20 buf[0] =3D 0x0f; __text_gen_insn(buf+1, op, insn+1, func, 5); --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -555,6 +555,8 @@ static int emit_patch(u8 **pprog, void * =20 static int emit_call(u8 **pprog, void *func, void *ip) { + if (is_endbr(func)) + func +=3D ENDBR_INSN_SIZE; return emit_patch(pprog, func, ip, 0xE8); } =20 @@ -562,11 +564,13 @@ static int emit_rsb_call(u8 **pprog, voi { OPTIMIZER_HIDE_VAR(func); ip +=3D x86_call_depth_emit_accounting(pprog, func, ip); - return emit_patch(pprog, func, ip, 0xE8); + return emit_call(pprog, func, ip); } =20 static int emit_jump(u8 **pprog, void *func, void *ip) { + if (is_endbr(func)) + func +=3D ENDBR_INSN_SIZE; return emit_patch(pprog, func, ip, 0xE9); } =20 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -269,6 +269,7 @@ objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HA objtool-args-$(CONFIG_HAVE_NOINSTR_HACK) +=3D --hacks=3Dnoinstr objtool-args-$(CONFIG_MITIGATION_CALL_DEPTH_TRACKING) +=3D --direct-call objtool-args-$(CONFIG_X86_KERNEL_IBT) +=3D --ibt +objtool-args-$(CONFIG_X86_KERNEL_IBT_PLUS) +=3D --direct-call objtool-args-$(CONFIG_FINEIBT) +=3D --cfi objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) +=3D --mcount ifdef CONFIG_FTRACE_MCOUNT_USE_OBJTOOL --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1455,7 +1455,7 @@ static void annotate_call_site(struct ob return; } =20 - if (!insn->sec->init && !insn->_call_dest->embedded_insn) { + if (!insn->_call_dest->embedded_insn) { if (insn->type =3D=3D INSN_CALL) list_add_tail(&insn->call_node, &file->call_list); else From nobody Thu Nov 28 19:52:08 2024 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB9AD1C231F for ; Fri, 27 Sep 2024 19:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466656; cv=none; b=PXsVDCm+busvqZ4Oo+IIxc4/qga78Eyb6PL4ZSsV7sZ4HhMbfDfoH8v4gQfEOcKo3wfowvT4eLcBWeuHWurck4B4qcwykRyrgHO3B0qhOi4V5EeCnIBXHzyhmQmj6j8Lf4fw/t8nKRlDH2ldk6YFWTm7e2/KTVsghWHpkpxt0a0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466656; c=relaxed/simple; bh=5uvF95TgyiLT6G2jUE9sdhJmR7W94HYAXCCl/2TomHA=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=ID8U/ph5kznopZx75ZclZ5litAGAolU7DRifgeyAtopg2/4hH59+or3CiRUso+PXYFhjdloF5Q5TnG2J6pciiXA0GuvzhIDg5lgQ5nAoviCGJpuYvu1aKTFzZ6dVhqk9jmaXV7BkP/I2WGrmBqmRtu58AXyLDmwsrBr19sJuZGQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=TukinRj6; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="TukinRj6" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=2mwu19/OYEXX9hX1sQCEguvJJl/pwlDcYstsYT0uFu8=; b=TukinRj6T4fYMAE5hSrgZIorfx RvO6hF+dYbk7EVXrvjOQwA36ex+7V0HinnmTGTyqOu4zZdCbcImrJfAdBh88gDAAWZPTUsh5E6V1y nR3Av6gIsvPoZxQt4JQkf2hJpXKkTEF49qXh0UaGHJhPk3z2FFSPWtdNtERNPzVgtDlZcwJAxcTH2 vJeIz7K74zY3LQW9sK7GQSZGppGLWpqrKd4acvdB5mMp5rWVbdAZN3svDeD8Os7vFLl04u/+HxX9H S3nw61JZVuXmpN35IdmKIOMpwf0n0Du1h5rdKElTrAXM+boCtIyAryuqnmp1djhi2iwgt3Yt2cM0E ionBptYg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzG-0000000AVcw-3FDP; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CE635308CBD; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.389465951@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:06 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 10/14] x86/early_printk: Harden early_serial References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Scott found that mem32_serial_in() is an ideal speculation gadget, an indirectly callable function that takes an adddress and offset and immediately does a load. Use static_call() to take away the need for indirect calls and explicitly seal the functions to ensure they're not callable on IBT enabled parts. Reported-by: Scott Constable Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/early_printk.c | 51 ++++++++++++++++++++----------------= ----- 1 file changed, 25 insertions(+), 26 deletions(-) --- a/arch/x86/kernel/early_printk.c +++ b/arch/x86/kernel/early_printk.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 /* Simple VGA output */ #define VGABASE (__ISA_IO_base + 0xb8000) @@ -94,26 +95,28 @@ static unsigned long early_serial_base =3D #define DLL 0 /* Divisor Latch Low */ #define DLH 1 /* Divisor latch High */ =20 -static unsigned int io_serial_in(unsigned long addr, int offset) +static __noendbr unsigned int io_serial_in(unsigned long addr, int offset) { return inb(addr + offset); } +NOENDBR_SYMBOL(io_serial_in); =20 -static void io_serial_out(unsigned long addr, int offset, int value) +static __noendbr void io_serial_out(unsigned long addr, int offset, int va= lue) { outb(value, addr + offset); } +NOENDBR_SYMBOL(io_serial_out); =20 -static unsigned int (*serial_in)(unsigned long addr, int offset) =3D io_se= rial_in; -static void (*serial_out)(unsigned long addr, int offset, int value) =3D i= o_serial_out; +DEFINE_STATIC_CALL(serial_in, io_serial_in); +DEFINE_STATIC_CALL(serial_out, io_serial_out); =20 static int early_serial_putc(unsigned char ch) { unsigned timeout =3D 0xffff; =20 - while ((serial_in(early_serial_base, LSR) & XMTRDY) =3D=3D 0 && --timeout) + while ((static_call(serial_in)(early_serial_base, LSR) & XMTRDY) =3D=3D 0= && --timeout) cpu_relax(); - serial_out(early_serial_base, TXR, ch); + static_call(serial_out)(early_serial_base, TXR, ch); return timeout ? 0 : -1; } =20 @@ -131,16 +134,16 @@ static __init void early_serial_hw_init( { unsigned char c; =20 - serial_out(early_serial_base, LCR, 0x3); /* 8n1 */ - serial_out(early_serial_base, IER, 0); /* no interrupt */ - serial_out(early_serial_base, FCR, 0); /* no fifo */ - serial_out(early_serial_base, MCR, 0x3); /* DTR + RTS */ - - c =3D serial_in(early_serial_base, LCR); - serial_out(early_serial_base, LCR, c | DLAB); - serial_out(early_serial_base, DLL, divisor & 0xff); - serial_out(early_serial_base, DLH, (divisor >> 8) & 0xff); - serial_out(early_serial_base, LCR, c & ~DLAB); + static_call(serial_out)(early_serial_base, LCR, 0x3); /* 8n1 */ + static_call(serial_out)(early_serial_base, IER, 0); /* no interrupt */ + static_call(serial_out)(early_serial_base, FCR, 0); /* no fifo */ + static_call(serial_out)(early_serial_base, MCR, 0x3); /* DTR + RTS */ + + c =3D static_call(serial_in)(early_serial_base, LCR); + static_call(serial_out)(early_serial_base, LCR, c | DLAB); + static_call(serial_out)(early_serial_base, DLL, divisor & 0xff); + static_call(serial_out)(early_serial_base, DLH, (divisor >> 8) & 0xff); + static_call(serial_out)(early_serial_base, LCR, c & ~DLAB); } =20 #define DEFAULT_BAUD 9600 @@ -183,28 +186,26 @@ static __init void early_serial_init(cha /* Convert from baud to divisor value */ divisor =3D 115200 / baud; =20 - /* These will always be IO based ports */ - serial_in =3D io_serial_in; - serial_out =3D io_serial_out; - /* Set up the HW */ early_serial_hw_init(divisor); } =20 #ifdef CONFIG_PCI -static void mem32_serial_out(unsigned long addr, int offset, int value) +static __noendbr void mem32_serial_out(unsigned long addr, int offset, int= value) { u32 __iomem *vaddr =3D (u32 __iomem *)addr; /* shift implied by pointer type */ writel(value, vaddr + offset); } +NOENDBR_SYMBOL(mem32_serial_out); =20 -static unsigned int mem32_serial_in(unsigned long addr, int offset) +static __noendbr unsigned int mem32_serial_in(unsigned long addr, int offs= et) { u32 __iomem *vaddr =3D (u32 __iomem *)addr; /* shift implied by pointer type */ return readl(vaddr + offset); } +NOENDBR_SYMBOL(mem32_serial_in); =20 /* * early_pci_serial_init() @@ -278,15 +279,13 @@ static __init void early_pci_serial_init */ if ((bar0 & PCI_BASE_ADDRESS_SPACE) =3D=3D PCI_BASE_ADDRESS_SPACE_IO) { /* it is IO mapped */ - serial_in =3D io_serial_in; - serial_out =3D io_serial_out; early_serial_base =3D bar0 & PCI_BASE_ADDRESS_IO_MASK; write_pci_config(bus, slot, func, PCI_COMMAND, cmdreg|PCI_COMMAND_IO); } else { /* It is memory mapped - assume 32-bit alignment */ - serial_in =3D mem32_serial_in; - serial_out =3D mem32_serial_out; + static_call_update(serial_in, mem32_serial_in); + static_call_update(serial_out, mem32_serial_out); /* WARNING! assuming the address is always in the first 4G */ early_serial_base =3D (unsigned long)early_ioremap(bar0 & PCI_BASE_ADDRESS_MEM_MASK, 0x10); From nobody Thu Nov 28 19:52:08 2024 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30B111C3F3B for ; Fri, 27 Sep 2024 19:51:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466664; cv=none; b=erPM827v2yKxbsZySISMPn5Gw2Vj84CG8mkJrJqap1/HuklCDwM6YHe5LXYwwqL+AcuzvEKNGU46qEV7A+ZqotjuUuMNb9W9TcK9ov9gwpt87QxOfA8XcpM5nYuIcljplI/cTaEHROLA/5PPnKftkeP+Sce9f73JA+0AorISefo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466664; c=relaxed/simple; bh=py351CAWFlpDz3zpAmSiVjHSczEFA6gMsdfJlNnaVlw=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=awxhwCXwkj1ghFYGFROvQfmvYK5n1B2GYzhkByr8fXoJMQTAeeYCLDsTol1MVtjMeR4f7qJ11tKa7QsPkWasTk56P1pPwzVvLuE/A3KiC6Qo1dOyQswHnIX0s2a1+0ZQQ6iqbtoVxy6DumZdt3ki8jUytf0uslLbzVt+dBe39LI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=jXkJBXjj; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="jXkJBXjj" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=clz+4HfhflDwK3Xr2tLt2iX5CBtaxE7c7ymYTUq298k=; b=jXkJBXjjTICTsfXA3JjdRp7jTS lJIxgoX2jEc1dhrwQORV3ojKs08HUmu1mLpYpYtKYzcSyXnWOTgkYkh/8kSegtMwgUlqcEfQYyDZH CS9xkiNnqM7B7X7LZBtNSwyj+8mTiPACN2gJIVeylNzmjE/4qMoBdxDR40qX8mzS0YVcc8l0G6mYM DW1jhqkSkuHoBrGDz6hzVeeTrNWe81glnYSotmktBJJwWapJJtL/LcXrL0/SOIP/SHe9kvgjIZRj3 9QYYwEFmfoPYdGzSxTyMnQZ9djWI6Mt0s8G2AC6F2zZO+aYmXW631IKmKSn43KVJGgcSpNzVvCgn+ fp/zC3wQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzH-00000002Onx-2UXh; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D2C8B308CBF; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.498161564@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:07 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 11/14] llvm: kCFI pointer stuff References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Quick hack to extend the Clang-kCFI function meta-data (u32 hash) with a u8 bitmask of pointer arguments. This should really be under a new compiler flag, dependent on both x86_64 and kCFI. Per the comment, the bitmask represents the register based arguments as the first 6 bits and bit 6 is used to cover all stack based arguments. The high bit is used for invalid values. The purpose is to put a store dependency on the set registers, thereby blocking speculation paths that would otherwise expoit their value. Note1: This implementation simply sets the bit for any pointer type. A better implementation would only set the bit for any argument that is dereferenced in the function body. This better implementation would also capture things like: void foo(unsigned long addr, void *args) { u32 t =3D *(u32 *)addr; bar(t, args); } Which, in contrast to the implementation below, would set bit0 while leaving bit1 unset -- the exact opposite of this implementation. Notably, addr *is* dereferenced, even though it is not a pointer on entry, while args is a pointer, but is not derefereced but passed on to bar -- if bar uses it, it gets to deal with it. Note2: Do we want to make this a u32 to keep room for all registers? AFAICT the current use is only concerned with the argument registers and those are limited to 6 for the C ABI, but custom (assembly) functions could use things outside of that. --- diff --git a/llvm/lib/Target/X86/X86AsmPrinter.cpp b/llvm/lib/Target/X86/X8= 6AsmPrinter.cpp index 73c745062096..42dcbc40ab4b 100644 --- a/llvm/lib/Target/X86/X86AsmPrinter.cpp +++ b/llvm/lib/Target/X86/X86AsmPrinter.cpp @@ -143,11 +143,28 @@ void X86AsmPrinter::EmitKCFITypePadding(const Machine= Function &MF, // one. Otherwise, just pad with nops. The X86::MOV32ri instruction emit= ted // in X86AsmPrinter::emitKCFITypeId is 5 bytes long. if (HasType) - PrefixBytes +=3D 5; + PrefixBytes +=3D 7; =20 emitNops(offsetToAlignment(PrefixBytes, MF.getAlignment())); } =20 +static uint8_t getKCFIPointerArgs(const Function &F) +{ + uint8_t val =3D 0; + + if (F.isVarArg()) + return 0x7f; + + for (int i =3D 0; i < F.arg_size() ; i++) { + Argument *A =3D F.getArg(i); + Type *T =3D A->getType(); + if (T->getTypeID() =3D=3D Type::PointerTyID) + val |=3D 1 << std::min(i, 6); + } + + return val; +} + /// emitKCFITypeId - Emit the KCFI type information in architecture specif= ic /// format. void X86AsmPrinter::emitKCFITypeId(const MachineFunction &MF) { @@ -183,6 +200,26 @@ void X86AsmPrinter::emitKCFITypeId(const MachineFuncti= on &MF) { .addReg(X86::EAX) .addImm(MaskKCFIType(Type->getZExtValue()))); =20 + // Extend the kCFI meta-data with a byte that has a bit set for each arg= ument + // register that contains a pointer. Specifically for x86_64, which has 6 + // argument registers: + // + // bit0 - rdi + // bit1 - rsi + // bit2 - rdx + // bit3 - rcx + // bit4 - r8 + // bit5 - r9 + // + // bit6 will denote any pointer on stack (%rsp), and all 7 bits set will + // indicate vararg or any other 'unknown' configuration. Leaving bit7 for + // error states. + // + // XXX: should be conditional on some new x86_64 specific 'bhi' argument. + EmitAndCountInstruction(MCInstBuilder(X86::MOV8ri) + .addReg(X86::AL) + .addImm(getKCFIPointerArgs(F))); + if (MAI->hasDotTypeDotSizeDirective()) { MCSymbol *EndSym =3D OutContext.createTempSymbol("cfi_func_end"); OutStreamer->emitLabel(EndSym); diff --git a/llvm/lib/Target/X86/X86MCInstLower.cpp b/llvm/lib/Target/X86/X= 86MCInstLower.cpp index cbb012161524..c0776ef78153 100644 --- a/llvm/lib/Target/X86/X86MCInstLower.cpp +++ b/llvm/lib/Target/X86/X86MCInstLower.cpp @@ -897,7 +897,7 @@ void X86AsmPrinter::LowerKCFI_CHECK(const MachineInstr = &MI) { .addReg(AddrReg) .addImm(1) .addReg(X86::NoRegister) - .addImm(-(PrefixNops + 4)) + .addImm(-(PrefixNops + 6)) .addReg(X86::NoRegister)); =20 MCSymbol *Pass =3D OutContext.createTempSymbol(); From nobody Thu Nov 28 19:52:08 2024 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52C0A138490 for ; Fri, 27 Sep 2024 19:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466654; cv=none; b=h2ESTMJy5F1QOSBiA63vLgMkNGuCIbxETjzR65KDI7yY80Eys/9361GFdGSmYD7JcS8/emeGzOI9WRx9mzPBIDcKoX6HuuuHAf5SWpGJzSNy9l6DCKq7VS/eukQrFfK7Z4EFAhAJbOL16lMvGo9wRsfWEKk4aOfC7E8h7LeHvvw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466654; c=relaxed/simple; bh=enbd7fMfVf59B/v+M5Jn0PhJrPlcwc0U5V3LFFwjQZg=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=ScgKS85QC/GV+S2bkPl3m79T/NyO45fFeWf0YMg6Oklk5p1ds0dcMr1uSJK6gW1KLdqpnUNf9Ns0jf4T92/dpJc/Mmw/6Es4NTtaAz7NvFQVdoq7K4OmFNElmDsKuxkL9LwztoGmDFymWqGnt9nVuCVZEqBJGQbTm2laYu+dSw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=TcMZ1TCn; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="TcMZ1TCn" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=EqwVWQ0lEn5NKERRe1JMKwqsIxUttvdNmtQApOwpG7k=; b=TcMZ1TCn+kRP4tIheHf7Z7YQSq /ZwViQM8eyow9cLFJV+3UdN8IIZMD1JKDruD/PtHps4LSzIQ0+cOVsvWZh//9KK1KIhYlv+0VfJZG lJAtPNatbwG2/XW5NWlEABnTlwFO0AkgU4Adm6aLfAa+O1dmr0f9zhTfmymg2C7a0YhU+vgZDOhLK KZHQdjfV14edG+LVi0NIBvqKIg1JrnWCNKPdM9ehscK/tSWgWU95qlD8LHpG+DbFUKzngVYHc3ybR vlCkJi7HW9nqZp2Hr9wkN+BbmGU5F4/3AVUzaIGyE0gTFtIi9BxSGw53JRnrQ6f6cBjSThsRCbUqm 0mlg9h/A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzG-0000000AVcx-3DT6; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D6562308CC5; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.603504509@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:08 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 12/14] x86: Hacks for hacked up llvm References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" XXX do this nicely once the llvm hacks are done nice Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/Kconfig | 8 ++++---- arch/x86/include/asm/linkage.h | 2 ++ 2 files changed, 6 insertions(+), 4 deletions(-) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2444,10 +2444,10 @@ config CC_HAS_ENTRY_PADDING =20 config FUNCTION_PADDING_CFI int - default 59 if FUNCTION_ALIGNMENT_64B - default 27 if FUNCTION_ALIGNMENT_32B - default 11 if FUNCTION_ALIGNMENT_16B - default 3 if FUNCTION_ALIGNMENT_8B + default 57 if FUNCTION_ALIGNMENT_64B + default 25 if FUNCTION_ALIGNMENT_32B + default 9 if FUNCTION_ALIGNMENT_16B + default 1 if FUNCTION_ALIGNMENT_8B default 0 =20 # Basically: FUNCTION_ALIGNMENT - 5*CFI_CLANG --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -102,6 +102,8 @@ CFI_PRE_PADDING \ .byte 0xb8 ASM_NL \ .long __kcfi_typeid_##name ASM_NL \ + .byte 0xb0 ASM_NL \ + .byte 0x7f ASM_NL \ CFI_POST_PADDING \ SYM_FUNC_END(__cfi_##name) From nobody Thu Nov 28 19:52:08 2024 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52C941714B8 for ; Fri, 27 Sep 2024 19:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; cv=none; b=G0D19116BuqOsu7cI/EPS1K6EtM2LooZLfIECRX1s3K46hfHhB9TE5n+zzoDQPhWsFeU3rX4hdYvJ8OdIk4fowVawFj/Ji2R4b8C06NAZdq/kLaI0mx/46EVw8hd5xBlolAlOKMl9dpeR0rRL0o85xJy79/xLi9pNSgX46+iwYM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; c=relaxed/simple; bh=Qfj5skINFFABCKIJM2jyxqp/bgXcco5fHnAGbxuyFzY=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=jvAbayifBWjBKq8IN1fMiIv9dfQaq361PNoBVCW7Tf+iqvesOqHhuRg9CqcCz3zEZyMGV6AFcWygrWlrxiLTzX6M3EYkA81bk9WhpCabLiHn0mjfxYEQLhvPi8Fa3ZXeZUWPgSkjbBES769ZkyzQd2BeVSxk/NpBeX39tJTRzFM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=hk2Tlov0; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="hk2Tlov0" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=DYk/ii3NJ1TG/DXWXke3E5hmJXxv8Z/BRxQS68QUbZ4=; b=hk2Tlov0XA7JH9ceId///4cjZw 7s3wL/Hto5DU5WKOg6gtGeFw5+GoRDGqnTluE8PFO/abntRxPWfnGvtFINroI+nl3+WKXt3xmNUwY aDj7GV+JrzzXxGsSJ6+nuCfNg27geD0t2BbID4Sm6s7v4NXNYJlidXNpsV+FMP2l+V5wk7Bkfy9tW yuHo9fblWkds/FV6PM1p4bcG6E5aGBwoORYh1OKnMHMEn7Rc8VH5qwQ4HUs7zM85Ecq3GK/39Juab 1lzFA7ThNf2VPRLY0GcQsHBtVQ4z/dymWjLPMlDxAWOoz4Ns54nve3o6yD4aPk+a+bG2KhjzhCLUE RfPz7J9Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzG-0000000AVcy-3Ey6; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DA676308CC6; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.707462984@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:09 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 13/14] x86: BHI stubs References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mostly generated combinatorical stubs used to poison function argument pointers. Notably, since this targets eIBRS parts which do not suffer from retbleed, use normal return instructions to save some space. In total: 6c1 + 6c2 + 6c3 + 6c4 + 1 =3D 6 + 15 + 20 + 15 + 1 =3D 57 stubs. Note: Scott said only 0.6% of the kernel functions take 5 or more pointer arguments, if any of those turns out to be performance critical, we can add more stubs. Note: the nested for loops are horrid, should be fixed. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cfi.h | 11=20 arch/x86/kernel/alternative.c | 94 ++++++ arch/x86/lib/Makefile | 1=20 arch/x86/lib/bhi.S | 602 +++++++++++++++++++++++++++++++++++++= +++++ 4 files changed, 708 insertions(+) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -101,6 +101,17 @@ enum cfi_mode { =20 extern enum cfi_mode cfi_mode; =20 +typedef u8 bhi_thunk_8[8]; +typedef u8 bhi_thunk_16[16]; +typedef u8 bhi_thunk_32[32]; + +extern bhi_thunk_8 __bhi_args_6c1[]; +extern bhi_thunk_16 __bhi_args_6c2[]; +extern bhi_thunk_16 __bhi_args_6c3[]; +extern bhi_thunk_32 __bhi_args_6c4[]; + +extern u8 __bhi_args_all[]; + struct pt_regs; =20 #ifdef CONFIG_CFI_CLANG --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1039,10 +1039,104 @@ u32 cfi_get_func_hash(void *func) =20 return hash; } + #endif =20 #ifdef CONFIG_FINEIBT =20 +static void *bhi_args_1(u8 args, void *addr) +{ + u8 bytes[5]; + + for (int i =3D 0; i < 6; i++) { + if (args !=3D BIT(i)) + continue; + + bytes[0] =3D 0x2e; + memcpy(&bytes[1], &__bhi_args_6c1[i], 4); + + text_poke_early(addr, bytes, 5); + + return NULL; + } + + return __bhi_args_all; +} + +static void *bhi_args_2(u8 args, void *addr) +{ + int x =3D 0; + + for (int i =3D 0; i < 6; i++) { + for (int j =3D i + 1; j < 6; j++) { + if (args =3D=3D (BIT(i) | BIT(j))) + return &__bhi_args_6c2[x]; + x++; + } + } + + return __bhi_args_all; +} + +static void *bhi_args_3(u8 args, void *addr) +{ + int x =3D 0; + + for (int i =3D 0; i < 6; i++) { + for (int j =3D i + 1; j < 6; j++) { + for (int k =3D j + 1; k < 6; k++) { + if (args =3D=3D (BIT(i) | BIT(j) | BIT(k))) + return &__bhi_args_6c3[x]; + x++; + } + } + } + + return __bhi_args_all; +} + +static void *bhi_args_4(u8 args, void *addr) +{ + int x =3D 0; + + for (int i =3D 0; i < 6; i++) { + for (int j =3D i + 1; j < 6; j++) { + for (int k =3D j + 1; k < 6; k++) { + for (int l =3D k + 1; l < 6; l++) { + if (args =3D=3D (BIT(i) | BIT(j) | BIT(k) | BIT(l))) + return &__bhi_args_6c4[x]; + x++; + } + } + } + } + + return __bhi_args_all; +} + +static void bhi_args(u8 args, void *addr) +{ + void *dest =3D __bhi_args_all; + + if (WARN_ON_ONCE(!args)) + return; + + switch(hweight8(args)) { + case 1: if (bhi_args_1(args, addr) =3D=3D dest) + break; + return; + + case 2: dest =3D bhi_args_2(args, addr); break; + case 3: dest =3D bhi_args_3(args, addr); break; + case 4: dest =3D bhi_args_4(args, addr); break; + + default: + break; + } + + text_poke_early(addr, text_gen_insn(CALL_INSN_OPCODE, addr, dest), 5); +} + static bool cfi_rand __ro_after_init =3D true; static u32 cfi_seed __ro_after_init; =20 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -60,4 +60,5 @@ endif lib-y +=3D memmove_64.o memset_64.o lib-y +=3D copy_user_64.o copy_user_uncached_64.o lib-y +=3D cmpxchg16b_emu.o + lib-y +=3D bhi.o endif --- /dev/null +++ b/arch/x86/lib/bhi.S @@ -0,0 +1,602 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include + +/* + * At the function start, launder function arguments that are a pointer th= rough + * CMOVcc, this will create a write dependency in the speculation flow. + * + * Notably, the CFI preambles calling these will have ZF set and r10 zero. + */ + +.pushsection .noinstr.text, "ax" + + .align 8 +SYM_CODE_START(__bhi_args_6c1) + ANNOTATE_NOENDBR + .align 8 +SYM_INNER_LABEL(__bhi_args_0, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 8 +SYM_INNER_LABEL(__bhi_args_1, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 8 +SYM_INNER_LABEL(__bhi_args_2, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 8 +SYM_INNER_LABEL(__bhi_args_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 8 +SYM_INNER_LABEL(__bhi_args_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 8 +SYM_INNER_LABEL(__bhi_args_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + +SYM_CODE_END(__bhi_args_6c1) + + + .align 16 +SYM_CODE_START(__bhi_args_6c2) + ANNOTATE_NOENDBR + .align 16 +SYM_INNER_LABEL(__bhi_args_0_1, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_2, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rdx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_2, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rdx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_2_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_2_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_2_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_3_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_3_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rcx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + +SYM_CODE_END(__bhi_args_6c2) + + + .align 16 +SYM_CODE_START(__bhi_args_6c3) + ANNOTATE_NOENDBR + .align 16 +SYM_INNER_LABEL(__bhi_args_0_1_2, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_1_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_1_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_1_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_2_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rdx + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_2_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rdx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_2_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rdx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_3_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_3_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rcx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_0_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_2_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_2_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_2_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_3_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_3_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rcx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_1_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_2_3_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_2_3_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_2_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 16 +SYM_INNER_LABEL(__bhi_args_3_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + +SYM_CODE_END(__bhi_args_6c3) + + + .align 32 +SYM_CODE_START(__bhi_args_6c4) + ANNOTATE_NOENDBR + .align 32 +SYM_INNER_LABEL(__bhi_args_0_1_2_3, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_1_2_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_1_2_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_1_3_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_1_3_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rcx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_1_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_2_3_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_2_3_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_2_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rdx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_0_3_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_1_2_3_4, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_1_2_3_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_1_2_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_1_3_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rsi + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_2_3_4_5, SYM_L_LOCAL) + UNWIND_HINT_FUNC + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + +SYM_CODE_END(__bhi_args_6c4) + +SYM_CODE_START(__bhi_args_all) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + cmovne %r10, %rsp + ANNOTATE_UNRET_SAFE + ret + int3 +SYM_CODE_END(__bhi_args_all) + +.popsection From nobody Thu Nov 28 19:52:08 2024 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52BC046B8 for ; Fri, 27 Sep 2024 19:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; cv=none; b=R9BEhl34Pkr+nvOdcJKAvQ5snH/CyL+yGib0SCaO9gBjqoJ8cXWVfIz+ZBA9zbneycm5Nr8A+OmIGMEALE3SEMs6ZmUCbF9VgjJapgQH7fTe2v0M9dNjxwLSdh9EeiReUfsEpUJhRSBbushbvBe6pftGp8b8+Hss3ZExM5izkJM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727466655; c=relaxed/simple; bh=8sZxPuRbomSMJfL/W9LjjPUHMhM2X6DLtr5iTM4vWh8=; h=Message-Id:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=vBQHY4lygPaVOeGKgSrRaqWj7LGgx2+EvxBUlqkyckHoPXhmSIt++LUd4FHkgs4MfQzWwtHLiIBOU8WHGaYa2tso7XIDhV4ibC9Hm4e/L0uv2zSQziVKG6MVnVOaeZ26GAi7uKSmoFcTUhJ+x4Vka0OrJm9FBNGblYbY1rv/F18= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=boCoGFWl; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="boCoGFWl" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=RpIFeMYhriEM2/4B/8JXzCFYxG/jfIBuyxGPfy1TFiA=; b=boCoGFWllgiin496ZhslbBChPl YvAxQ/SKtVk9dyQMFw47OnySwaaD9/YqEF5h0fmOtsyMUv5D7HiEon1nklas9pfEjo9eira9ewJIk tmZuUR9DCfeycS79BQtqILg4SIxr9Rp5UFXjgjo7orxf3quRbrCAUNRd+iK72dwQgu9R/Ld3/TLc4 BEdQkpOQZRJzOduL20es0MW0+JDfCOhlkB9h/+lRfIC/C11EIQ+2qQ6x+dxf0dwr/jnnd8QVmEzse MEdE+XOXikBCeA6Cy83+niMHhZOb567ws8cgVAEdFqe0sB70VGjFjZvnj3/eJzYg0SBIpoHOsmC7l rYRig5ow==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1suGzH-0000000AVcz-04jp; Fri, 27 Sep 2024 19:50:47 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DE849308CC7; Fri, 27 Sep 2024 21:50:44 +0200 (CEST) Message-Id: <20240927194925.808912874@infradead.org> User-Agent: quilt/0.65 Date: Fri, 27 Sep 2024 21:49:10 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com Subject: [PATCH 14/14] x86/fineibt: Add FineIBT+BHI mitigation References: <20240927194856.096003183@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Due to FineIBT weakness, add an additional mitigation for BHI. Use the 5 bytes of the nop at -1 and the 4 byte poison to squirrel in a BHI mitigation. Relies on clang-cfi to emit an additional piece of magic in the kCFI pre-amble, identifying which function arguments are pointers. An additional u8 (next to the existing u32) is emitted like: movl 0x12345678, %eax // CFI hash movb 0x12, %al // CFI args This u8 is a bitmask, where BIT(n) indicates the n'th argument is a pointer, notably the 6 possible argument registers are: rdi, rsi, rdx, rcx, r8 and r9 Single bit can be inlined, while 2-4 bits have combinatoric stubs with the required magic in. Anything more will fall back to __bhi_args_all which additionally poisons %rsp for good measure, in case things overflowed to the stack. FineIBT+ FineIBT+BHI __cfi_foo: __cfi_foo: endbr64 endbr64 subl $0x12345678, %r10d subl $0x12345678, %r10d jz foo+4 jz +2 ud2 ud2 nop call __bhi_args_foo foo: foo+4: ud1 0x0(%eax), %eax ... ... Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cfi.h | 1=20 arch/x86/kernel/alternative.c | 82 +++++++++++++++++++++++++++++++++++++= +---- arch/x86/net/bpf_jit_comp.c | 16 ++++++-- tools/objtool/check.c | 16 ++++---- 4 files changed, 98 insertions(+), 17 deletions(-) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -97,6 +97,7 @@ enum cfi_mode { CFI_OFF, /* Taditional / IBT depending on .config */ CFI_KCFI, /* Optionally CALL_PADDING, IBT, RETPOLINE */ CFI_FINEIBT, /* see arch/x86/kernel/alternative.c */ + CFI_FINEIBT_BHI, }; =20 extern enum cfi_mode cfi_mode; --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -932,7 +932,31 @@ __noendbr bool is_endbr(u32 *val) if (get_kernel_nofault(endbr, val)) return false; =20 - return __is_endbr(endbr); + if (__is_endbr(endbr)) + return true; + +#if defined(CONFIG_FINEIBT) && defined(CONFIG_X86_KERNEL_IBT_PLUS) + if (cfi_mode !=3D CFI_FINEIBT_BHI) + return false; + + /* + * The BHI clobbers 'replace' the ENDBR poison, but dynamic call + * patching (static_call, kprobes, etc..) still need to be able + * to identify and skip the foo()+0 'endbr'. + */ + + /* REX CMOVNE, see bhi_args_1() */ + if ((endbr & 0xc2fffff9) =3D=3D 0xc2450f49) + return true; + + /* CALL __bhi_args_* */ + void *dest =3D (void *)val + 4 + (s32)endbr; + if (dest >=3D (void *)__bhi_args_6c1 && + dest <=3D (void *)__bhi_args_all) + return true; +#endif + + return false; } =20 static void poison_cfi(void *addr); @@ -1190,6 +1214,8 @@ static __init int cfi_parse_cmdline(char cfi_mode =3D CFI_KCFI; } else if (!strcmp(str, "fineibt")) { cfi_mode =3D CFI_FINEIBT; + } else if (IS_ENABLED(CONFIG_X86_KERNEL_IBT_PLUS) && !strcmp(str, "finei= bt+bhi")) { + cfi_mode =3D CFI_FINEIBT_BHI; } else if (!strcmp(str, "norand")) { cfi_rand =3D false; } else { @@ -1208,10 +1234,9 @@ early_param("cfi", cfi_parse_cmdline); * * __cfi_\func: __cfi_\func: * movl $0x12345678,%eax // 5 endbr64 // 4 - * nop subl $0x12345678,%r10d // 7 + * movb $0x12,%al // 2 subl $0x12345678,%r10d // 7 * nop jz 1f // 2 * nop ud2 // 2 - * nop 1: nop // 1 * nop * nop * nop @@ -1279,6 +1304,17 @@ static u32 decode_preamble_hash(void *ad return 0; /* invalid hash value */ } =20 +static u8 decode_preamble_args(void *addr) +{ + u8 *p =3D addr; + + /* b0 12 movb $0x12, %al */ + if (p[5] =3D=3D 0xb0) + return p[6]; + + return 0xff; /* invalid args */ +} + static u32 decode_caller_hash(void *addr) { u8 *p =3D addr; @@ -1371,6 +1407,7 @@ static int cfi_rewrite_preamble(s32 *sta for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; u32 hash; + u8 args; =20 /* * When the function doesn't start with ENDBR the compiler will @@ -1385,11 +1422,25 @@ static int cfi_rewrite_preamble(s32 *sta addr, addr, 5, addr)) return -EINVAL; =20 + args =3D decode_preamble_args(addr); + text_poke_early(addr, fineibt_preamble_start, fineibt_preamble_size); WARN_ON(*(u32 *)(addr + fineibt_preamble_hash) !=3D 0x12345678); text_poke_early(addr + fineibt_preamble_hash, &hash, 4); =20 *(u8 *)(addr + fineibt_preamble_jccd8) +=3D 4; + + if (cfi_mode !=3D CFI_FINEIBT_BHI) + continue; + + WARN_ONCE(args =3D=3D 0xff, "no CFI args found at %pS %px %*ph\n", + addr, addr, 7, addr); + + /* + * Stash the ARGS byte in the NOP at __cfi_foo+15, see + * cfi_rewrite_endbr(). + */ + *(u8 *)(addr + fineibt_preamble_size - 1) =3D args; } =20 return 0; @@ -1401,11 +1452,26 @@ static void cfi_rewrite_endbr(s32 *start =20 for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; + u8 args; =20 if (!is_endbr(addr + 16)) continue; =20 - poison_endbr(addr + 16); + if (cfi_mode !=3D CFI_FINEIBT_BHI) { + poison_endbr(addr + 16); + continue; + } + + /* recover the args byte */ + args =3D *(u8 *)(addr + fineibt_preamble_size - 1); + *(u8 *)(addr + fineibt_preamble_size - 1) =3D BYTES_NOP1; + if (args) { + /* only skip the UD2 */ + *(u8 *)(addr + fineibt_preamble_jccd8) =3D 2; + + /* write BHI clobber in the 5 bytes that hold: nop + poison */ + bhi_args(args, addr + fineibt_preamble_size - 1); + } } } =20 @@ -1506,6 +1572,7 @@ static void __apply_fineibt(s32 *start_r return; =20 case CFI_FINEIBT: + case CFI_FINEIBT_BHI: /* place the FineIBT preamble at func()-16 */ ret =3D cfi_rewrite_preamble(start_cfi, end_cfi); if (ret) @@ -1519,8 +1586,10 @@ static void __apply_fineibt(s32 *start_r /* now that nobody targets func()+0, remove ENDBR there */ cfi_rewrite_endbr(start_cfi, end_cfi); =20 - if (builtin) - pr_info("Using FineIBT CFI\n"); + if (builtin) { + pr_info("Using FineIBT%s CFI\n", + cfi_mode =3D=3D CFI_FINEIBT_BHI ? "+BHI" : ""); + } return; =20 default: @@ -1548,6 +1617,7 @@ static void poison_cfi(void *addr) */ switch (cfi_mode) { case CFI_FINEIBT: + case CFI_FINEIBT_BHI: /* * FineIBT prefix should start with an ENDBR. */ --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -401,10 +401,17 @@ static void emit_fineibt(u8 **pprog, u32 =20 EMIT_ENDBR(); EMIT3_off32(0x41, 0x81, 0xea, hash); /* subl $hash, %r10d */ - EMIT2(0x74, 0x07); /* jz.d8 +7 */ - EMIT2(0x0f, 0x0b); /* ud2 */ - EMIT1(0x90); /* nop */ - EMIT_ENDBR_POISON(); + if (cfi_mode =3D=3D CFI_FINEIBT_BHI) { + EMIT2(0x74, 0x02); /* jz.d8 +2 */ + EMIT2(0x0f, 0x0b); /* ud2 */ + EMIT1(0x2e); /* cs */ + EMIT4(0x49, 0x0f, 0x45, 0xfa); /* cmovne %r10, %rdi */ + } else { + EMIT2(0x74, 0x07); /* jz.d8 +7 */ + EMIT2(0x0f, 0x0b); /* ud2 */ + EMIT1(0x90); /* nop */ + EMIT_ENDBR_POISON(); + } =20 *pprog =3D prog; } @@ -438,6 +445,7 @@ static void emit_cfi(u8 **pprog, u32 has =20 switch (cfi_mode) { case CFI_FINEIBT: + case CFI_FINEIBT_BHI: emit_fineibt(&prog, hash); break; =20 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -4397,11 +4397,9 @@ static int validate_ibt_insn(struct objt continue; =20 off =3D reloc->sym->offset; - if (reloc_type(reloc) =3D=3D R_X86_64_PC32 || - reloc_type(reloc) =3D=3D R_X86_64_PLT32) - off +=3D arch_dest_reloc_offset(reloc_addend(reloc)); - else - off +=3D reloc_addend(reloc); + off +=3D reloc_addend(reloc); + if (arch_pc_relative_reloc(reloc)) + off =3D arch_dest_reloc_offset(off); =20 dest =3D find_insn(file, reloc->sym->sec, off); if (!dest) @@ -4456,10 +4454,14 @@ static int validate_ibt_insn(struct objt static int validate_ibt_data_reloc(struct objtool_file *file, struct reloc *reloc) { + long offset =3D reloc->sym->offset; struct instruction *dest; =20 - dest =3D find_insn(file, reloc->sym->sec, - reloc->sym->offset + reloc_addend(reloc)); + offset +=3D reloc_addend(reloc); + if (reloc_type(reloc) =3D=3D R_X86_64_PLT32) // the fuck ?! + offset =3D arch_dest_reloc_offset(offset); + + dest =3D find_insn(file, reloc->sym->sec, offset); if (!dest) return 0;