From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E850A1E5B67 for ; Fri, 7 Feb 2025 12:28:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931315; cv=none; b=SvKFIG1X3GDSeH5dueptj2SAuobrkCr2PRSbR6M78NmcDQxCrcmYtAx6ZbK7BUvVzj/66zYzchvjLNyJs64RB/NLmfdvLEjrxuatwk37bIziarpyaAS/zmOF2UjQKJP5pkF8chaRD40F/3xAsPGV21UDAr4cxqFrKLJBkP9sBxM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931315; c=relaxed/simple; bh=qozU28y69mqm1H5QpZwiPTDKHpkC3GO5O5/ETqP5OLk=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=MKa+7gukiApipW+1mfsLOieAh+rpwBiBSwc173O6JPCVLBQX07eXV80YYzWz8W5hkvQmUDSOWe+5eibM/4wIahGBuvUeREbgJAog/tNvHBXoUXUp2SbcB9xEZn2iqM5vu68vP6Kj3ydZCQRrzPMJ8Gf3ZbGUIgoO6XiZ8TxEY2U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=Gd2uaoAd; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Gd2uaoAd" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=pTyW3xOyUBNtVPXAs4am3thZhwfMayjzqs4QeGMPqnE=; b=Gd2uaoAdm/r9gVuJkJmGCae6S/ H1lPR6Uo2IJTXtzn6DQX/RZMU9DsOpHnpd+fetzAx0zVVoAr4xcOdHPLtuWiEuhDi4R4MvIBMecmt jGGipA34kSN7LOGgQBXzVE5md+IPOyTgwgrobEAuz/J5uvQ0qwGpg7p68K79b2wJZB7RtH2s5Su9Q VbSfzDfF8kd0HEk9Jy3dUEtc7Ikj8xqGZ7jTN8K3JsmTPs/qbADTZTR9gf0fujB0S+zB+TaVXu6sY cgB03oOeSfYf3LkD0K1tKsf/86d7x/muevVcDArLg32/pHkpotpD8EuNUr1eEgpaOoi5DkK4DjTBU FtRMR95g==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT7-0000000HAeT-2m5y; Fri, 07 Feb 2025 12:28:30 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 256193002F0; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.068435759@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:30 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 01/11] objtool: Move dodgy linker warn to verbose References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The lld.ld borkage is fixed in the latest llvm release (?) but will not be backported, meaning we're stuck with broken linker for a fair while. Lets not spam all clang build logs and move warning to verbose. Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- tools/objtool/check.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2264,7 +2264,7 @@ static int read_annotate(struct objtool_ =20 if (sec->sh.sh_entsize !=3D 8) { static bool warned =3D false; - if (!warned) { + if (!warned && opts.verbose) { WARN("%s: dodgy linker, sh_entsize !=3D 8", sec->name); warned =3D true; } From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 155D61F78F7 for ; Fri, 7 Feb 2025 12:28:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931320; cv=none; b=qTPgsWf7daNdTmrO0CP0DO+5sV1W6ODrNAk5K4oXzh+vMPb8UjyLVImIg2Wk623y27pUx2FEHAjN+DjB4Y03mT7L0lZDowQu4arVKMJd5afjfySrtW73dcGwr2LfLSAcZQHQXwq5/5bugx76gzA6yxFKeopmqGpN4+mUqSfPoLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931320; c=relaxed/simple; bh=kztONzt9qazUS7blIa55eCBrBZ9fsT+kQft6opbqNvA=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=PP/tcPNF9xTlJAXnhy/cgWAfi+rxBe3N6fzZG/qWWiPDGlRnpIdUuLpQZw8gvz8xedVDr0kGEXmWtUxUQ42OqQzfzWNydervmIj79vEWIJAcDKW6Jc9i5Jr0HLmjPYOobBu1wX9a+Rjcwew5UDkFOwqMz4PWVS+wOa4rjZBf/JM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=OSd7oOIu; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OSd7oOIu" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=hqPWks6/rGav+fO54QPGE1iljQwGuQ9KKEAve1G8rjE=; b=OSd7oOIuGMefIB7bluCv+mWf4E WzwROg8IdAw581ZTQz4FB3VxwpeShz3Z2683DEuOOVUWQ/tBvLScc0X+ss4tWNcuF/JMPuQz3rmsg wF6nWT862ZkZyPf+Vq4li1dVsfktEfGoAHhW2EEqNAgsdbgKnvuZr72cGRpm5Ov7wbZzIStOmuI1J J9LQ48LQGkE1QOcmyP26biK3fXYrxtrguI95EdiDvHeHcvs7vcCFmJ+AbQPFv8qjSCO+UoJfOg84x Fxf0gU8/FT9be/RhVck2gZa+86GZHnkXu1yUwWm27BU9JuXNBEQTRKMfN7+ZZu2pqATFYQqN/KgB4 wf2JUIdw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT7-0000000HAeU-2eH8; Fri, 07 Feb 2025 12:28:33 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 27C36300642; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.181367417@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:31 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, Alexei Starovoitov , Andrii Nakryiko Subject: [PATCH 02/11] x86/ibt: Clean up is_endbr() References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pretty much every caller of is_endbr() actually wants to test something at = an address and ends up doing get_kernel_nofault(). Fold the lot into a more convenient helper. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Alexei Starovoitov Acked-by: Andrii Nakryiko Acked-by: Masami Hiramatsu (Google) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/events/core.c | 2 +- arch/x86/include/asm/ftrace.h | 16 ++-------------- arch/x86/include/asm/ibt.h | 5 +++-- arch/x86/kernel/alternative.c | 20 ++++++++++++++------ arch/x86/kernel/kprobes/core.c | 11 +---------- arch/x86/net/bpf_jit_comp.c | 4 ++-- kernel/trace/bpf_trace.c | 21 ++++----------------- 7 files changed, 27 insertions(+), 52 deletions(-) --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -2857,7 +2857,7 @@ static bool is_uprobe_at_func_entry(stru return true; =20 /* endbr64 (64-bit only) */ - if (user_64bit_mode(regs) && is_endbr(*(u32 *)auprobe->insn)) + if (user_64bit_mode(regs) && is_endbr((u32 *)auprobe->insn)) return true; =20 return false; --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -36,21 +36,9 @@ static inline unsigned long ftrace_call_ =20 static inline unsigned long arch_ftrace_get_symaddr(unsigned long fentry_i= p) { -#ifdef CONFIG_X86_KERNEL_IBT - u32 instr; - - /* We want to be extra safe in case entry ip is on the page edge, - * but otherwise we need to avoid get_kernel_nofault()'s overhead. - */ - if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) { - if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE))) - return fentry_ip; - } else { - instr =3D *(u32 *)(fentry_ip - ENDBR_INSN_SIZE); - } - if (is_endbr(instr)) + if (is_endbr((void*)(fentry_ip - ENDBR_INSN_SIZE))) fentry_ip -=3D ENDBR_INSN_SIZE; -#endif + return fentry_ip; } #define ftrace_get_symaddr(fentry_ip) arch_ftrace_get_symaddr(fentry_ip) --- a/arch/x86/include/asm/ibt.h +++ b/arch/x86/include/asm/ibt.h @@ -65,7 +65,7 @@ static __always_inline __attribute_const return 0x001f0f66; /* osp nopl (%rax) */ } =20 -static inline bool is_endbr(u32 val) +static inline bool __is_endbr(u32 val) { if (val =3D=3D gen_endbr_poison()) return true; @@ -74,6 +74,7 @@ static inline bool is_endbr(u32 val) return val =3D=3D gen_endbr(); } =20 +extern __noendbr bool is_endbr(u32 *val); extern __noendbr u64 ibt_save(bool disable); extern __noendbr void ibt_restore(u64 save); =20 @@ -98,7 +99,7 @@ extern __noendbr void ibt_restore(u64 sa =20 #define __noendbr =20 -static inline bool is_endbr(u32 val) { return false; } +static inline bool is_endbr(u32 *val) { return false; } =20 static inline u64 ibt_save(bool disable) { return 0; } static inline void ibt_restore(u64 save) { } --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -852,16 +852,24 @@ void __init_or_module noinline apply_ret =20 #ifdef CONFIG_X86_KERNEL_IBT =20 +__noendbr bool is_endbr(u32 *val) +{ + u32 endbr; + + __get_kernel_nofault(&endbr, val, u32, Efault); + return __is_endbr(endbr); + +Efault: + return false; +} + static void poison_cfi(void *addr); =20 static void __init_or_module poison_endbr(void *addr, bool warn) { - u32 endbr, poison =3D gen_endbr_poison(); - - if (WARN_ON_ONCE(get_kernel_nofault(endbr, addr))) - return; + u32 poison =3D gen_endbr_poison(); =20 - if (!is_endbr(endbr)) { + if (!is_endbr(addr)) { WARN_ON_ONCE(warn); return; } @@ -984,7 +992,7 @@ static u32 cfi_seed __ro_after_init; static u32 cfi_rehash(u32 hash) { hash ^=3D cfi_seed; - while (unlikely(is_endbr(hash) || is_endbr(-hash))) { + while (unlikely(__is_endbr(hash) || __is_endbr(-hash))) { bool lsb =3D hash & 1; hash >>=3D 1; if (lsb) --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -373,16 +373,7 @@ static bool can_probe(unsigned long padd kprobe_opcode_t *arch_adjust_kprobe_addr(unsigned long addr, unsigned long= offset, bool *on_func_entry) { - u32 insn; - - /* - * Since 'addr' is not guaranteed to be safe to access, use - * copy_from_kernel_nofault() to read the instruction: - */ - if (copy_from_kernel_nofault(&insn, (void *)addr, sizeof(u32))) - return NULL; - - if (is_endbr(insn)) { + if (is_endbr((u32 *)addr)) { *on_func_entry =3D !offset || offset =3D=3D 4; if (*on_func_entry) offset =3D 4; --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -641,7 +641,7 @@ int bpf_arch_text_poke(void *ip, enum bp * See emit_prologue(), for IBT builds the trampoline hook is preceded * with an ENDBR instruction. */ - if (is_endbr(*(u32 *)ip)) + if (is_endbr(ip)) ip +=3D ENDBR_INSN_SIZE; =20 return __bpf_arch_text_poke(ip, t, old_addr, new_addr); @@ -3036,7 +3036,7 @@ static int __arch_prepare_bpf_trampoline /* skip patched call instruction and point orig_call to actual * body of the kernel function. */ - if (is_endbr(*(u32 *)orig_call)) + if (is_endbr(orig_call)) orig_call +=3D ENDBR_INSN_SIZE; orig_call +=3D X86_PATCH_SIZE; } --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1038,27 +1038,14 @@ static const struct bpf_func_proto bpf_g .arg1_type =3D ARG_PTR_TO_CTX, }; =20 -#ifdef CONFIG_X86_KERNEL_IBT -static unsigned long get_entry_ip(unsigned long fentry_ip) +static inline unsigned long get_entry_ip(unsigned long fentry_ip) { - u32 instr; - - /* We want to be extra safe in case entry ip is on the page edge, - * but otherwise we need to avoid get_kernel_nofault()'s overhead. - */ - if ((fentry_ip & ~PAGE_MASK) < ENDBR_INSN_SIZE) { - if (get_kernel_nofault(instr, (u32 *)(fentry_ip - ENDBR_INSN_SIZE))) - return fentry_ip; - } else { - instr =3D *(u32 *)(fentry_ip - ENDBR_INSN_SIZE); - } - if (is_endbr(instr)) +#ifdef CONFIG_X86_KERNEL_IBT + if (is_endbr((void *)(fentry_ip - ENDBR_INSN_SIZE))) fentry_ip -=3D ENDBR_INSN_SIZE; +#endif return fentry_ip; } -#else -#define get_entry_ip(fentry_ip) fentry_ip -#endif =20 BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs) { From nobody Sun Feb 8 00:49:27 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 630E61DE2D7 for ; Fri, 7 Feb 2025 12:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931314; cv=none; b=GL9lPhZn8+G1lkzLp70ZUOiG+7X6AUdLkL6o0AuNa39RIJRqEE+HioOUyXkPrQ9VaImtmyy6jWxZoCpvqohqyAl+jHyrN93c6axnKrJfdCdFzhs82SbUO/CevIu63cLT9u9+/KRl+/6IPsu+M4L+rxAAc1+zQHsBEfrsQ0H2E3Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931314; c=relaxed/simple; bh=JBcZXkMbHaasTgMYbtpZTIxIdpc2x4ruCpuijKxhm7g=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=BtTfSyBk1YYq3SrAmhpKtaZmKqcZc3a/Rwg9XsupKwSnl3yvpxfXieJ2AgXVSnA+jgqd5W0OvVyskQSnTfQGIxPq6CoZ8kqdZdb86K+sz3/Z2WiQrJ04LgA27wfOceyPsgl9fHLnLo7uXWSUAk9VR9Wh7vjTr8+f31iFTEMZ9UM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=fb3aQVHn; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="fb3aQVHn" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ZUv5K/NDjIg1Db+z6YMSwmMOaXtAc4RYuDDWykJP/1A=; b=fb3aQVHnfYISsMPkNXTGWM3ZlS A9Os6rOW61SdozXfENBRgd6SaJJ4JndgenFbhFk9/cWG0tV+I3PNfTWSpTE0b73SCiRQqoHcB7bnI CSUede+7rMHo5Ip6nGycLvHz7TK3aD9FhcZV9yH60scebs0Cb68gekScMk4fxsI5jnFvExJH9Xpr6 6aoTFngEQhiS2oJMkwI19eN6PVO3g3lPDiUKqpbJh0LWkjsRNRbJVOteTl6mQQ/xUMC9JkVWsnh9g cyCtQA0PP2IeEDb2dKJ8ZZoi1HGrjHua3SUz5ual7jG+yQWzi8r1RggtIa+4C7Go2XEsEcfnrfP7d uv2TJM4w==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT8-00000007nBg-0L0Y; Fri, 07 Feb 2025 12:28:26 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 2B8783006C0; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.302679189@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:32 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 03/11] x86,kcfi: Fix EXPORT_SYMBOL vs kCFI References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The expectation is that all EXPORT'ed symbols are free to have their address taken and called indirectly. The majority of the assembly defined functions currently violate this expectation. Make then all use SYM_TYPED_FUNC_START() in order to emit the proper kCFI preamble. Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/crypto/camellia-aesni-avx-asm_64.S | 7 ++++--- arch/x86/crypto/camellia-aesni-avx2-asm_64.S | 1 + arch/x86/crypto/camellia-x86_64-asm_64.S | 9 +++++---- arch/x86/crypto/serpent-avx-x86_64-asm_64.S | 7 ++++--- arch/x86/crypto/twofish-x86_64-asm_64-3way.S | 5 +++-- arch/x86/crypto/twofish-x86_64-asm_64.S | 5 +++-- arch/x86/entry/entry.S | 3 ++- arch/x86/entry/entry_64.S | 5 +++-- arch/x86/lib/clear_page_64.S | 9 +++++---- arch/x86/lib/copy_page_64.S | 3 ++- arch/x86/lib/copy_user_64.S | 3 ++- arch/x86/lib/copy_user_uncached_64.S | 3 ++- arch/x86/lib/getuser.S | 17 +++++++++-------- arch/x86/lib/hweight.S | 5 +++-- arch/x86/lib/memmove_64.S | 3 ++- arch/x86/lib/memset_64.S | 3 ++- arch/x86/lib/msr-reg.S | 3 ++- arch/x86/lib/putuser.S | 17 +++++++++-------- 18 files changed, 63 insertions(+), 45 deletions(-) Index: linux-2.6/arch/x86/crypto/camellia-aesni-avx-asm_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/crypto/camellia-aesni-avx-asm_64.S +++ linux-2.6/arch/x86/crypto/camellia-aesni-avx-asm_64.S @@ -16,6 +16,7 @@ */ =20 #include +#include #include =20 #define CAMELLIA_TABLE_BYTE_LEN 272 @@ -882,7 +883,7 @@ SYM_FUNC_START_LOCAL(__camellia_dec_blk1 jmp .Ldec_max24; SYM_FUNC_END(__camellia_dec_blk16) =20 -SYM_FUNC_START(camellia_ecb_enc_16way) +SYM_TYPED_FUNC_START(camellia_ecb_enc_16way) /* input: * %rdi: ctx, CTX * %rsi: dst (16 blocks) @@ -907,7 +908,7 @@ SYM_FUNC_START(camellia_ecb_enc_16way) RET; SYM_FUNC_END(camellia_ecb_enc_16way) =20 -SYM_FUNC_START(camellia_ecb_dec_16way) +SYM_TYPED_FUNC_START(camellia_ecb_dec_16way) /* input: * %rdi: ctx, CTX * %rsi: dst (16 blocks) @@ -937,7 +938,7 @@ SYM_FUNC_START(camellia_ecb_dec_16way) RET; SYM_FUNC_END(camellia_ecb_dec_16way) =20 -SYM_FUNC_START(camellia_cbc_dec_16way) +SYM_TYPED_FUNC_START(camellia_cbc_dec_16way) /* input: * %rdi: ctx, CTX * %rsi: dst (16 blocks) Index: linux-2.6/arch/x86/crypto/camellia-aesni-avx2-asm_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/crypto/camellia-aesni-avx2-asm_64.S +++ linux-2.6/arch/x86/crypto/camellia-aesni-avx2-asm_64.S @@ -6,6 +6,7 @@ */ =20 #include +#include #include =20 #define CAMELLIA_TABLE_BYTE_LEN 272 Index: linux-2.6/arch/x86/crypto/camellia-x86_64-asm_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/crypto/camellia-x86_64-asm_64.S +++ linux-2.6/arch/x86/crypto/camellia-x86_64-asm_64.S @@ -6,6 +6,7 @@ */ =20 #include +#include =20 .file "camellia-x86_64-asm_64.S" .text @@ -177,7 +178,7 @@ bswapq RAB0; \ movq RAB0, 4*2(RIO); =20 -SYM_FUNC_START(__camellia_enc_blk) +SYM_TYPED_FUNC_START(__camellia_enc_blk) /* input: * %rdi: ctx, CTX * %rsi: dst @@ -224,7 +225,7 @@ SYM_FUNC_START(__camellia_enc_blk) RET; SYM_FUNC_END(__camellia_enc_blk) =20 -SYM_FUNC_START(camellia_dec_blk) +SYM_TYPED_FUNC_START(camellia_dec_blk) /* input: * %rdi: ctx, CTX * %rsi: dst @@ -411,7 +412,7 @@ SYM_FUNC_END(camellia_dec_blk) bswapq RAB1; \ movq RAB1, 12*2(RIO); =20 -SYM_FUNC_START(__camellia_enc_blk_2way) +SYM_TYPED_FUNC_START(__camellia_enc_blk_2way) /* input: * %rdi: ctx, CTX * %rsi: dst @@ -460,7 +461,7 @@ SYM_FUNC_START(__camellia_enc_blk_2way) RET; SYM_FUNC_END(__camellia_enc_blk_2way) =20 -SYM_FUNC_START(camellia_dec_blk_2way) +SYM_TYPED_FUNC_START(camellia_dec_blk_2way) /* input: * %rdi: ctx, CTX * %rsi: dst Index: linux-2.6/arch/x86/crypto/serpent-avx-x86_64-asm_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/crypto/serpent-avx-x86_64-asm_64.S +++ linux-2.6/arch/x86/crypto/serpent-avx-x86_64-asm_64.S @@ -9,6 +9,7 @@ */ =20 #include +#include #include #include "glue_helper-asm-avx.S" =20 @@ -656,7 +657,7 @@ SYM_FUNC_START_LOCAL(__serpent_dec_blk8_ RET; SYM_FUNC_END(__serpent_dec_blk8_avx) =20 -SYM_FUNC_START(serpent_ecb_enc_8way_avx) +SYM_TYPED_FUNC_START(serpent_ecb_enc_8way_avx) /* input: * %rdi: ctx, CTX * %rsi: dst @@ -674,7 +675,7 @@ SYM_FUNC_START(serpent_ecb_enc_8way_avx) RET; SYM_FUNC_END(serpent_ecb_enc_8way_avx) =20 -SYM_FUNC_START(serpent_ecb_dec_8way_avx) +SYM_TYPED_FUNC_START(serpent_ecb_dec_8way_avx) /* input: * %rdi: ctx, CTX * %rsi: dst @@ -692,7 +693,7 @@ SYM_FUNC_START(serpent_ecb_dec_8way_avx) RET; SYM_FUNC_END(serpent_ecb_dec_8way_avx) =20 -SYM_FUNC_START(serpent_cbc_dec_8way_avx) +SYM_TYPED_FUNC_START(serpent_cbc_dec_8way_avx) /* input: * %rdi: ctx, CTX * %rsi: dst Index: linux-2.6/arch/x86/crypto/twofish-x86_64-asm_64-3way.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/crypto/twofish-x86_64-asm_64-3way.S +++ linux-2.6/arch/x86/crypto/twofish-x86_64-asm_64-3way.S @@ -6,6 +6,7 @@ */ =20 #include +#include =20 .file "twofish-x86_64-asm-3way.S" .text @@ -220,7 +221,7 @@ rorq $32, RAB2; \ outunpack3(mov, RIO, 2, RAB, 2); =20 -SYM_FUNC_START(__twofish_enc_blk_3way) +SYM_TYPED_FUNC_START(__twofish_enc_blk_3way) /* input: * %rdi: ctx, CTX * %rsi: dst @@ -269,7 +270,7 @@ SYM_FUNC_START(__twofish_enc_blk_3way) RET; SYM_FUNC_END(__twofish_enc_blk_3way) =20 -SYM_FUNC_START(twofish_dec_blk_3way) +SYM_TYPED_FUNC_START(twofish_dec_blk_3way) /* input: * %rdi: ctx, CTX * %rsi: dst Index: linux-2.6/arch/x86/crypto/twofish-x86_64-asm_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/crypto/twofish-x86_64-asm_64.S +++ linux-2.6/arch/x86/crypto/twofish-x86_64-asm_64.S @@ -8,6 +8,7 @@ .text =20 #include +#include #include =20 #define a_offset 0 @@ -202,7 +203,7 @@ xor %r8d, d ## D;\ ror $1, d ## D; =20 -SYM_FUNC_START(twofish_enc_blk) +SYM_TYPED_FUNC_START(twofish_enc_blk) pushq R1 =20 /* %rdi contains the ctx address */ @@ -255,7 +256,7 @@ SYM_FUNC_START(twofish_enc_blk) RET SYM_FUNC_END(twofish_enc_blk) =20 -SYM_FUNC_START(twofish_dec_blk) +SYM_TYPED_FUNC_START(twofish_dec_blk) pushq R1 =20 /* %rdi contains the ctx address */ Index: linux-2.6/arch/x86/lib/clear_page_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/lib/clear_page_64.S +++ linux-2.6/arch/x86/lib/clear_page_64.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0-only */ #include #include +#include #include =20 /* @@ -14,7 +15,7 @@ * Zero a page. * %rdi - page */ -SYM_FUNC_START(clear_page_rep) +SYM_TYPED_FUNC_START(clear_page_rep) movl $4096/8,%ecx xorl %eax,%eax rep stosq @@ -22,7 +23,7 @@ SYM_FUNC_START(clear_page_rep) SYM_FUNC_END(clear_page_rep) EXPORT_SYMBOL_GPL(clear_page_rep) =20 -SYM_FUNC_START(clear_page_orig) +SYM_TYPED_FUNC_START(clear_page_orig) xorl %eax,%eax movl $4096/64,%ecx .p2align 4 @@ -44,7 +45,7 @@ SYM_FUNC_START(clear_page_orig) SYM_FUNC_END(clear_page_orig) EXPORT_SYMBOL_GPL(clear_page_orig) =20 -SYM_FUNC_START(clear_page_erms) +SYM_TYPED_FUNC_START(clear_page_erms) movl $4096,%ecx xorl %eax,%eax rep stosb Index: linux-2.6/arch/x86/lib/copy_page_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/lib/copy_page_64.S +++ linux-2.6/arch/x86/lib/copy_page_64.S @@ -3,6 +3,7 @@ =20 #include #include +#include #include #include =20 @@ -13,7 +14,7 @@ * prefetch distance based on SMP/UP. */ ALIGN -SYM_FUNC_START(copy_page) +SYM_TYPED_FUNC_START(copy_page) ALTERNATIVE "jmp copy_page_regs", "", X86_FEATURE_REP_GOOD movl $4096/8, %ecx rep movsq Index: linux-2.6/arch/x86/lib/memmove_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/lib/memmove_64.S +++ linux-2.6/arch/x86/lib/memmove_64.S @@ -8,6 +8,7 @@ */ #include #include +#include #include #include =20 @@ -26,7 +27,7 @@ * Output: * rax: dest */ -SYM_FUNC_START(__memmove) +SYM_TYPED_FUNC_START(__memmove) =20 mov %rdi, %rax =20 Index: linux-2.6/arch/x86/lib/memset_64.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/lib/memset_64.S +++ linux-2.6/arch/x86/lib/memset_64.S @@ -3,6 +3,7 @@ =20 #include #include +#include #include #include =20 @@ -28,7 +29,7 @@ * only for the return value that is the same as the source input, * which the compiler could/should do much better anyway. */ -SYM_FUNC_START(__memset) +SYM_TYPED_FUNC_START(__memset) ALTERNATIVE "jmp memset_orig", "", X86_FEATURE_FSRS =20 movq %rdi,%r9 Index: linux-2.6/arch/x86/lib/msr-reg.S =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-2.6.orig/arch/x86/lib/msr-reg.S +++ linux-2.6/arch/x86/lib/msr-reg.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include #include +#include #include #include =20 @@ -12,7 +13,7 @@ * */ .macro op_safe_regs op -SYM_FUNC_START(\op\()_safe_regs) +SYM_TYPED_FUNC_START(\op\()_safe_regs) pushq %rbx pushq %r12 movq %rdi, %r10 /* Save pointer */ From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16DD21F7904 for ; Fri, 7 Feb 2025 12:28:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931321; cv=none; b=KiE/ayYl+vUrg4xj9P8EIOhM4skZhESGo1wiG6vSXNN2zKUmO4TPFoD0pI9a3ymbEQGEgChJL+kXSlD47hxNzeY4YYRyHvLLW/WhJ9ZlDx58orlbmyFbNyqPrgO5bz1mkynb5TLUNJOKpG/QThfxo+SXAxL80/1roLeoQpmAinM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931321; c=relaxed/simple; bh=2+nV77ZoLopzx2d+EAMCuPEOFYVhcX7f4wQ8ZKn4Tf4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Vajs5kJ657pq0wlEQ1PY5mhJuOV+qGOome3YmyEsVEJSq/Np2xNZinFKC1ZgFwxBGWvGjULlDTqIbjho2dcbs8rsvsTrGDDNoZt5IRVg4d04iTTvtKHW/mUStfkzs9ukppcIHwesxLJ98DmReu/RjQqrQVqwUrMNeE7ZVi0DP9g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=dP9oELo4; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="dP9oELo4" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=KRrkEeDD30H5DgTE9HjTTFPhhcTzdzfEEZMtDf5cvSo=; b=dP9oELo4EjQ74lK3CVFC7grIT0 zEaKKh3YNaXPYEb+R4Ov8KrtUqSOpPPqs0jjEO41YoHHiXDdjxpH1CuaBPt6gchx6Fq34A2OmqZG7 7etzRiXLJydC22o07NamHhBNOnpdMtM2lcsct9fDdKCDgQdfOXjTDiUh5trouey7Y8m/GTMDSsS3R qIqAVi6cQUWouzVCeJ4LJYdX3nC4PoNuiVx2q1CPsPcdVUEW4t6daDfkGKcccP0kOxiBn0S0IdeTf T6EiFH9mS+rWdkU6yd4RjJ6Q3RoriyBSwlmZafRNJepa4bG+k/LbqOYTHacRlD/eMLzhWchXqlB9H 95q+iA9A==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT7-0000000HAeV-2fqp; Fri, 07 Feb 2025 12:28:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 2F3BD30077D; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.409116003@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:33 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 04/11] x86/cfi: Clean up linkage References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With the introduction of kCFI the addition of ENDBR to SYM_FUNC_START* no longer suffices to make the function indirectly callable. This now requires the use of SYM_TYPED_FUNC_START. As such, remove the implicit ENDBR from SYM_FUNC_START* and add some explicit annotations to fix things up again. Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/crypto/aesni-intel_asm.S | 2 ++ arch/x86/entry/calling.h | 1 + arch/x86/entry/entry.S | 2 ++ arch/x86/entry/entry_64.S | 3 +++ arch/x86/entry/entry_64_fred.S | 1 + arch/x86/entry/vdso/Makefile | 1 + arch/x86/include/asm/linkage.h | 18 ++++++------------ arch/x86/include/asm/page_64.h | 1 + arch/x86/include/asm/paravirt_types.h | 12 +++++++++++- arch/x86/include/asm/special_insns.h | 4 ++-- arch/x86/include/asm/string_64.h | 2 ++ arch/x86/kernel/acpi/madt_playdead.S | 1 + arch/x86/kernel/acpi/wakeup_64.S | 1 + arch/x86/kernel/alternative.c | 8 ++------ arch/x86/kernel/ftrace_64.S | 5 +++++ arch/x86/kernel/irqflags.S | 1 + arch/x86/kernel/paravirt.c | 14 ++++++++++++-- arch/x86/lib/clear_page_64.S | 2 ++ arch/x86/lib/copy_user_64.S | 3 +++ arch/x86/lib/copy_user_uncached_64.S | 2 ++ arch/x86/lib/getuser.S | 9 +++++++++ arch/x86/lib/hweight.S | 3 +++ arch/x86/lib/putuser.S | 9 +++++++++ arch/x86/lib/retpoline.S | 1 + arch/x86/mm/mem_encrypt_boot.S | 1 + arch/x86/power/hibernate_asm_64.S | 2 ++ arch/x86/xen/xen-asm.S | 5 +++++ arch/x86/xen/xen-head.S | 2 ++ include/linux/compiler.h | 10 ++++++++++ 29 files changed, 103 insertions(+), 23 deletions(-) --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -17,6 +17,7 @@ */ =20 #include +#include #include =20 #define STATE1 %xmm0 @@ -1071,6 +1072,7 @@ SYM_FUNC_END(_aesni_inc) * size_t len, u8 *iv) */ SYM_FUNC_START(aesni_ctr_enc) + ANNOTATE_NOENDBR FRAME_BEGIN cmp $16, LEN jb .Lctr_enc_just_ret --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -431,6 +431,7 @@ For 32-bit we have the following convent /* rdi: arg1 ... normal C conventions. rax is saved/restored. */ .macro THUNK name, func SYM_FUNC_START(\name) + ANNOTATE_NOENDBR pushq %rbp movq %rsp, %rbp =20 --- a/arch/x86/entry/entry.S +++ b/arch/x86/entry/entry.S @@ -5,6 +5,7 @@ =20 #include #include +#include #include #include #include @@ -17,6 +18,7 @@ .pushsection .noinstr.text, "ax" =20 SYM_FUNC_START(entry_ibpb) + ANNOTATE_NOENDBR movl $MSR_IA32_PRED_CMD, %ecx movl $PRED_CMD_IBPB, %eax xorl %edx, %edx --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -175,6 +175,7 @@ SYM_CODE_END(entry_SYSCALL_64) */ .pushsection .text, "ax" SYM_FUNC_START(__switch_to_asm) + ANNOTATE_NOENDBR /* * Save callee-saved registers * This must match the order in inactive_task_frame @@ -742,6 +743,7 @@ _ASM_NOKPROBE(common_interrupt_return) * Is in entry.text as it shouldn't be instrumented. */ SYM_FUNC_START(asm_load_gs_index) + ANNOTATE_NOENDBR FRAME_BEGIN swapgs .Lgs_change: @@ -1526,6 +1528,7 @@ SYM_CODE_END(rewind_stack_and_make_dead) * refactored in the future if needed. */ SYM_FUNC_START(clear_bhb_loop) + ANNOTATE_NOENDBR push %rbp mov %rsp, %rbp movl $5, %ecx --- a/arch/x86/entry/entry_64_fred.S +++ b/arch/x86/entry/entry_64_fred.S @@ -58,6 +58,7 @@ SYM_CODE_END(asm_fred_entrypoint_kernel) =20 #if IS_ENABLED(CONFIG_KVM_INTEL) SYM_FUNC_START(asm_fred_entry_from_kvm) + ANNOTATE_NOENDBR push %rbp mov %rsp, %rbp =20 --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -133,6 +133,7 @@ KBUILD_CFLAGS_32 +=3D -fno-stack-protector KBUILD_CFLAGS_32 +=3D $(call cc-option, -foptimize-sibling-calls) KBUILD_CFLAGS_32 +=3D -fno-omit-frame-pointer KBUILD_CFLAGS_32 +=3D -DDISABLE_BRANCH_PROFILING +KBUILD_CFLAGS_32 +=3D -DBUILD_VDSO =20 ifdef CONFIG_MITIGATION_RETPOLINE ifneq ($(RETPOLINE_VDSO_CFLAGS),) --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -119,33 +119,27 @@ =20 /* SYM_FUNC_START -- use for global functions */ #define SYM_FUNC_START(name) \ - SYM_START(name, SYM_L_GLOBAL, SYM_F_ALIGN) \ - ENDBR + SYM_START(name, SYM_L_GLOBAL, SYM_F_ALIGN) =20 /* SYM_FUNC_START_NOALIGN -- use for global functions, w/o alignment */ #define SYM_FUNC_START_NOALIGN(name) \ - SYM_START(name, SYM_L_GLOBAL, SYM_A_NONE) \ - ENDBR + SYM_START(name, SYM_L_GLOBAL, SYM_A_NONE) =20 /* SYM_FUNC_START_LOCAL -- use for local functions */ #define SYM_FUNC_START_LOCAL(name) \ - SYM_START(name, SYM_L_LOCAL, SYM_F_ALIGN) \ - ENDBR + SYM_START(name, SYM_L_LOCAL, SYM_F_ALIGN) =20 /* SYM_FUNC_START_LOCAL_NOALIGN -- use for local functions, w/o alignment = */ #define SYM_FUNC_START_LOCAL_NOALIGN(name) \ - SYM_START(name, SYM_L_LOCAL, SYM_A_NONE) \ - ENDBR + SYM_START(name, SYM_L_LOCAL, SYM_A_NONE) =20 /* SYM_FUNC_START_WEAK -- use for weak functions */ #define SYM_FUNC_START_WEAK(name) \ - SYM_START(name, SYM_L_WEAK, SYM_F_ALIGN) \ - ENDBR + SYM_START(name, SYM_L_WEAK, SYM_F_ALIGN) =20 /* SYM_FUNC_START_WEAK_NOALIGN -- use for weak functions, w/o alignment */ #define SYM_FUNC_START_WEAK_NOALIGN(name) \ - SYM_START(name, SYM_L_WEAK, SYM_A_NONE) \ - ENDBR + SYM_START(name, SYM_L_WEAK, SYM_A_NONE) =20 #endif /* _ASM_X86_LINKAGE_H */ =20 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -60,6 +60,7 @@ static inline void clear_page(void *page } =20 void copy_page(void *to, void *from); +KCFI_REFERENCE(copy_page); =20 #ifdef CONFIG_X86_5LEVEL /* --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -244,7 +244,17 @@ extern struct paravirt_patch_template pv =20 int paravirt_disable_iospace(void); =20 -/* This generates an indirect call based on the operation type number. */ +/* + * This generates an indirect call based on the operation type number. + * + * Since alternatives run after enabling CET/IBT -- the latter setting/cle= aring + * capabilities and the former requiring all capabilities being finalized = -- + * these indirect calls are subject to IBT and the paravirt stubs should h= ave + * ENDBR on. + * + * OTOH since this is effectively a __nocfi indirect call, the paravirt st= ubs + * don't need to bother with CFI prefixes. + */ #define PARAVIRT_CALL \ ANNOTATE_RETPOLINE_SAFE \ "call *%[paravirt_opptr];" --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -42,14 +42,14 @@ static __always_inline void native_write asm volatile("mov %0,%%cr2": : "r" (val) : "memory"); } =20 -static inline unsigned long __native_read_cr3(void) +static __always_inline unsigned long __native_read_cr3(void) { unsigned long val; asm volatile("mov %%cr3,%0\n\t" : "=3Dr" (val) : __FORCE_ORDER); return val; } =20 -static inline void native_write_cr3(unsigned long val) +static __always_inline void native_write_cr3(unsigned long val) { asm volatile("mov %0,%%cr3": : "r" (val) : "memory"); } --- a/arch/x86/include/asm/string_64.h +++ b/arch/x86/include/asm/string_64.h @@ -21,6 +21,7 @@ extern void *__memcpy(void *to, const vo #define __HAVE_ARCH_MEMSET void *memset(void *s, int c, size_t n); void *__memset(void *s, int c, size_t n); +KCFI_REFERENCE(__memset); =20 /* * KMSAN needs to instrument as much code as possible. Use C versions of @@ -70,6 +71,7 @@ static inline void *memset64(uint64_t *s #define __HAVE_ARCH_MEMMOVE void *memmove(void *dest, const void *src, size_t count); void *__memmove(void *dest, const void *src, size_t count); +KCFI_REFERENCE(__memmove); =20 int memcmp(const void *cs, const void *ct, size_t count); size_t strlen(const char *s); --- a/arch/x86/kernel/acpi/madt_playdead.S +++ b/arch/x86/kernel/acpi/madt_playdead.S @@ -14,6 +14,7 @@ * rsi: PGD of the identity mapping */ SYM_FUNC_START(asm_acpi_mp_play_dead) + ANNOTATE_NOENDBR /* Turn off global entries. Following CR3 write will flush them. */ movq %cr4, %rdx andq $~(X86_CR4_PGE), %rdx --- a/arch/x86/kernel/acpi/wakeup_64.S +++ b/arch/x86/kernel/acpi/wakeup_64.S @@ -17,6 +17,7 @@ * Hooray, we are in Long 64-bit mode (but still running in low memory) */ SYM_FUNC_START(wakeup_long64) + ANNOTATE_NOENDBR movq saved_magic(%rip), %rax movq $0x123456789abcdef0, %rdx cmpq %rdx, %rax --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -918,11 +918,7 @@ struct bpf_insn; extern unsigned int __bpf_prog_runX(const void *ctx, const struct bpf_insn *insn); =20 -/* - * Force a reference to the external symbol so the compiler generates - * __kcfi_typid. - */ -__ADDRESSABLE(__bpf_prog_runX); +KCFI_REFERENCE(__bpf_prog_runX); =20 /* u32 __ro_after_init cfi_bpf_hash =3D __kcfi_typeid___bpf_prog_runX; */ asm ( @@ -939,7 +935,7 @@ asm ( /* Must match bpf_callback_t */ extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); =20 -__ADDRESSABLE(__bpf_callback_fn); +KCFI_REFERENCE(__bpf_callback_fn); =20 /* u32 __ro_after_init cfi_bpf_subprog_hash =3D __kcfi_typeid___bpf_callba= ck_fn; */ asm ( --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -146,12 +146,14 @@ SYM_FUNC_END(ftrace_stub_graph) #ifdef CONFIG_DYNAMIC_FTRACE =20 SYM_FUNC_START(__fentry__) + ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT RET SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) =20 SYM_FUNC_START(ftrace_caller) + ANNOTATE_NOENDBR /* save_mcount_regs fills in first two parameters */ save_mcount_regs =20 @@ -197,6 +199,7 @@ SYM_FUNC_END(ftrace_caller); STACK_FRAME_NON_STANDARD_FP(ftrace_caller) =20 SYM_FUNC_START(ftrace_regs_caller) + ANNOTATE_NOENDBR /* Save the current flags before any operations that can change them */ pushfq =20 @@ -310,6 +313,7 @@ SYM_FUNC_END(ftrace_regs_caller) STACK_FRAME_NON_STANDARD_FP(ftrace_regs_caller) =20 SYM_FUNC_START(ftrace_stub_direct_tramp) + ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT RET SYM_FUNC_END(ftrace_stub_direct_tramp) @@ -317,6 +321,7 @@ SYM_FUNC_END(ftrace_stub_direct_tramp) #else /* ! CONFIG_DYNAMIC_FTRACE */ =20 SYM_FUNC_START(__fentry__) + ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT =20 cmpq $ftrace_stub, ftrace_trace_function --- a/arch/x86/kernel/irqflags.S +++ b/arch/x86/kernel/irqflags.S @@ -9,6 +9,7 @@ */ .pushsection .noinstr.text, "ax" SYM_FUNC_START(native_save_fl) + ENDBR pushf pop %_ASM_AX RET --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -116,6 +116,16 @@ static noinstr void pv_native_write_cr2( native_write_cr2(val); } =20 +static noinstr unsigned long pv_native_read_cr3(void) +{ + return __native_read_cr3(); +} + +static noinstr void pv_native_write_cr3(unsigned long cr3) +{ + native_write_cr3(cr3); +} + static noinstr unsigned long pv_native_get_debugreg(int regno) { return native_get_debugreg(regno); @@ -203,8 +213,8 @@ struct paravirt_patch_template pv_ops =3D #ifdef CONFIG_PARAVIRT_XXL .mmu.read_cr2 =3D __PV_IS_CALLEE_SAVE(pv_native_read_cr2), .mmu.write_cr2 =3D pv_native_write_cr2, - .mmu.read_cr3 =3D __native_read_cr3, - .mmu.write_cr3 =3D native_write_cr3, + .mmu.read_cr3 =3D pv_native_read_cr3, + .mmu.write_cr3 =3D pv_native_write_cr3, =20 .mmu.pgd_alloc =3D __paravirt_pgd_alloc, .mmu.pgd_free =3D paravirt_nop, --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -2,6 +2,7 @@ #include #include #include +#include #include =20 /* @@ -64,6 +65,7 @@ EXPORT_SYMBOL_GPL(clear_page_erms) * rcx: uncleared bytes or 0 if successful. */ SYM_FUNC_START(rep_stos_alternative) + ANNOTATE_NOENDBR cmpq $64,%rcx jae .Lunrolled =20 --- a/arch/x86/lib/copy_user_64.S +++ b/arch/x86/lib/copy_user_64.S @@ -8,6 +8,8 @@ =20 #include #include +#include +#include #include #include #include @@ -30,6 +32,7 @@ * it simpler for us, we can clobber rsi/rdi and rax freely. */ SYM_FUNC_START(rep_movs_alternative) + ANNOTATE_NOENDBR cmpq $64,%rcx jae .Llarge =20 --- a/arch/x86/lib/copy_user_uncached_64.S +++ b/arch/x86/lib/copy_user_uncached_64.S @@ -5,6 +5,7 @@ =20 #include #include +#include #include =20 /* @@ -27,6 +28,7 @@ * rax uncopied bytes or 0 if successful. */ SYM_FUNC_START(__copy_user_nocache) + ANNOTATE_NOENDBR /* If destination is not 7-byte aligned, we'll have to align it */ testb $7,%dil jne .Lalign --- a/arch/x86/lib/getuser.S +++ b/arch/x86/lib/getuser.S @@ -28,6 +28,7 @@ =20 #include #include +#include #include #include #include @@ -62,6 +63,7 @@ =20 .text SYM_FUNC_START(__get_user_1) + ANNOTATE_NOENDBR check_range size=3D1 ASM_STAC UACCESS movzbl (%_ASM_AX),%edx @@ -72,6 +74,7 @@ SYM_FUNC_END(__get_user_1) EXPORT_SYMBOL(__get_user_1) =20 SYM_FUNC_START(__get_user_2) + ANNOTATE_NOENDBR check_range size=3D2 ASM_STAC UACCESS movzwl (%_ASM_AX),%edx @@ -82,6 +85,7 @@ SYM_FUNC_END(__get_user_2) EXPORT_SYMBOL(__get_user_2) =20 SYM_FUNC_START(__get_user_4) + ANNOTATE_NOENDBR check_range size=3D4 ASM_STAC UACCESS movl (%_ASM_AX),%edx @@ -92,6 +96,7 @@ SYM_FUNC_END(__get_user_4) EXPORT_SYMBOL(__get_user_4) =20 SYM_FUNC_START(__get_user_8) + ANNOTATE_NOENDBR #ifndef CONFIG_X86_64 xor %ecx,%ecx #endif @@ -111,6 +116,7 @@ EXPORT_SYMBOL(__get_user_8) =20 /* .. and the same for __get_user, just without the range checks */ SYM_FUNC_START(__get_user_nocheck_1) + ANNOTATE_NOENDBR ASM_STAC ASM_BARRIER_NOSPEC UACCESS movzbl (%_ASM_AX),%edx @@ -121,6 +127,7 @@ SYM_FUNC_END(__get_user_nocheck_1) EXPORT_SYMBOL(__get_user_nocheck_1) =20 SYM_FUNC_START(__get_user_nocheck_2) + ANNOTATE_NOENDBR ASM_STAC ASM_BARRIER_NOSPEC UACCESS movzwl (%_ASM_AX),%edx @@ -131,6 +138,7 @@ SYM_FUNC_END(__get_user_nocheck_2) EXPORT_SYMBOL(__get_user_nocheck_2) =20 SYM_FUNC_START(__get_user_nocheck_4) + ANNOTATE_NOENDBR ASM_STAC ASM_BARRIER_NOSPEC UACCESS movl (%_ASM_AX),%edx @@ -141,6 +149,7 @@ SYM_FUNC_END(__get_user_nocheck_4) EXPORT_SYMBOL(__get_user_nocheck_4) =20 SYM_FUNC_START(__get_user_nocheck_8) + ANNOTATE_NOENDBR ASM_STAC ASM_BARRIER_NOSPEC #ifdef CONFIG_X86_64 --- a/arch/x86/lib/hweight.S +++ b/arch/x86/lib/hweight.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include #include +#include =20 #include =20 @@ -9,6 +10,7 @@ * %rdi: w */ SYM_FUNC_START(__sw_hweight32) + ANNOTATE_NOENDBR =20 #ifdef CONFIG_X86_64 movl %edi, %eax # w @@ -42,6 +44,7 @@ EXPORT_SYMBOL(__sw_hweight32) */ #ifdef CONFIG_X86_64 SYM_FUNC_START(__sw_hweight64) + ANNOTATE_NOENDBR pushq %rdi pushq %rdx =20 --- a/arch/x86/lib/putuser.S +++ b/arch/x86/lib/putuser.S @@ -13,6 +13,7 @@ */ #include #include +#include #include #include #include @@ -45,6 +46,7 @@ =20 .text SYM_FUNC_START(__put_user_1) + ANNOTATE_NOENDBR check_range size=3D1 ASM_STAC 1: movb %al,(%_ASM_CX) @@ -55,6 +57,7 @@ SYM_FUNC_END(__put_user_1) EXPORT_SYMBOL(__put_user_1) =20 SYM_FUNC_START(__put_user_nocheck_1) + ANNOTATE_NOENDBR ASM_STAC 2: movb %al,(%_ASM_CX) xor %ecx,%ecx @@ -64,6 +67,7 @@ SYM_FUNC_END(__put_user_nocheck_1) EXPORT_SYMBOL(__put_user_nocheck_1) =20 SYM_FUNC_START(__put_user_2) + ANNOTATE_NOENDBR check_range size=3D2 ASM_STAC 3: movw %ax,(%_ASM_CX) @@ -74,6 +78,7 @@ SYM_FUNC_END(__put_user_2) EXPORT_SYMBOL(__put_user_2) =20 SYM_FUNC_START(__put_user_nocheck_2) + ANNOTATE_NOENDBR ASM_STAC 4: movw %ax,(%_ASM_CX) xor %ecx,%ecx @@ -83,6 +88,7 @@ SYM_FUNC_END(__put_user_nocheck_2) EXPORT_SYMBOL(__put_user_nocheck_2) =20 SYM_FUNC_START(__put_user_4) + ANNOTATE_NOENDBR check_range size=3D4 ASM_STAC 5: movl %eax,(%_ASM_CX) @@ -93,6 +99,7 @@ SYM_FUNC_END(__put_user_4) EXPORT_SYMBOL(__put_user_4) =20 SYM_FUNC_START(__put_user_nocheck_4) + ANNOTATE_NOENDBR ASM_STAC 6: movl %eax,(%_ASM_CX) xor %ecx,%ecx @@ -102,6 +109,7 @@ SYM_FUNC_END(__put_user_nocheck_4) EXPORT_SYMBOL(__put_user_nocheck_4) =20 SYM_FUNC_START(__put_user_8) + ANNOTATE_NOENDBR check_range size=3D8 ASM_STAC 7: mov %_ASM_AX,(%_ASM_CX) @@ -115,6 +123,7 @@ SYM_FUNC_END(__put_user_8) EXPORT_SYMBOL(__put_user_8) =20 SYM_FUNC_START(__put_user_nocheck_8) + ANNOTATE_NOENDBR ASM_STAC 9: mov %_ASM_AX,(%_ASM_CX) #ifdef CONFIG_X86_32 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -326,6 +326,7 @@ SYM_FUNC_END(retbleed_untrain_ret) #if defined(CONFIG_MITIGATION_UNRET_ENTRY) || defined(CONFIG_MITIGATION_SR= SO) =20 SYM_FUNC_START(entry_untrain_ret) + ANNOTATE_NOENDBR ALTERNATIVE JMP_RETBLEED_UNTRAIN_RET, JMP_SRSO_UNTRAIN_RET, X86_FEATURE_S= RSO SYM_FUNC_END(entry_untrain_ret) __EXPORT_THUNK(entry_untrain_ret) --- a/arch/x86/mm/mem_encrypt_boot.S +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -72,6 +72,7 @@ SYM_FUNC_START(sme_encrypt_execute) SYM_FUNC_END(sme_encrypt_execute) =20 SYM_FUNC_START(__enc_copy) + ANNOTATE_NOENDBR /* * Routine used to encrypt memory in place. * This routine must be run outside of the kernel proper since --- a/arch/x86/power/hibernate_asm_64.S +++ b/arch/x86/power/hibernate_asm_64.S @@ -26,6 +26,7 @@ /* code below belongs to the image kernel */ .align PAGE_SIZE SYM_FUNC_START(restore_registers) + ANNOTATE_NOENDBR /* go back to the original page tables */ movq %r9, %cr3 =20 @@ -119,6 +120,7 @@ SYM_FUNC_END(restore_image) =20 /* code below has been relocated to a safe page */ SYM_FUNC_START(core_restore_code) + ANNOTATE_NOENDBR /* switch to temporary page tables */ movq %rax, %cr3 /* flush TLB */ --- a/arch/x86/xen/xen-asm.S +++ b/arch/x86/xen/xen-asm.S @@ -51,6 +51,7 @@ SYM_FUNC_END(xen_hypercall_pv) * non-zero. */ SYM_FUNC_START(xen_irq_disable_direct) + ENDBR movb $1, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) RET SYM_FUNC_END(xen_irq_disable_direct) @@ -90,6 +91,7 @@ SYM_FUNC_END(check_events) * then enter the hypervisor to get them handled. */ SYM_FUNC_START(xen_irq_enable_direct) + ENDBR FRAME_BEGIN /* Unmask events */ movb $0, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) @@ -120,6 +122,7 @@ SYM_FUNC_END(xen_irq_enable_direct) * x86 use opposite senses (mask vs enable). */ SYM_FUNC_START(xen_save_fl_direct) + ENDBR testb $0xff, PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_mask) setz %ah addb %ah, %ah @@ -127,6 +130,7 @@ SYM_FUNC_START(xen_save_fl_direct) SYM_FUNC_END(xen_save_fl_direct) =20 SYM_FUNC_START(xen_read_cr2) + ENDBR FRAME_BEGIN _ASM_MOV PER_CPU_VAR(xen_vcpu), %_ASM_AX _ASM_MOV XEN_vcpu_info_arch_cr2(%_ASM_AX), %_ASM_AX @@ -135,6 +139,7 @@ SYM_FUNC_START(xen_read_cr2) SYM_FUNC_END(xen_read_cr2); =20 SYM_FUNC_START(xen_read_cr2_direct) + ENDBR FRAME_BEGIN _ASM_MOV PER_CPU_VAR(xen_vcpu_info + XEN_vcpu_info_arch_cr2), %_ASM_AX FRAME_END --- a/arch/x86/xen/xen-head.S +++ b/arch/x86/xen/xen-head.S @@ -138,11 +138,13 @@ SYM_FUNC_START(xen_hypercall_hvm) SYM_FUNC_END(xen_hypercall_hvm) =20 SYM_FUNC_START(xen_hypercall_amd) + ANNOTATE_NOENDBR vmmcall RET SYM_FUNC_END(xen_hypercall_amd) =20 SYM_FUNC_START(xen_hypercall_intel) + ANNOTATE_NOENDBR vmcall RET SYM_FUNC_END(xen_hypercall_intel) --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -193,6 +193,16 @@ void ftrace_likely_update(struct ftrace_ =20 #endif /* __KERNEL__ */ =20 +#if defined(CONFIG_CFI_CLANG) && !defined(__DISABLE_EXPORTS) && !defined(B= UILD_VDSO) +/* + * Force a reference to the external symbol so the compiler generates + * __kcfi_typid. + */ +#define KCFI_REFERENCE(sym) __ADDRESSABLE(sym) +#else +#define KCFI_REFERENCE(sym) +#endif + /** * offset_to_ptr - convert a relative memory offset to an absolute pointer * @off: the address of the 32-bit offset value From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9D951E0DE5 for ; Fri, 7 Feb 2025 12:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931313; cv=none; b=MQse8PekR2LOTI1DPWex66t+uZZ6zOkC1k60/btF9YYFAM91xU3eYpfidAPlOBbQ4+vx1i+nbAh6kxt2NlL78pTdcnF0M+RRt1ZLjpuoYEhoXTyK1rR7YZSW7k2kWtpGGZm3wHAVyGht+UFuJmR+NQGQcR5gHuOkBOqRQvfpnfY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931313; c=relaxed/simple; bh=6Q1bNawOVx4uNLkKUtLr5ZKzVL0qJcYQRtjgWlkEBLo=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=L9Y+QAerGHs8U3yzyZ41CNnhHs5SiO/9tRfOtCcQ7a3IhBxYvWDo1dZpdslWLlvR4B6YhaC2U34aYqS91KK/Edlh0AF9eJ4chr1aTA4qAcmJzTXEmOpIICOuy4Kr0h2E2oMRT8qMNfZNMmpJkJLuAkrD71CGJ/2aP4xh5UUpgQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=MrzKa/jY; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="MrzKa/jY" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=8YwrG2M1NyVkR8ph2MYJ1NCP71F6/4x9C2wwGwvDUvk=; b=MrzKa/jYlpQsyAYsJzfAdOJ/qC Ri9HhwVgXMNWyh2ZRFO8wnzJZL4lAfoHGnsWI3EVJHYIl6nqIHcKZhmqP+vxOZLMgEMZmTWK9TMGg 3Mcrf+Jj0/4Ptl10QNeCIF6W0zYHbPPaFFrii4j4KljgzX5yNRgB3tpzJJcFl43NXzJBau9jgLxh5 cpcZgl519PtkFewpmt9ymhUPdR1zrn0IWEVJePj4MQX+X31pYn71vdzuUP0FJoFI3L/cO1tSUfsq+ BSEzerLdY7NotinxayWPSRde5NXxPeqU5PB6GmrKWz5/W8/4huYlMoqdRInaZhW6VYNM/sfZh0CVU hwcAM3RA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT8-0000000HAec-3FsO; Fri, 07 Feb 2025 12:28:27 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 328C130088D; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.509520369@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:34 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 05/11] x86/boot: Mark start_secondary() with __noendbr References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The handoff between the boot stubs and start_secondary() are before IBT is enabled and is definitely not subject to kCFI. As such, suppress all that f= or this function. Notably when the ENDBR poison would become fatal (ud1 instead of nop) this = will trigger a tripple fault because we haven't set up the IDT to handle #UD yet. Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/kernel/smpboot.c | 3 ++- include/linux/objtool.h | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -229,7 +229,7 @@ static void ap_calibrate_delay(void) /* * Activate a secondary processor. */ -static void notrace start_secondary(void *unused) +static void notrace __noendbr start_secondary(void *unused) { /* * Don't put *anything* except direct CPU state initialization @@ -314,6 +314,7 @@ static void notrace start_secondary(void wmb(); cpu_startup_entry(CPUHP_AP_ONLINE_IDLE); } +ANNOTATE_NOENDBR_SYM(start_secondary); =20 /* * The bootstrap kernel entry code has set these up. Save them for --- a/include/linux/objtool.h +++ b/include/linux/objtool.h @@ -128,7 +128,7 @@ #define UNWIND_HINT(type, sp_reg, sp_offset, signal) "\n\t" #define STACK_FRAME_NON_STANDARD(func) #define STACK_FRAME_NON_STANDARD_FP(func) -#define __ASM_ANNOTATE(label, type) +#define __ASM_ANNOTATE(label, type) "" #define ASM_ANNOTATE(type) #else .macro UNWIND_HINT type:req sp_reg=3D0 sp_offset=3D0 signal=3D0 @@ -147,6 +147,8 @@ * these relocations will never be used for indirect calls. */ #define ANNOTATE_NOENDBR ASM_ANNOTATE(ANNOTYPE_NOENDBR) +#define ANNOTATE_NOENDBR_SYM(sym) asm(__ASM_ANNOTATE(sym, ANNOTYPE_NOENDBR= )) + /* * This should be used immediately before an indirect jump/call. It tells * objtool the subsequent indirect jump/call is vouched safe for retpoline From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0F251F8918 for ; Fri, 7 Feb 2025 12:28:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931323; cv=none; b=K6d6pH2Bfswj3oJXGfloPU13Y/Kbv/BwDYAMLoi9fZEqRCI0MILCcIbTbd8WHsytqfiuzTMcP3JKh6HQe3w0oGG8B2NPZDQcl0KMCDFZ/y84NyO9MiOEhr15lKT4l6dp4JEutM3VXscM/AsTw6AGwyHA971yw6JP7at3OKf4oAY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931323; c=relaxed/simple; bh=BdRShRuIoqsgS68j9Muq3hOau2UecWNNuUq5IdIQaKw=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=o4vH3nxjSi5vfx0OnwlTF2/QyikCUHIe/pSuMkX50WJKeVe2cKO8JPePlQ0KJLfiAt6HXc9mFCbSlQbjXaNV+xJ3HHq/2prqXZz9F6PtnlcjvTrctB79MZ0aGL2N4fD9UXvIAWur27HRdr395sR6hSmkuSYt9Edhl2LFZbMbXCk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=XPl6HgrX; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="XPl6HgrX" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=q6LpPIzm96aet4wmhQh2mb7wqpAhvniRSOi7gL2hIfI=; b=XPl6HgrXbMuHQSSmFQU/kjGg+U LPO6jXWy95cLk/ntI34apvLkYM/0lFUXKdcO7/WNbKzg3bIU01dysvKk7GhRX9CGoJcGVUghOind7 THUHxQCTnkDwzsTsOQACKJ/4amp0S03UY1X9mrl9/K91Vm0Nr+VCljBKr9oKVyQQ8e5qTXK7V9K1R PZqUSPHM6S0RMwztk/mmgZFw5NLIWLo7emIytdM/DMawgPEYYJn4LOW8MhZGoc+flucfJFK9sk/rg 8tAFBoCMNk4PIe7x1w/RRzoDuIjEZ/hATIJ9Jr+AK5WuSUdrngFp/UOat8Y8oPqLc7MvXPboRtPsT W9cDcngw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT8-0000000HAed-3HSo; Fri, 07 Feb 2025 12:28:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 36313302E7A; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.617187089@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:35 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 06/11] x86/alternative: Simplify callthunk patching References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that paravirt call patching is implemented using alternatives, it is possible to avoid having to patch the alternative sites by including the altinstr_replacement calls in the call_sites list. This means we're now stacking relative adjustments like so: callthunks_patch_builtin_calls(): patches all function calls to target: func() -> func()-10 since the CALL accounting lives in the CALL_PADDING. This explicitly includes .altinstr_replacement alt_replace_call(): patches: x86_BUG() -> target() this patching is done in a relative manner, and will preserve the above adjustment, meaning that with calldepth patching it will do: x86_BUG()-10 -> target()-10 apply_relocation(): does code relocation, and adjusts all RIP-relative instructions to the new location, also in a relative manner. Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/include/asm/alternative.h | 1 - arch/x86/kernel/alternative.c | 8 ++++---- arch/x86/kernel/callthunks.c | 13 ------------- arch/x86/kernel/module.c | 17 ++++++----------- tools/objtool/arch/x86/decode.c | 1 + tools/objtool/check.c | 12 ++---------- 6 files changed, 13 insertions(+), 39 deletions(-) --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -100,7 +100,6 @@ struct module; =20 struct callthunk_sites { s32 *call_start, *call_end; - struct alt_instr *alt_start, *alt_end; }; =20 #ifdef CONFIG_CALL_THUNKS --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1697,14 +1697,14 @@ void __init alternative_instructions(voi apply_retpolines(__retpoline_sites, __retpoline_sites_end); apply_returns(__return_sites, __return_sites_end); =20 - apply_alternatives(__alt_instructions, __alt_instructions_end); - /* - * Now all calls are established. Apply the call thunks if - * required. + * Adjust all CALL instructions to point to func()-10, including + * those in .altinstr_replacement. */ callthunks_patch_builtin_calls(); =20 + apply_alternatives(__alt_instructions, __alt_instructions_end); + /* * Seal all functions that do not have their address taken. */ --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -240,21 +240,10 @@ patch_call_sites(s32 *start, s32 *end, c } =20 static __init_or_module void -patch_alt_call_sites(struct alt_instr *start, struct alt_instr *end, - const struct core_text *ct) -{ - struct alt_instr *a; - - for (a =3D start; a < end; a++) - patch_call((void *)&a->instr_offset + a->instr_offset, ct); -} - -static __init_or_module void callthunks_setup(struct callthunk_sites *cs, const struct core_text *ct) { prdbg("Patching call sites %s\n", ct->name); patch_call_sites(cs->call_start, cs->call_end, ct); - patch_alt_call_sites(cs->alt_start, cs->alt_end, ct); prdbg("Patching call sites done%s\n", ct->name); } =20 @@ -263,8 +252,6 @@ void __init callthunks_patch_builtin_cal struct callthunk_sites cs =3D { .call_start =3D __call_sites, .call_end =3D __call_sites_end, - .alt_start =3D __alt_instructions, - .alt_end =3D __alt_instructions_end }; =20 if (!cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -275,12 +275,7 @@ int module_finalize(const Elf_Ehdr *hdr, void *rseg =3D (void *)returns->sh_addr; apply_returns(rseg, rseg + returns->sh_size); } - if (alt) { - /* patch .altinstructions */ - void *aseg =3D (void *)alt->sh_addr; - apply_alternatives(aseg, aseg + alt->sh_size); - } - if (calls || alt) { + if (calls) { struct callthunk_sites cs =3D {}; =20 if (calls) { @@ -288,13 +283,13 @@ int module_finalize(const Elf_Ehdr *hdr, cs.call_end =3D (void *)calls->sh_addr + calls->sh_size; } =20 - if (alt) { - cs.alt_start =3D (void *)alt->sh_addr; - cs.alt_end =3D (void *)alt->sh_addr + alt->sh_size; - } - callthunks_patch_module_calls(&cs, me); } + if (alt) { + /* patch .altinstructions */ + void *aseg =3D (void *)alt->sh_addr; + apply_alternatives(aseg, aseg + alt->sh_size); + } if (ibt_endbr) { void *iseg =3D (void *)ibt_endbr->sh_addr; apply_seal_endbr(iseg, iseg + ibt_endbr->sh_size); --- a/tools/objtool/arch/x86/decode.c +++ b/tools/objtool/arch/x86/decode.c @@ -850,5 +850,6 @@ bool arch_is_rethunk(struct symbol *sym) bool arch_is_embedded_insn(struct symbol *sym) { return !strcmp(sym->name, "retbleed_return_thunk") || + !strcmp(sym->name, "srso_alias_safe_ret") || !strcmp(sym->name, "srso_safe_ret"); } --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1283,15 +1283,6 @@ static void annotate_call_site(struct ob if (!sym) sym =3D reloc->sym; =20 - /* - * Alternative replacement code is just template code which is - * sometimes copied to the original instruction. For now, don't - * annotate it. (In the future we might consider annotating the - * original instruction if/when it ever makes sense to do so.) - */ - if (!strcmp(insn->sec->name, ".altinstr_replacement")) - return; - if (sym->static_call_tramp) { list_add_tail(&insn->call_node, &file->static_call_list); return; @@ -1349,7 +1340,8 @@ static void annotate_call_site(struct ob return; } =20 - if (insn->type =3D=3D INSN_CALL && !insn->sec->init) + if (insn->type =3D=3D INSN_CALL && !insn->sec->init && + !insn->_call_dest->embedded_insn) list_add_tail(&insn->call_node, &file->call_list); =20 if (!sibling && dead_end_function(file, sym)) From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0EC91F8916 for ; Fri, 7 Feb 2025 12:28:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931323; cv=none; b=nXWBhmrsrkxC30qIAflsEDWIFTmUjAVyJ1k7xpHZNWl+PwmHZ5VhtHpEZctRtxH2c8uKfIKzPvawkqU6JMoLZjnRQRFd9YXUyXAntnZnVgcFYnVV7wQo9ljgQtWvoMYGD46Xxa4RcPUR/zQHOWftV9geMJ6cRR6g34YWL+gm2Mc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931323; c=relaxed/simple; bh=UIpXAWe7qZ2zYq+NLH5QockvHxWbiZUlnY+HasPUK7Y=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=P4ytEy2IWGsg1FBSd9r0gR7bbXhsdrYBdTtGR7KAKXz7dR7OeWZsRqMfOvjv5V8IndODo9xNEOWNDCti/Y4kQoQgkz1Yo2VUdAhCY3Ad7+UVgEQAAObNHIm+dtYeLVbDkZ/v3m8DW3A1/Bf6OLE105+gzuO35YJoLJJWuSkXq/s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=UiQ+oj6b; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="UiQ+oj6b" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=L5KFRG9+qs4ZpSTEsGUBPsv/WS7rphmSwxBen5PJStA=; b=UiQ+oj6b/U1jcYvxHaiDYyFvjd dXxgCh3TbEW0oL65w441Ciqjq3/Nplw7DIpKnsj6tNF+nWv04IwDiBT3Hkgq+vygXHV7hsOgji6AB WaJtX9mV7F7KqSTQWqZnyqwe+TW1SSOQ8q9y3nJl+TPdwiCZppMWcjHE8Dh80I2BGkWSMLDl7QoMw cLnlkjXkD0uQvsMhftdf/BJIw2dutbPbeNu0GEmXJhnpNzNAvu6VllXxSkO/YiSwaEqOEVqOmS0RD f7HnFDVCjqM0pEQefQn//2yy+vwfzBwZ6QHi/hm2n08ngZL9M9ZLuvynY1EUZUHOl4+uD4vgvwpJ+ l02ingsw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT8-0000000HAee-4362; Fri, 07 Feb 2025 12:28:38 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 39A8C302E9C; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.721120726@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:36 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 07/11] x86/traps: Cleanup and robustify decode_bug() References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Notably, don't attempt to decode an immediate when MOD =3D=3D 3. Additionally have it return the instruction length, such that WARN like bugs can more reliably skip to the correct instruction. Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/include/asm/bug.h | 5 +- arch/x86/include/asm/ibt.h | 4 +- arch/x86/kernel/traps.c | 82 ++++++++++++++++++++++++++++++++--------= ----- 3 files changed, 65 insertions(+), 26 deletions(-) --- a/arch/x86/include/asm/bug.h +++ b/arch/x86/include/asm/bug.h @@ -22,8 +22,9 @@ #define SECOND_BYTE_OPCODE_UD2 0x0b =20 #define BUG_NONE 0xffff -#define BUG_UD1 0xfffe -#define BUG_UD2 0xfffd +#define BUG_UD2 0xfffe +#define BUG_UD1 0xfffd +#define BUG_UD1_UBSAN 0xfffc =20 #ifdef CONFIG_GENERIC_BUG =20 --- a/arch/x86/include/asm/ibt.h +++ b/arch/x86/include/asm/ibt.h @@ -41,7 +41,7 @@ _ASM_PTR fname "\n\t" \ ".popsection\n\t" =20 -static inline __attribute_const__ u32 gen_endbr(void) +static __always_inline __attribute_const__ u32 gen_endbr(void) { u32 endbr; =20 @@ -56,7 +56,7 @@ static inline __attribute_const__ u32 ge return endbr; } =20 -static inline __attribute_const__ u32 gen_endbr_poison(void) +static __always_inline __attribute_const__ u32 gen_endbr_poison(void) { /* * 4 byte NOP that isn't NOP4 (in fact it is OSP NOP3), such that it --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -94,10 +94,17 @@ __always_inline int is_valid_bugaddr(uns =20 /* * Check for UD1 or UD2, accounting for Address Size Override Prefixes. - * If it's a UD1, get the ModRM byte to pass along to UBSan. + * If it's a UD1, further decode to determine its use: + * + * UBSan{0}: 67 0f b9 00 ud1 (%eax),%eax + * UBSan{10}: 67 0f b9 40 10 ud1 0x10(%eax),%eax + * static_call: 0f b9 cc ud1 %esp,%ecx + * + * Notably UBSAN uses EAX, static_call uses ECX. */ -__always_inline int decode_bug(unsigned long addr, u32 *imm) +__always_inline int decode_bug(unsigned long addr, s32 *imm, int *len) { + unsigned long start =3D addr; u8 v; =20 if (addr < TASK_SIZE_MAX) @@ -110,24 +117,42 @@ __always_inline int decode_bug(unsigned return BUG_NONE; =20 v =3D *(u8 *)(addr++); - if (v =3D=3D SECOND_BYTE_OPCODE_UD2) + if (v =3D=3D SECOND_BYTE_OPCODE_UD2) { + *len =3D addr - start; return BUG_UD2; + } =20 - if (!IS_ENABLED(CONFIG_UBSAN_TRAP) || v !=3D SECOND_BYTE_OPCODE_UD1) + if (v !=3D SECOND_BYTE_OPCODE_UD1) return BUG_NONE; =20 - /* Retrieve the immediate (type value) for the UBSAN UD1 */ - v =3D *(u8 *)(addr++); - if (X86_MODRM_RM(v) =3D=3D 4) - addr++; - *imm =3D 0; - if (X86_MODRM_MOD(v) =3D=3D 1) - *imm =3D *(u8 *)addr; - else if (X86_MODRM_MOD(v) =3D=3D 2) - *imm =3D *(u32 *)addr; - else - WARN_ONCE(1, "Unexpected MODRM_MOD: %u\n", X86_MODRM_MOD(v)); + v =3D *(u8 *)(addr++); /* ModRM */ + + if (X86_MODRM_MOD(v) !=3D 3 && X86_MODRM_RM(v) =3D=3D 4) + addr++; /* SIB */ + + /* Decode immediate, if present */ + switch (X86_MODRM_MOD(v)) { + case 0: if (X86_MODRM_RM(v) =3D=3D 5) + addr +=3D 4; /* RIP + disp32 */ + break; + + case 1: *imm =3D *(s8 *)addr; + addr +=3D 1; + break; + + case 2: *imm =3D *(s32 *)addr; + addr +=3D 4; + break; + + case 3: break; + } + + /* record instruction length */ + *len =3D addr - start; + + if (X86_MODRM_REG(v) =3D=3D 0) /* EAX */ + return BUG_UD1_UBSAN; =20 return BUG_UD1; } @@ -258,10 +283,10 @@ static inline void handle_invalid_op(str static noinstr bool handle_bug(struct pt_regs *regs) { bool handled =3D false; - int ud_type; - u32 imm; + int ud_type, ud_len; + s32 ud_imm; =20 - ud_type =3D decode_bug(regs->ip, &imm); + ud_type =3D decode_bug(regs->ip, &ud_imm, &ud_len); if (ud_type =3D=3D BUG_NONE) return handled; =20 @@ -281,15 +306,28 @@ static noinstr bool handle_bug(struct pt */ if (regs->flags & X86_EFLAGS_IF) raw_local_irq_enable(); - if (ud_type =3D=3D BUG_UD2) { + + switch (ud_type) { + case BUG_UD2: if (report_bug(regs->ip, regs) =3D=3D BUG_TRAP_TYPE_WARN || handle_cfi_failure(regs) =3D=3D BUG_TRAP_TYPE_WARN) { - regs->ip +=3D LEN_UD2; + regs->ip +=3D ud_len; handled =3D true; } - } else if (IS_ENABLED(CONFIG_UBSAN_TRAP)) { - pr_crit("%s at %pS\n", report_ubsan_failure(regs, imm), (void *)regs->ip= ); + break; + + case BUG_UD1_UBSAN: + if (IS_ENABLED(CONFIG_UBSAN_TRAP)) { + pr_crit("%s at %pS\n", + report_ubsan_failure(regs, ud_imm), + (void *)regs->ip); + } + break; + + default: + break; } + if (regs->flags & X86_EFLAGS_IF) raw_local_irq_disable(); instrumentation_end(); From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE5961F8697 for ; Fri, 7 Feb 2025 12:28:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931323; cv=none; b=ANdJWQZReNTbWj383xJxg48HCKeUF7xJOG+Mfp9FR27piQcg52Jpw1VahWp8QJO6LLZp6032NNGiRRXe4LHM6C1412qEXqrUvG6qU+kV01driBrdEU0GIC8cxG5PpEquu9VSgNSQYfnYNEtmGQHgtvgxSaNFPi0xFvyhqtIdqus= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931323; c=relaxed/simple; bh=s3M+NkV0tESd6HPi0p7Oe7KL+c5J0dxnA8hEAnIxKh0=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=M3ZP2MwBMZrF8IhdNLwmPzJJQYUfphrrQrKV0oaAHs46sp79XbNWH1aX+MGoAAKFtl4/he7BdIt5Gi0QgEukdfH1GdSYUJPZCcTo7ybofIsQPckGq2IB8fE/DlcFarjebWHEOdxnervbsCyZtGcP+MtxWhiehLOLthGFd2S7soI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=iHz6KG+u; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="iHz6KG+u" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=eRROD8MHw5KbWF+hIKE4+BwnoGjptOnpyqPd+fBz2Os=; b=iHz6KG+uqE5bkQiZ1avRflLvWN 8hXQO4IB8b+PnCnbkl8nzJTUzn/RbVfTADXl14sctp0Kk7x7vadvYbMNm4W6XCx5XqZQEfU+iDASr OaKeDQnH32SfDwZ++FqfEDaGgZWzmvuKHVMYaFI3YAsu8PrmOReR5ZOpPATHkQADEVUufcFvnaF0j 0LXwjuHhXBxLL4nLLXVMed1d4khQqNIm6rfg7HPf02Gv/sFOzaT3OSo49ghgbSZ6D/kTFrbELV5YC B1erMj1A/8PQYt3/ROVmdU78a/JyySR1RNoETYmUkUueRoDCziFfyy+fNTMHHERc6RPYKMUU13am8 qyfjr6VA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNT8-0000000HAef-48yP; Fri, 07 Feb 2025 12:28:38 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 3D2403031DE; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.815505775@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:37 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 08/11] x86/ibt: Clean up poison_endbr() References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Basically, get rid of the .warn argument and explicitly don't call the function when we know there isn't an endbr. This makes the calling code clearer. Note: perhaps don't add functions to .cfi_sites when the function doesn't have endbr -- OTOH why would the compiler emit the prefix if it has already determined there are no indirect callers and has omitted the ENDBR instruction. Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/kernel/alternative.c | 43 +++++++++++++++++++++++++++++++++++--= ----- 1 file changed, 36 insertions(+), 7 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -865,14 +865,12 @@ __noendbr bool is_endbr(u32 *val) =20 static void poison_cfi(void *addr); =20 -static void __init_or_module poison_endbr(void *addr, bool warn) +static void __init_or_module poison_endbr(void *addr) { u32 poison =3D gen_endbr_poison(); =20 - if (!is_endbr(addr)) { - WARN_ON_ONCE(warn); + if (WARN_ON_ONCE(!is_endbr(addr))) return; - } =20 DPRINTK(ENDBR, "ENDBR at: %pS (%px)", addr, addr); =20 @@ -897,7 +895,7 @@ void __init_or_module noinline apply_sea for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; =20 - poison_endbr(addr, true); + poison_endbr(addr); if (IS_ENABLED(CONFIG_FINEIBT)) poison_cfi(addr - 16); } @@ -1200,6 +1198,14 @@ static int cfi_rewrite_preamble(s32 *sta void *addr =3D (void *)s + *s; u32 hash; =20 + /* + * When the function doesn't start with ENDBR the compiler will + * have determined there are no indirect calls to it and we + * don't need no CFI either. + */ + if (!is_endbr(addr + 16)) + continue; + hash =3D decode_preamble_hash(addr); if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", addr, addr, 5, addr)) @@ -1220,7 +1226,10 @@ static void cfi_rewrite_endbr(s32 *start for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; =20 - poison_endbr(addr+16, false); + if (!is_endbr(addr + 16)) + continue; + + poison_endbr(addr + 16); } } =20 @@ -1353,9 +1362,23 @@ static inline void poison_hash(void *add =20 static void poison_cfi(void *addr) { + /* + * Compilers manage to be inconsistent with ENDBR vs __cfi prefixes, + * some (static) functions for which they can determine the address + * is never taken do not get a __cfi prefix, but *DO* get an ENDBR. + * + * As such, these functions will get sealed, but we need to be careful + * to not unconditionally scribble the previous function. + */ switch (cfi_mode) { case CFI_FINEIBT: /* + * FineIBT prefix should start with an ENDBR. + */ + if (!is_endbr(addr)) + break; + + /* * __cfi_\func: * osp nopl (%rax) * subl $0, %r10d @@ -1363,12 +1386,18 @@ static void poison_cfi(void *addr) * ud2 * 1: nop */ - poison_endbr(addr, false); + poison_endbr(addr); poison_hash(addr + fineibt_preamble_hash); break; =20 case CFI_KCFI: /* + * kCFI prefix should start with a valid hash. + */ + if (!decode_preamble_hash(addr)) + break; + + /* * __cfi_\func: * movl $0, %eax * .skip 11, 0x90 From nobody Sun Feb 8 00:49:27 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E84AD1E5B62 for ; Fri, 7 Feb 2025 12:28:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931315; cv=none; b=UcUZm8vJC7WjWMvyAzXW6h90Q8chUcy6S2QF6b4pOyBG4M038H9d3bgng3bFar6OOSaoEDLkNv6aujOH0qAJhP9in6usWVwMlOS+CIDftH2oxJH3Ft2CesFbkECPuKK3vFLMFNzcTAP/OkV9OAra07LrLNnjpYKogc2ZTjVhsLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931315; c=relaxed/simple; bh=Mzij60vAxhAbsXoitUQ2oz9b4b+YA7La0Kw53jhzhIo=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=XqWAh+3UUH01cA0pnyHJtERtsWo1MimhqAhhzByV9BQuglB9iWM+xRk+YuKl1IU3mL5oO0+pv72tuXK7QMfaP0zv0obckaMibCC7EvhLzNwvXdEsJ68LRY23AXd5mBiieZyP29TAomMvn+wvO6KylPLeeZMTbaTv1lEqOMKU4po= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=K43Zd8tB; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="K43Zd8tB" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=GGICIuSmU81I9vgz/fZjNMdH5NJA8Uv1Q7wuMMc0df8=; b=K43Zd8tBC1xXp3teSccvvlfxzO 1/g7gTEVFvcXXvQzQDUWQGxdOAPY5vWTkoW8aIQLTjHnNrgQHZ7LA9faSpSVQfNBkHrCj83POCs5Z TAExYFFE9cTnlzw4ZLGNT0HeSzCu0W2D5LgvjWPGoPXMKwlevKenm872emkP7wnAOk3BvRiDaUqyI wSvjlN6xrjsUunv400bBWoOUpBs1bC4dAM8zwk2wQ6vkMlLsKSpcphyortWl2ad/x+PxeV/GASMC9 3ere4Jpg+QqY7siDUMiFo42ElS4mEmU/4bNW78GZnk+Nk5ElO1+yKDTLSEFrkHG/w4qtL5gf13vIq P3ntUiVw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNTA-0000000HAek-0WHL; Fri, 07 Feb 2025 12:28:29 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 4113D3033E9; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122546.919773202@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:38 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 09/11] x86/early_printk: Harden early_serial References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Scott found that mem32_serial_in() is an ideal speculation gadget, an indirectly callable function that takes an adddress and offset and immediately does a load. Use static_call() to take away the need for indirect calls and explicitly seal the functions to ensure they're not callable on IBT enabled parts. Reported-by: Scott Constable Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/kernel/early_printk.c | 51 ++++++++++++++++++++----------------= ----- 1 file changed, 25 insertions(+), 26 deletions(-) --- a/arch/x86/kernel/early_printk.c +++ b/arch/x86/kernel/early_printk.c @@ -19,6 +19,7 @@ #include #include #include +#include =20 /* Simple VGA output */ #define VGABASE (__ISA_IO_base + 0xb8000) @@ -94,26 +95,28 @@ static unsigned long early_serial_base =3D #define DLL 0 /* Divisor Latch Low */ #define DLH 1 /* Divisor latch High */ =20 -static unsigned int io_serial_in(unsigned long addr, int offset) +static __noendbr unsigned int io_serial_in(unsigned long addr, int offset) { return inb(addr + offset); } +ANNOTATE_NOENDBR_SYM(io_serial_in); =20 -static void io_serial_out(unsigned long addr, int offset, int value) +static __noendbr void io_serial_out(unsigned long addr, int offset, int va= lue) { outb(value, addr + offset); } +ANNOTATE_NOENDBR_SYM(io_serial_out); =20 -static unsigned int (*serial_in)(unsigned long addr, int offset) =3D io_se= rial_in; -static void (*serial_out)(unsigned long addr, int offset, int value) =3D i= o_serial_out; +DEFINE_STATIC_CALL(serial_in, io_serial_in); +DEFINE_STATIC_CALL(serial_out, io_serial_out); =20 static int early_serial_putc(unsigned char ch) { unsigned timeout =3D 0xffff; =20 - while ((serial_in(early_serial_base, LSR) & XMTRDY) =3D=3D 0 && --timeout) + while ((static_call(serial_in)(early_serial_base, LSR) & XMTRDY) =3D=3D 0= && --timeout) cpu_relax(); - serial_out(early_serial_base, TXR, ch); + static_call(serial_out)(early_serial_base, TXR, ch); return timeout ? 0 : -1; } =20 @@ -131,16 +134,16 @@ static __init void early_serial_hw_init( { unsigned char c; =20 - serial_out(early_serial_base, LCR, 0x3); /* 8n1 */ - serial_out(early_serial_base, IER, 0); /* no interrupt */ - serial_out(early_serial_base, FCR, 0); /* no fifo */ - serial_out(early_serial_base, MCR, 0x3); /* DTR + RTS */ - - c =3D serial_in(early_serial_base, LCR); - serial_out(early_serial_base, LCR, c | DLAB); - serial_out(early_serial_base, DLL, divisor & 0xff); - serial_out(early_serial_base, DLH, (divisor >> 8) & 0xff); - serial_out(early_serial_base, LCR, c & ~DLAB); + static_call(serial_out)(early_serial_base, LCR, 0x3); /* 8n1 */ + static_call(serial_out)(early_serial_base, IER, 0); /* no interrupt */ + static_call(serial_out)(early_serial_base, FCR, 0); /* no fifo */ + static_call(serial_out)(early_serial_base, MCR, 0x3); /* DTR + RTS */ + + c =3D static_call(serial_in)(early_serial_base, LCR); + static_call(serial_out)(early_serial_base, LCR, c | DLAB); + static_call(serial_out)(early_serial_base, DLL, divisor & 0xff); + static_call(serial_out)(early_serial_base, DLH, (divisor >> 8) & 0xff); + static_call(serial_out)(early_serial_base, LCR, c & ~DLAB); } =20 #define DEFAULT_BAUD 9600 @@ -183,28 +186,26 @@ static __init void early_serial_init(cha /* Convert from baud to divisor value */ divisor =3D 115200 / baud; =20 - /* These will always be IO based ports */ - serial_in =3D io_serial_in; - serial_out =3D io_serial_out; - /* Set up the HW */ early_serial_hw_init(divisor); } =20 #ifdef CONFIG_PCI -static void mem32_serial_out(unsigned long addr, int offset, int value) +static __noendbr void mem32_serial_out(unsigned long addr, int offset, int= value) { u32 __iomem *vaddr =3D (u32 __iomem *)addr; /* shift implied by pointer type */ writel(value, vaddr + offset); } +ANNOTATE_NOENDBR_SYM(mem32_serial_out); =20 -static unsigned int mem32_serial_in(unsigned long addr, int offset) +static __noendbr unsigned int mem32_serial_in(unsigned long addr, int offs= et) { u32 __iomem *vaddr =3D (u32 __iomem *)addr; /* shift implied by pointer type */ return readl(vaddr + offset); } +ANNOTATE_NOENDBR_SYM(mem32_serial_in); =20 /* * early_pci_serial_init() @@ -278,15 +279,13 @@ static __init void early_pci_serial_init */ if ((bar0 & PCI_BASE_ADDRESS_SPACE) =3D=3D PCI_BASE_ADDRESS_SPACE_IO) { /* it is IO mapped */ - serial_in =3D io_serial_in; - serial_out =3D io_serial_out; early_serial_base =3D bar0 & PCI_BASE_ADDRESS_IO_MASK; write_pci_config(bus, slot, func, PCI_COMMAND, cmdreg|PCI_COMMAND_IO); } else { /* It is memory mapped - assume 32-bit alignment */ - serial_in =3D mem32_serial_in; - serial_out =3D mem32_serial_out; + static_call_update(serial_in, mem32_serial_in); + static_call_update(serial_out, mem32_serial_out); /* WARNING! assuming the address is always in the first 4G */ early_serial_base =3D (unsigned long)early_ioremap(bar0 & PCI_BASE_ADDRESS_MEM_MASK, 0x10); From nobody Sun Feb 8 00:49:27 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 631411DFE04 for ; Fri, 7 Feb 2025 12:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931313; cv=none; b=J1Zo2VVd9pFoVvQZgdlx2IuzBJ97TpiJ1IVYRW8gITJQ6sVEc2L1IrMeVpam3xsTJ4VmYjKQUSusIy0HxltdrB+czT08YzvJHA+FQGtVPUPAgnFe9IZBN+TtaOQKhfnj6cOPuNRFZ+y4C4AcxDBxpwitsCKnCBt3MdSdod1+XXw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931313; c=relaxed/simple; bh=XvNpgsUYXXKn3GS7s8clOJ4znqcbSSqGxW1VJsuwa8o=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=q1NDf79sDA43g2y/gJ9nctJIO1PfjbCoNyV5QoBnyWfx2fZ4nqMGQY3M5EJkDYb/ZHmG72iHUkPbhAFJabblxzgP+N4CVEXTJ5SkDvBcL4Ku+hdGL7rBYysSyxt9asGuWrisxebMC0lHQxrLM/Oh9SpgqGawjUoJb2zTPab2DGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=X0dlSdZ7; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="X0dlSdZ7" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=qQYv110ck7jBvbbLipQh+0XWA1ymIh2E1606AKuN1Ic=; b=X0dlSdZ77pTEGXc7j74IOYrCTO uwR/ZAOn3j59pBlPq+Wuykts8VShz7ZtymM/e3xWI1YmbPUkUdNAyGD2KemZRf+IPQdwkpzCPbymE rYmblMaevin3QyQmywb/M71ffDWFcRJWihf8+pm6ixCWmjDtpaG7rJWeO4+f93RCPUyO6jb4g+v78 532RtcalDO088xvoE7pxExIW3EM0hvtOJrkKaQ6yE1J3KmyU6C+gibP5tL8sx2GgFpX7ttZVJ8rAd yfMes32HlN//1JpHSoTSMd3Sg26sCpOk6W8iUc97MfqWcWe1UfhUi9riwZbuVmgZp6RxI84DJjbtf jXIShYlw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNTA-00000007nBw-0W3v; Fri, 07 Feb 2025 12:28:28 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 451F930615B; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122547.022463564@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:39 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 10/11] x86: BHI stubs References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- arch/x86/include/asm/cfi.h | 3 + arch/x86/lib/Makefile | 3 - arch/x86/lib/bhi.S | 130 ++++++++++++++++++++++++++++++++++++++++= +++++ 3 files changed, 135 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -101,6 +101,9 @@ enum cfi_mode { =20 extern enum cfi_mode cfi_mode; =20 +typedef u8 bhi_thunk[32]; +extern bhi_thunk __bhi_args[]; + struct pt_regs; =20 #ifdef CONFIG_CFI_CLANG --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -66,5 +66,6 @@ endif lib-y +=3D clear_page_64.o copy_page_64.o lib-y +=3D memmove_64.o memset_64.o lib-y +=3D copy_user_64.o copy_user_uncached_64.o - lib-y +=3D cmpxchg16b_emu.o + lib-y +=3D cmpxchg16b_emu.o + lib-y +=3D bhi.o endif --- /dev/null +++ b/arch/x86/lib/bhi.S @@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include + +/* + * Notably, the FineIBT preamble calling these will have ZF set and r10 ze= ro. + * + * The very last element is in fact larger than 32 bytes, but since its the + * last element, this does not matter, + */ + +.pushsection .noinstr.text, "ax" + + .align 32 +SYM_CODE_START(__bhi_args) + + .align 32 +SYM_INNER_LABEL(__bhi_args_0, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_1, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: cmovne %r10, %rdi + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_2, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: cmovne %r10, %rdi + cmovne %r10, %rsi + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_3, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_4, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_5, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_6, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_7, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jz 1f + ud2 +1: cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + cmovne %r10, %rsp + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_CODE_END(__bhi_args) + +.popsection From nobody Sun Feb 8 00:49:27 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 314411E102A for ; Fri, 7 Feb 2025 12:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931314; cv=none; b=mBbDJlsPzh+bAllvOKfxA28o3+ybape8sYmszucUi7iejuvm/cfOUFSWJSt/Dn8hMmidcmrrMePHdxSArDkN0ekdQvehAJ+yL43GoOUj5CY7sojeNMgd3GW8H8HQcI9XIHGcIbZvUHsY7YYV+GodgrpeMfpXDD2JjQ9ugn4z5D8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738931314; c=relaxed/simple; bh=1yhsFTAJC0fyLfAlQ7ZXicMfYzFRyVFlwOSBTX9grv4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=bFfnHFLIaON35XKIXGhB8hLADTzPdOCGNmpSDGL9imnBKEEx6HgqLuAaN0hzUAbPyCWDVi0krAf/VJMYS+2IVIX1tj3XOO5KqlKTHn7TT+5Rt41jn0juVdkXKTiJrr+WaCAoGQhLY7RtyxGYIzneaq+2Daksj0BjAr8xlo4IfL0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=NgHZNCmp; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="NgHZNCmp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=L+COfsvQhb7CX9P6B6i2gM9H/qGrRRtoSf00jF2EqEI=; b=NgHZNCmpKTbr5/IWPtwlj1MSl6 jObMLasIj8dwRPBNZxwS6Foc8uao49b2oJb0ucdUhorePEcNeiQSwCfsHFE792Y2hrjzYnoL5M8++ t5vtsVFCNuugvpwUw+eNYsEeCICJG9hXD2C2aONBJuxM29tJC7X+lJuFC6qPZorWCXYVfkSq0CZni nuNpIyGxN/BDVmj0sHHt/MDDVMGOqe4gxXslM3yM69/X5eaN6ajVP/wMmazr1iSl+IfO1eKNys4du M+V7VcRhST1CUgN37hZIHHuFRGYiNja/UUtbT8imADM/lTuBIh9eCvVrAmsmU/SfrlGXRZZ/AtLSu XQtFHiMA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tgNTA-00000007nBy-2kTv; Fri, 07 Feb 2025 12:28:28 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 499A530619B; Fri, 7 Feb 2025 13:28:25 +0100 (CET) Message-ID: <20250207122547.124276834@infradead.org> User-Agent: quilt/0.66 Date: Fri, 07 Feb 2025 13:15:40 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org Subject: [PATCH 11/11] x86/fineibt: Add FineIBT+BHI mitigation References: <20250207121529.222723073@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Due to FineIBT weakness, add an additional mitigation for BHI. Relies on clang-cfi to emit an additional piece of magic in the kCFI pre-amble: https://github.com/llvm/llvm-project/commit/e223485c9b38a5579991b8cebb6a2= 00153eee245 Which encodes the number of argument registers in the target register: movl 0x12345678, %reg // CFI hash XXX get Scott to write a blurb about how this works to mitigate BHI FineIBT FineIBT+BHI __cfi_foo: __cfi_foo: endbr64 endbr64 subl $0x12345678, %r10d subl $0x12345678, %r10d jz 1f call __bhi_args[\reg] ud2 1: nop foo: foo: osp nop3 osp nop3 ... ... Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sami Tolvanen Reviewed-by: Sami Tolvanen --- Makefile | 3 ++ arch/x86/Kconfig | 8 +++++ arch/x86/include/asm/cfi.h | 9 +++++- arch/x86/kernel/alternative.c | 60 +++++++++++++++++++++++++++++++++++--= ----- arch/x86/net/bpf_jit_comp.c | 32 +++++++++++++++------- kernel/bpf/core.c | 4 +- 6 files changed, 94 insertions(+), 22 deletions(-) --- a/Makefile +++ b/Makefile @@ -1014,6 +1014,9 @@ CC_FLAGS_CFI :=3D -fsanitize=3Dkcfi ifdef CONFIG_CFI_ICALL_NORMALIZE_INTEGERS CC_FLAGS_CFI +=3D -fsanitize-cfi-icall-experimental-normalize-integers endif +ifdef CONFIG_FINEIBT_BHI + CC_FLAGS_CFI +=3D -fsanitize-kcfi-arity +endif ifdef CONFIG_RUST # Always pass -Zsanitizer-cfi-normalize-integers as CONFIG_RUST selects # CONFIG_CFI_ICALL_NORMALIZE_INTEGERS. --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2473,6 +2473,10 @@ config CC_HAS_RETURN_THUNK config CC_HAS_ENTRY_PADDING def_bool $(cc-option,-fpatchable-function-entry=3D16,16) =20 +config CC_HAS_KCFI_ARITY + def_bool $(cc-option,-fsanitize=3Dkcfi -fsanitize-kcfi-arity) + depends on CC_IS_CLANG && !RUST + config FUNCTION_PADDING_CFI int default 59 if FUNCTION_ALIGNMENT_64B @@ -2498,6 +2502,10 @@ config FINEIBT depends on X86_KERNEL_IBT && CFI_CLANG && MITIGATION_RETPOLINE select CALL_PADDING =20 +config FINEIBT_BHI + def_bool y + depends on FINEIBT && CC_HAS_KCFI_ARITY + config HAVE_CALL_THUNKS def_bool y depends on CC_HAS_ENTRY_PADDING && MITIGATION_RETHUNK && OBJTOOL --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -72,7 +72,7 @@ * __cfi_foo: * endbr64 * subl 0x12345678, %r10d - * jz foo + * jz foo # call __bhi_args[\reg] when CONFIG_FINEIBT_BHI * ud2 * nop * foo: @@ -97,6 +97,7 @@ enum cfi_mode { CFI_OFF, /* Taditional / IBT depending on .config */ CFI_KCFI, /* Optionally CALL_PADDING, IBT, RETPOLINE */ CFI_FINEIBT, /* see arch/x86/kernel/alternative.c */ + CFI_FINEIBT_BHI, }; =20 extern enum cfi_mode cfi_mode; @@ -116,6 +117,7 @@ static inline int cfi_get_offset(void) { switch (cfi_mode) { case CFI_FINEIBT: + case CFI_FINEIBT_BHI: return 16; case CFI_KCFI: if (IS_ENABLED(CONFIG_CALL_PADDING)) @@ -128,6 +130,7 @@ static inline int cfi_get_offset(void) #define cfi_get_offset cfi_get_offset =20 extern u32 cfi_get_func_hash(void *func); +extern int cfi_get_func_arity(void *func); =20 #else static inline enum bug_trap_type handle_cfi_failure(struct pt_regs *regs) @@ -140,6 +143,10 @@ static inline u32 cfi_get_func_hash(void { return 0; } +static inline int cfi_get_func_arity(void *func) +{ + return 0; +} #endif /* CONFIG_CFI_CLANG */ =20 #if HAS_KERNEL_IBT =3D=3D 1 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -962,6 +962,7 @@ u32 cfi_get_func_hash(void *func) func -=3D cfi_get_offset(); switch (cfi_mode) { case CFI_FINEIBT: + case CFI_FINEIBT_BHI: func +=3D 7; break; case CFI_KCFI: @@ -976,6 +977,21 @@ u32 cfi_get_func_hash(void *func) =20 return hash; } + +int cfi_get_func_arity(void *func) +{ + bhi_thunk *target; + s32 disp; + + if (cfi_mode !=3D CFI_FINEIBT_BHI) + return 0; + + if (get_kernel_nofault(disp, func-4)) + return 0; + + target =3D func + disp; + return target - __bhi_args; +} #endif =20 #ifdef CONFIG_FINEIBT @@ -1020,6 +1036,8 @@ static __init int cfi_parse_cmdline(char cfi_mode =3D CFI_KCFI; } else if (!strcmp(str, "fineibt")) { cfi_mode =3D CFI_FINEIBT; + } else if (IS_ENABLED(CONFIG_FINEIBT_BHI) && !strcmp(str, "fineibt+bhi")= ) { + cfi_mode =3D CFI_FINEIBT_BHI; } else if (!strcmp(str, "norand")) { cfi_rand =3D false; } else { @@ -1064,6 +1082,7 @@ asm( ".pushsection .rodata \n" "fineibt_preamble_start: \n" " endbr64 \n" " subl $0x12345678, %r10d \n" + "fineibt_preamble_bhi_call: \n" " je fineibt_preamble_end \n" " ud2 \n" " nop \n" @@ -1072,10 +1091,12 @@ asm( ".pushsection .rodata \n" ); =20 extern u8 fineibt_preamble_start[]; +extern u8 fineibt_preamble_bhi_call[]; extern u8 fineibt_preamble_end[]; =20 #define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_sta= rt) #define fineibt_preamble_hash 7 +#define fineibt_preamble_bhi (fineibt_preamble_bhi_call - fineibt_preambl= e_start) =20 asm( ".pushsection .rodata \n" "fineibt_caller_start: \n" @@ -1094,13 +1115,16 @@ extern u8 fineibt_caller_end[]; =20 #define fineibt_caller_jmp (fineibt_caller_size - 2) =20 -static u32 decode_preamble_hash(void *addr) +static u32 decode_preamble_hash(void *addr, int *reg) { u8 *p =3D addr; =20 - /* b8 78 56 34 12 mov $0x12345678,%eax */ - if (p[0] =3D=3D 0xb8) + /* b8+reg 78 56 34 12 movl $0x12345678,\reg */ + if (p[0] >=3D 0xb8 && p[0] < 0xc0) { + if (reg) + *reg =3D p[0] - 0xb8; return *(u32 *)(addr + 1); + } =20 return 0; /* invalid hash value */ } @@ -1109,7 +1133,7 @@ static u32 decode_caller_hash(void *addr { u8 *p =3D addr; =20 - /* 41 ba 78 56 34 12 mov $0x12345678,%r10d */ + /* 41 ba 78 56 34 12 movl $(-0x12345678),%r10d */ if (p[0] =3D=3D 0x41 && p[1] =3D=3D 0xba) return -*(u32 *)(addr + 2); =20 @@ -1178,7 +1202,7 @@ static int cfi_rand_preamble(s32 *start, void *addr =3D (void *)s + *s; u32 hash; =20 - hash =3D decode_preamble_hash(addr); + hash =3D decode_preamble_hash(addr, NULL); if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", addr, addr, 5, addr)) return -EINVAL; @@ -1197,6 +1221,7 @@ static int cfi_rewrite_preamble(s32 *sta for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; u32 hash; + int reg; =20 /* * When the function doesn't start with ENDBR the compiler will @@ -1206,7 +1231,7 @@ static int cfi_rewrite_preamble(s32 *sta if (!is_endbr(addr + 16)) continue; =20 - hash =3D decode_preamble_hash(addr); + hash =3D decode_preamble_hash(addr, ®); if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", addr, addr, 5, addr)) return -EINVAL; @@ -1214,6 +1239,19 @@ static int cfi_rewrite_preamble(s32 *sta text_poke_early(addr, fineibt_preamble_start, fineibt_preamble_size); WARN_ON(*(u32 *)(addr + fineibt_preamble_hash) !=3D 0x12345678); text_poke_early(addr + fineibt_preamble_hash, &hash, 4); + + WARN_ONCE(!IS_ENABLED(CONFIG_FINEIBT_BHI) && reg, + "kCFI preamble has wrong register at: %pS %px %*ph\n", + addr, addr, 5, addr); + + if (cfi_mode !=3D CFI_FINEIBT_BHI || !reg) + continue; + + text_poke_early(addr + fineibt_preamble_bhi, + text_gen_insn(CALL_INSN_OPCODE, + addr + fineibt_preamble_bhi, + __bhi_args[reg]), + CALL_INSN_SIZE); } =20 return 0; @@ -1330,6 +1368,7 @@ static void __apply_fineibt(s32 *start_r return; =20 case CFI_FINEIBT: + case CFI_FINEIBT_BHI: /* place the FineIBT preamble at func()-16 */ ret =3D cfi_rewrite_preamble(start_cfi, end_cfi); if (ret) @@ -1343,8 +1382,10 @@ static void __apply_fineibt(s32 *start_r /* now that nobody targets func()+0, remove ENDBR there */ cfi_rewrite_endbr(start_cfi, end_cfi); =20 - if (builtin) - pr_info("Using FineIBT CFI\n"); + if (builtin) { + pr_info("Using FineIBT%s CFI\n", + cfi_mode =3D=3D CFI_FINEIBT_BHI ? "+BHI" : ""); + } return; =20 default: @@ -1372,6 +1413,7 @@ static void poison_cfi(void *addr) */ switch (cfi_mode) { case CFI_FINEIBT: + case CFI_FINEIBT_BHI: /* * FineIBT prefix should start with an ENDBR. */ @@ -1394,7 +1436,7 @@ static void poison_cfi(void *addr) /* * kCFI prefix should start with a valid hash. */ - if (!decode_preamble_hash(addr)) + if (!decode_preamble_hash(addr, NULL)) break; =20 /* --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -410,16 +410,21 @@ static void emit_nops(u8 **pprog, int le * Emit the various CFI preambles, see asm/cfi.h and the comments about Fi= neIBT * in arch/x86/kernel/alternative.c */ +static int emit_call(u8 **pprog, void *func, void *ip); =20 -static void emit_fineibt(u8 **pprog, u32 hash) +static void emit_fineibt(u8 **pprog, u8 *ip, u32 hash, int reg) { u8 *prog =3D *pprog; =20 EMIT_ENDBR(); EMIT3_off32(0x41, 0x81, 0xea, hash); /* subl $hash, %r10d */ - EMIT2(0x74, 0x07); /* jz.d8 +7 */ - EMIT2(0x0f, 0x0b); /* ud2 */ - EMIT1(0x90); /* nop */ + if (cfi_mode =3D=3D CFI_FINEIBT_BHI && reg) { + emit_call(&prog, __bhi_args[reg], ip); + } else { + EMIT2(0x74, 0x02); /* jz.d8 +2 */ + EMIT2(0x0f, 0x0b); /* ud2 */ + EMIT1(0x90); /* nop */ + } EMIT_ENDBR_POISON(); =20 *pprog =3D prog; @@ -448,13 +453,14 @@ static void emit_kcfi(u8 **pprog, u32 ha *pprog =3D prog; } =20 -static void emit_cfi(u8 **pprog, u32 hash) +static void emit_cfi(u8 **pprog, u8 *ip, u32 hash, int reg) { u8 *prog =3D *pprog; =20 switch (cfi_mode) { case CFI_FINEIBT: - emit_fineibt(&prog, hash); + case CFI_FINEIBT_BHI: + emit_fineibt(&prog, ip, hash, reg); break; =20 case CFI_KCFI: @@ -505,13 +511,17 @@ static void emit_prologue_tail_call(u8 * * bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes * while jumping to another program */ -static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf, +static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_f= rom_cbpf, bool tail_call_reachable, bool is_subprog, bool is_exception_cb) { u8 *prog =3D *pprog; =20 - emit_cfi(&prog, is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash); + if (is_subprog) { + emit_cfi(&prog, ip, cfi_bpf_subprog_hash, 5); + } else { + emit_cfi(&prog, ip, cfi_bpf_hash, 1); + } /* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ @@ -1480,7 +1490,7 @@ static int do_jit(struct bpf_prog *bpf_p =20 detect_reg_usage(insn, insn_cnt, callee_regs_used); =20 - emit_prologue(&prog, stack_depth, + emit_prologue(&prog, image, stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb); /* Exception callback will clobber callee regs for its own use, and @@ -3047,7 +3057,9 @@ static int __arch_prepare_bpf_trampoline /* * Indirect call for bpf_struct_ops */ - emit_cfi(&prog, cfi_get_func_hash(func_addr)); + emit_cfi(&prog, image, + cfi_get_func_hash(func_addr), + cfi_get_func_arity(func_addr)); } else { /* * Direct-call fentry stub, as such it needs accounting for the --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -707,7 +707,7 @@ void bpf_prog_kallsyms_add(struct bpf_pr * When FineIBT, code in the __cfi_foo() symbols can get executed * and hence unwinder needs help. */ - if (cfi_mode !=3D CFI_FINEIBT) + if (cfi_mode !=3D CFI_FINEIBT && cfi_mode !=3D CFI_FINEIBT_BHI) return; =20 snprintf(fp->aux->ksym_prefix.name, KSYM_NAME_LEN, @@ -727,7 +727,7 @@ void bpf_prog_kallsyms_del(struct bpf_pr =20 bpf_ksym_del(&fp->aux->ksym); #ifdef CONFIG_FINEIBT - if (cfi_mode !=3D CFI_FINEIBT) + if (cfi_mode !=3D CFI_FINEIBT && cfi_mode !=3D CFI_FINEIBT_BHI) return; bpf_ksym_del(&fp->aux->ksym_prefix); #endif