From nobody Wed Dec 17 07:17:03 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50AFC2192EB for ; Mon, 24 Feb 2025 12:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=mdeO3/iRw4ySqGPydoFizFl13VSEiEoWmJN0CGin3F40tlRaJzd/gApuXwWkW0W4kuEfCoJPhF5BEYjglKVBXyRVepobC/caf2fBbZ3qtcqEejEbjHL/AsJAU2f25LH1mlqV1ran0Sj8KBO1oKK0TNLCfiE8xqauPoEpR+i8Vk8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=Cf8HCTGmOx/qRDAOIVvByhVe67UrJTu9m54l6Wmgn8c=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=c21MDLR0B2a16Gp1QwE+BWRrGtOl69UX9i6bYXH7hSqXQ5Y34CUZ4PqLDdxI7sDO3TTmKRPRqDlWSUcpic35U9rSaTHcbZBRFeHwmHIdytL6KshzeBWw4vdlsBFdfBE4X+7Rz6JwgZIbO7NQ0HmAlfU4hFeZBw1/IynHQSg0X14= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=BMo6Ql1P; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="BMo6Ql1P" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=EhQ12QSerGcZU41eLzZyyGD3alpvSewb0+Htu+1LdoA=; b=BMo6Ql1P1+eSr7GzOloLqd79f+ Mk/REyJk/eAdyQDCYwNhTR18crPfKbkVlyon2879cFiTdCw1KlqGP6B7ljnOIG0Gtz1iVTH9diwNu GnZIeO97DGzysIQqNFJi+8cHKzjOdJENPAVBxXvC+ujP2DulYi+IVqVWEFfBeYM7iKaohMZtjLdeN AkF8cl8Obovhj/JSXKQY3eqGmfI8gDrc9vtdAlcrT4d66W3JHQADx7p6ZqBrTQ3EaCEFbDvhNWOjN Y/jD1WXoGVBjWXaw8jJ+IQxSIR7NdNIf7E5yTrfLqSRfAWrjojM141MAFOYEPbP/gq8etAyqWUomM 7hMSpykw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqM-00000003C2P-2CgY; Mon, 24 Feb 2025 12:45:54 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 14D3D30078F; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124159.924496481@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:04 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 01/10] x86/cfi: Add warn option References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rebuilding with CFI_PERMISSIVE toggled is such a pain, esp. since clang is so slow. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/kernel/alternative.c | 3 +++ arch/x86/kernel/cfi.c | 8 ++++---- include/linux/cfi.h | 2 ++ kernel/cfi.c | 4 +++- 4 files changed, 12 insertions(+), 5 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1022,6 +1022,9 @@ static __init int cfi_parse_cmdline(char cfi_mode =3D CFI_FINEIBT; } else if (!strcmp(str, "norand")) { cfi_rand =3D false; + } else if (!strcmp(str, "warn")) { + pr_alert("CFI mismatch non-fatal!\n"); + cfi_warn =3D true; } else { pr_err("Ignoring unknown cfi option (%s).", str); } --- a/include/linux/cfi.h +++ b/include/linux/cfi.h @@ -11,6 +11,8 @@ #include #include =20 +extern bool cfi_warn; + #ifndef cfi_get_offset static inline int cfi_get_offset(void) { --- a/kernel/cfi.c +++ b/kernel/cfi.c @@ -7,6 +7,8 @@ =20 #include =20 +bool cfi_warn __ro_after_init =3D IS_ENABLED(CONFIG_CFI_PERMISSIVE); + enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long = addr, unsigned long *target, u32 type) { @@ -17,7 +19,7 @@ enum bug_trap_type report_cfi_failure(st pr_err("CFI failure at %pS (no target information)\n", (void *)addr); =20 - if (IS_ENABLED(CONFIG_CFI_PERMISSIVE)) { + if (cfi_warn) { __warn(NULL, 0, (void *)addr, 0, regs, NULL); return BUG_TRAP_TYPE_WARN; } From nobody Wed Dec 17 07:17:03 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50D6423372A for ; Mon, 24 Feb 2025 12:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=JdDxTD/ymycn8eqr80joiAvWtgbl62DwZnZ7c03sUfxNQOrRL36t04AF60PI41louZm5i8AFzOlYON4tZt4W83XPtOfVT4JfW9dM0mAYycaJIEyExvKQ48Avapv6PCJVZ2EItMxj8IvlQShyNPYc7/GBfq+hl0BJR3DsL9zqkec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=qiHWEsUFhyrAhvzR/3W2KqNPPoF05MG34ACD4hT96GM=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=UJ6ffqP55UoZRBNSRXfdXqOe+sksGQ4UJ/1FFhIvFaGrVR2NGTWvb/YVjPiPKet8leg1+8uHXBkf2eC2vUevXNtK7rQ8mYWRQ7DKDG4SGbJULVn+SnV4bHF5bG6OcxVt4a768ZHOAVodyqpsdAdweJt0/Qn66Rx2GdOt9Ebof+k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=P9azl+wq; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="P9azl+wq" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=XBH0IHGZUCGG4OKLWJTsoSfxC5AGMSYb5ZmdUnOdGdE=; b=P9azl+wqq1NuEIBo/OoXMGmsgD SED9BZrXD6cuLTFlqCokO/1VK82YPNcf9wyV+fTFzm/ZC8LIgYSSIPPxC6rCYSLrYUtsF4pP0mE25 eWM5vRAEGoCIBzNXnQxli8bHDBadAUf9L/MSc+Rj+1JHRnZXW0MUZFOvvuNZxfsln0AltCgk6v5jC AnKr7Nl6Y39TdU1YO1i9/vJeNRpD2XknSZDWlkwNFMiaE6fR//JUMF6sA7SadHmj/IrHUfptsUjB2 glnGf7FPruIUY4L3mffOg07l/eZfbho2ydorJhjsF14ORM/4RuTr3oU531emFueo75JqOBf8ZXvXo 38R9zcKg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqM-000000079lb-2IxM; Mon, 24 Feb 2025 12:45:54 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 18E6C3007F0; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.059556588@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:05 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 02/10] x86/ibt: Add exact_endbr() helper References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For when we want to exactly match ENDBR, and not everything that we can scribble it with. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/kernel/alternative.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -863,6 +863,17 @@ __noendbr bool is_endbr(u32 *val) return false; } =20 +static __noendbr bool exact_endbr(u32 *val) +{ + u32 endbr; + + __get_kernel_nofault(&endbr, val, u32, Efault); + return endbr =3D=3D gen_endbr(); + +Efault: + return false; +} + static void poison_cfi(void *addr); =20 static void __init_or_module poison_endbr(void *addr) @@ -1427,10 +1438,9 @@ static void poison_cfi(void *addr) bool decode_fineibt_insn(struct pt_regs *regs, unsigned long *target, u32 = *type) { unsigned long addr =3D regs->ip - fineibt_preamble_ud2; - u32 endbr, hash; + u32 hash; =20 - __get_kernel_nofault(&endbr, addr, u32, Efault); - if (endbr !=3D gen_endbr()) + if (!exact_endbr((void *)addr)) return false; =20 *target =3D addr + fineibt_preamble_size; From nobody Wed Dec 17 07:17:03 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E95EF23372C for ; Mon, 24 Feb 2025 12:46:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=TTHnbENi8E+NSOGOYVuRp9CBTY11gUMfUtXkG3WtLrOFXl83T1IqKd78t3TqOX+hIPteTInatMYMtbdxpyU7fHjZ/CQ3f/msjxIOeij0zro9ipqvpq+sfB5fce4vDtkGXv3zMybwwNgJu16/EIRJEwgVtfgMc9kUkbyiDzTPnlE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=rY8hfjQ46+vIUuOFxc40CbgFaBbrCmchkZaVaf0CnzQ=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Lof4HVSSQTl7zN0o2D8d3li7X+Q0jZ1B53YWsPcOfUAE9Cfa9HthEugCZ7aZIa7d/J7cNhHcKRtKaeLID0vrbfHLTy5vcloic4bBcz3NwV0KP2be0F7JOmg3qhfOpU61tBTSJxcWGWQ9UPS6UEPgyuMv6UQ8JCaQu3hnkOZgf3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=a6OJvkUb; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="a6OJvkUb" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=PBme8ixFppy5L5nmk2W/qkNAxpMYSIP56Kuzq+VvDQM=; b=a6OJvkUbU99jcYrVckVv4UjDvF qR5US1ipYBEkIxBbUtWwR9oJGWVfpYlbtlI2eXizJEgW257yNPh0YarMqHfhDXpH6eVk+BMwDwqQs yj67AIY6eZhv4EqUhJmjIe5fZBAlf9NmI3dHJVUO+stXkiLL5o6zZXtDZTBVnmJt8yy2YANZM4o+M Uxyq0a4oV4cxc4rt2Ga7PNi1b9TdUMXi9JdcjNe7tJ3VgA1m9aMT5S5oyLpyy1j/F0/fA3ldF6l1I QsUDNXbqDbpmghR/pXIPnDsLrFdU126FXWNnDebtBdUkFdRXa8Iih9dmWKsoKiOiO+lMIdnz3egrU c5O+wNiA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqM-000000079le-2FmP; Mon, 24 Feb 2025 12:45:54 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 1C9CA300898; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.166774696@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:06 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 03/10] x86/traps: Decode 0xEA #UD References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" FineIBT will start using 0xEA as #UD Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/include/asm/bug.h | 1 + arch/x86/kernel/traps.c | 20 +++++++++++++++++--- 2 files changed, 18 insertions(+), 3 deletions(-) --- a/arch/x86/include/asm/bug.h +++ b/arch/x86/include/asm/bug.h @@ -25,6 +25,7 @@ #define BUG_UD2 0xfffe #define BUG_UD1 0xfffd #define BUG_UD1_UBSAN 0xfffc +#define BUG_EA 0xffea =20 #ifdef CONFIG_GENERIC_BUG =20 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -96,6 +96,7 @@ __always_inline int is_valid_bugaddr(uns * Check for UD1 or UD2, accounting for Address Size Override Prefixes. * If it's a UD1, further decode to determine its use: * + * FineIBT: ea (bad) * UBSan{0}: 67 0f b9 00 ud1 (%eax),%eax * UBSan{10}: 67 0f b9 40 10 ud1 0x10(%eax),%eax * static_call: 0f b9 cc ud1 %esp,%ecx @@ -113,6 +114,10 @@ __always_inline int decode_bug(unsigned v =3D *(u8 *)(addr++); if (v =3D=3D INSN_ASOP) v =3D *(u8 *)(addr++); + if (v =3D=3D 0xea) { + *len =3D addr - start; + return BUG_EA; + } if (v !=3D OPCODE_ESCAPE) return BUG_NONE; =20 @@ -309,10 +314,16 @@ static noinstr bool handle_bug(struct pt =20 switch (ud_type) { case BUG_UD2: - if (report_bug(regs->ip, regs) =3D=3D BUG_TRAP_TYPE_WARN || - handle_cfi_failure(regs) =3D=3D BUG_TRAP_TYPE_WARN) { - regs->ip +=3D ud_len; + if (report_bug(regs->ip, regs) =3D=3D BUG_TRAP_TYPE_WARN) { + handled =3D true; + break; + } + fallthrough; + + case BUG_EA: + if (handle_cfi_failure(regs) =3D=3D BUG_TRAP_TYPE_WARN) { handled =3D true; + break; } break; =20 @@ -328,6 +339,9 @@ static noinstr bool handle_bug(struct pt break; } =20 + if (handled) + regs->ip +=3D ud_len; + if (regs->flags & X86_EFLAGS_IF) raw_local_irq_disable(); instrumentation_end(); From nobody Wed Dec 17 07:17:03 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50C5B221F0B for ; Mon, 24 Feb 2025 12:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=BrFfIHQHqE1diXzansX6/eT+klIAiW0AM9u+tXgiUuiuANZUpyw+hrCWNHU8GPoO2oqdhJ9TU67mzC5p/poKhmjtVAmMZWO+4snTYdOQb8RD9jEwcwizjlFLyHNmMOgp6ii2oEdtYf9G4sjzV+1lbumU7SRV625XIekSO61FEWA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=QZcp1ab1WZIM4jKs4x15fULENe5ASYKqrGQXPEXTUy0=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=OQz8DuCca+5M6yCYfz43V5mzHoh4ordZcFhXUAnoOk8Yy+uIeJgD1y/ln7OIYh14fAekt+6yH5QqB2jrQEESF8xkNJYefR8oFtsMb/W1KF78UO38GXfk1AqgNVixPy5lROqKYFAFUwkZN88859RjSrSAGHh71V5o3HjiNSksK8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=RIOgrBco; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="RIOgrBco" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=oXBu27fwzftsCfrJsqq+dp8YXeAeYXuO4u/fE4qHiAE=; b=RIOgrBcofBoKWOxnWmkjFg7LYx bG6XFvnFkKWwWoiGLEizBfF9rEsOPPxwnJmpGmhuhi0kRlvszqz/rtOXH7TYOkXNWKodqzOIXxfMO SsE/irVcy/cFTH1jMIfJXl9vWB36/eWPWpVhS61uA7nNQXVzJhfBJGeB4jyw4Hw+a9FNG813uDTa+ df/1vBcZ+sfBzWNSBgtx0U1aY6PwwWduuFtNjGbOi2gkdxkhGy32B1zAMV10Cj1/BmJYELXMBqsBm 3H/Kq3ASxdyRS8OqOg5TECYhR2Igu8MuNE28WpUfq5nNP1iglabbel3PrgAEEVe/7j8CQpS4HaAKJ /G36jZ2w==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqM-00000003C2Q-2EIT; Mon, 24 Feb 2025 12:45:54 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 20B923011E4; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.275223080@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:07 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 04/10] x86/traps: Allow custom fixups in handle_bug() References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The normal fixup in handle_bug() is simply continuing at the next instruction. However upcomming patches make this the wrong thing, so allow handlers (specifically handle_cfi_failure()) to over-ride regs->ip. The callchain is such that the fixup needs to be done before it is determined if the exception is fatal, as such, revert any changes in that case. Additinoally, have handle_cfi_failure() remember the regs->ip value it starts with for reporting. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/kernel/cfi.c | 8 ++++---- arch/x86/kernel/traps.c | 16 +++++++++++++--- 2 files changed, 17 insertions(+), 7 deletions(-) --- a/arch/x86/kernel/cfi.c +++ b/arch/x86/kernel/cfi.c @@ -67,16 +67,16 @@ static bool decode_cfi_insn(struct pt_re */ enum bug_trap_type handle_cfi_failure(struct pt_regs *regs) { - unsigned long target; + unsigned long target, addr =3D regs->ip; u32 type; =20 switch (cfi_mode) { case CFI_KCFI: - if (!is_cfi_trap(regs->ip)) + if (!is_cfi_trap(addr)) return BUG_TRAP_TYPE_NONE; =20 if (!decode_cfi_insn(regs, &target, &type)) - return report_cfi_failure_noaddr(regs, regs->ip); + return report_cfi_failure_noaddr(regs, addr); =20 break; =20 @@ -90,7 +90,7 @@ enum bug_trap_type handle_cfi_failure(st return BUG_TRAP_TYPE_NONE; } =20 - return report_cfi_failure(regs, regs->ip, &target, type); + return report_cfi_failure(regs, addr, &target, type); } =20 /* --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -287,11 +287,12 @@ static inline void handle_invalid_op(str =20 static noinstr bool handle_bug(struct pt_regs *regs) { + unsigned long addr =3D regs->ip; bool handled =3D false; int ud_type, ud_len; s32 ud_imm; =20 - ud_type =3D decode_bug(regs->ip, &ud_imm, &ud_len); + ud_type =3D decode_bug(addr, &ud_imm, &ud_len); if (ud_type =3D=3D BUG_NONE) return handled; =20 @@ -339,8 +340,17 @@ static noinstr bool handle_bug(struct pt break; } =20 - if (handled) - regs->ip +=3D ud_len; + /* + * When continuing, and regs->ip hasn't changed, move it to the next + * instruction. When not continuing execution, restore the instruction + * pointer. + */ + if (handled) { + if (regs->ip =3D=3D addr) + regs->ip +=3D ud_len; + } else { + regs->ip =3D addr; + } =20 if (regs->flags & X86_EFLAGS_IF) raw_local_irq_disable(); From nobody Wed Dec 17 07:17:03 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0318146A63 for ; Mon, 24 Feb 2025 12:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401162; cv=none; b=KDKd7sL1uXRfJxJeljivlGdDHKkpuSNCDQnRc8I4i0N6Lwl+ta7IY4zvKLYtOb0cF5LGcIESue/qPp0j11N+4sL4HELgz9QhWBUEMLUSebeQGrbBr0NMdrWCBYcSn/T0fL1Pc6UwNKqeqyi3DXEFqGC/0/o+OCj3g8MSMcGXZWs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401162; c=relaxed/simple; bh=5HRe711K32MgdSeGSmn3iIsY73LTJGgBmEQQpjf/M7s=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=cLzsExpIUPBQ8sQBBfVEhsLxyj7mDCtBIMwxA/3WGk9xthVYJZuClFdNUpM9lyUAi5lp+kv9muttNwHerukONZnH13r65/CbV3ESt3WSxlQmcql63nqP/7rWfpdE2lbfOdrvgtcDkF4iwest7nVYtIsZSD0Wu7H520NEAeedoIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=e7jEfJdI; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="e7jEfJdI" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=skQhI48ZE2eb5m7CnNrn60rghFGyRSDQ4tkQHwJzt74=; b=e7jEfJdITWz569zG0ZUB7IxMcQ 5tAwmSjxz8BgcdBoxjyCS6OhSTyt3iCjYvpBlCfhNMZIDCrhxF4Zb/XC8zWQirkT0XI1BL99F3QcG OareUdMwG0X/5Dt8GMfZZqIoek3aOc7/Iux6SDt/b8YUaO+Da1R2qYCE5LBrI4fDIeFCC81PB3zM0 rfQoqgHzbw3j0K1rbQb35PPx/nXSeJklJlD24D3lbFQW/AqC4O5lXGPnrIjYDXtQTwVDa0TRbvf5p JbTqpDMx++NCk3gDx0CslIW7l11qAC7EABbjDG/Y7lOAJJ5lYfMX/JfIMNQdDm7du7n/uKiyAcavl M9LFmqkg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqN-00000003C2T-0VZ0; Mon, 24 Feb 2025 12:45:55 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 248E93063B5; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.371942555@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:08 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 05/10] x86/ibt: Optimize FineIBT sequence References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Scott notes that non-taken branches are faster. Abuse overlapping code that traps instead of explicit UD2 instructions. And LEA does not modify flags and will have less dependencies. Suggested-by: Scott Constable Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/kernel/alternative.c | 61 +++++++++++++++++++++++++++----------= ----- arch/x86/net/bpf_jit_comp.c | 5 +-- 2 files changed, 42 insertions(+), 24 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1053,9 +1053,9 @@ early_param("cfi", cfi_parse_cmdline); * __cfi_\func: __cfi_\func: * movl $0x12345678,%eax // 5 endbr64 // 4 * nop subl $0x12345678,%r10d // 7 - * nop jz 1f // 2 - * nop ud2 // 2 - * nop 1: nop // 1 + * nop jne __cfi_\func+6 // 2 + * nop nop3 // 3 + * nop * nop * nop * nop @@ -1067,37 +1067,50 @@ early_param("cfi", cfi_parse_cmdline); * * caller: caller: * movl $(-0x12345678),%r10d // 6 movl $0x12345678,%r10d // 6 - * addl $-15(%r11),%r10d // 4 sub $16,%r11 // 4 + * addl $-15(%r11),%r10d // 4 lea -0x10(%r11),%r11 // 4 * je 1f // 2 nop4 // 4 * ud2 // 2 - * 1: call __x86_indirect_thunk_r11 // 5 call *%r11; nop2; // 5 + * 1: cs call __x86_indirect_thunk_r11 // 6 call *%r11; nop3; // 6 * */ =20 -asm( ".pushsection .rodata \n" - "fineibt_preamble_start: \n" - " endbr64 \n" - " subl $0x12345678, %r10d \n" - " je fineibt_preamble_end \n" - "fineibt_preamble_ud2: \n" - " ud2 \n" - " nop \n" - "fineibt_preamble_end: \n" +/* + * : + * 0: f3 0f 1e fa endbr64 + * 4: 41 81 78 56 34 12 sub $0x12345678, %r10d + * b: 75 f9 jne 6 + * d: 0f 1f 00 nopl (%rax) + * + * Note that the JNE target is the 0xEA byte inside the SUB, this decodes = as + * (bad) on x86_64 and raises #UD. + */ +asm( ".pushsection .rodata \n" + "fineibt_preamble_start: \n" + " endbr64 \n" + " subl $0x12345678, %r10d \n" + " jne fineibt_preamble_start+6 \n" + ASM_NOP3 + "fineibt_preamble_end: \n" ".popsection\n" ); =20 extern u8 fineibt_preamble_start[]; -extern u8 fineibt_preamble_ud2[]; extern u8 fineibt_preamble_end[]; =20 #define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_sta= rt) -#define fineibt_preamble_ud2 (fineibt_preamble_ud2 - fineibt_preamble_sta= rt) +#define fineibt_preamble_ud 6 #define fineibt_preamble_hash 7 =20 +/* + * : + * 0: 41 ba 78 56 34 12 mov $0x12345678, %r10d + * 6: 4d 8d 5b f0 lea -0x10(%r11), %r11 + * a: 0f 1f 40 00 nopl 0x0(%rax) + */ asm( ".pushsection .rodata \n" "fineibt_caller_start: \n" " movl $0x12345678, %r10d \n" - " sub $16, %r11 \n" + " lea -0x10(%r11), %r11 \n" ASM_NOP4 "fineibt_caller_end: \n" ".popsection \n" @@ -1428,15 +1441,15 @@ static void poison_cfi(void *addr) } =20 /* - * regs->ip points to a UD2 instruction, return true and fill out target a= nd - * type when this UD2 is from a FineIBT preamble. + * When regs->ip points to a 0xEA byte in the FineIBT preamble, + * return true and fill out target and type. * * We check the preamble by checking for the ENDBR instruction relative to= the - * UD2 instruction. + * 0xEA instruction. */ bool decode_fineibt_insn(struct pt_regs *regs, unsigned long *target, u32 = *type) { - unsigned long addr =3D regs->ip - fineibt_preamble_ud2; + unsigned long addr =3D regs->ip - fineibt_preamble_ud; u32 hash; =20 if (!exact_endbr((void *)addr)) @@ -1447,6 +1460,12 @@ bool decode_fineibt_insn(struct pt_regs __get_kernel_nofault(&hash, addr + fineibt_preamble_hash, u32, Efault); *type =3D (u32)regs->r10 + hash; =20 + /* + * Since regs->ip points to the middle of an instruction; it cannot + * continue with the normal fixup. + */ + regs->ip =3D *target; + return true; =20 Efault: --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -417,9 +417,8 @@ static void emit_fineibt(u8 **pprog, u32 =20 EMIT_ENDBR(); EMIT3_off32(0x41, 0x81, 0xea, hash); /* subl $hash, %r10d */ - EMIT2(0x74, 0x07); /* jz.d8 +7 */ - EMIT2(0x0f, 0x0b); /* ud2 */ - EMIT1(0x90); /* nop */ + EMIT2(0x75, 0xf9); /* jne.d8 .-7 */ + EMIT3(0x0f, 0x1f, 0x00); /* nop3 */ EMIT_ENDBR_POISON(); =20 *pprog =3D prog; From nobody Wed Dec 17 07:17:03 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50CC6233729 for ; Mon, 24 Feb 2025 12:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=FmV7j0xrEWVF4OgDjbYOb22iIYRGNBlZd/+VM+q/h5MBF22Vv4dbtSuha+BGbNrdMbtxf4ruyyPwgSuu5DWwXx7ZUCsSnReI3dr0Fwg594NHtCEjZ7CxKZ8oo8f4JgBxyBvwMQTJFQ/0J3M5oqtrb+H5RK/MGkzTfUmcxrkxBt4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=rV8WiMc3I/ryESvRkeKTE5+/FDqde/84N1oS1AJJf4E=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=i2H0fuom+v7hHbcHUWUeantKJSQhZCwYaWr3TPmUR56VQqeoGxy0DMo8BXMXQwi+v7MxdPyu27kM/sCiL/oVNjQYffKahi49nOapAny5u8x1HSQm5LQiSSbRgBujeVMQOp+JnuPSX/4UN2pKNA8mTqyyJ4yZBU8fBzPLBO/1sGQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=mW0IDIXk; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="mW0IDIXk" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=O8nlW/K8BIYLcoU8UNNApEKwuDGERPFhYSQAbOj23I8=; b=mW0IDIXkQ5lRIWqcDZIzGWW38V l0pqGOL9P1MhAkFhyWY+wOT5EW75/IBg5xJ+qmSs01KgVAg/wKR2/FykpW+yubZvMo7QwtMTGZVG1 M0Dhb63d7SlVORhr8B2K1r7fNMObEdaAiwD5Gi75kVYCS7o8YejFCSraWHf8nNr0Ol17yqxsJ3Qbv ho493bXF+psarR4zfbI6kTY2avCROTVHiVIeFzovyifJF9P2xeuLcnEhkKEn3LOUBgtwUQTa31Mmh 7nVrkf9FK8hIrgWFFdpKQGl8C8u3jeIPor8yxjPF8ZAjcWfFf8p5l1xvWsWILEQaV144kZV7JiiMh 8bybHuXA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqN-00000003C2U-0aum; Mon, 24 Feb 2025 12:45:55 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 286E6306573; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.486463917@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:09 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 06/10] x86/traps: Decode LOCK Jcc.d8 #UD References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Because overlapping code sequences are all the rage. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/include/asm/bug.h | 2 ++ arch/x86/kernel/traps.c | 26 +++++++++++++++++++++++--- 2 files changed, 25 insertions(+), 3 deletions(-) --- a/arch/x86/include/asm/bug.h +++ b/arch/x86/include/asm/bug.h @@ -17,6 +17,7 @@ * In clang we have UD1s reporting UBSAN failures on X86, 64 and 32bit. */ #define INSN_ASOP 0x67 +#define INSN_LOCK 0xf0 #define OPCODE_ESCAPE 0x0f #define SECOND_BYTE_OPCODE_UD1 0xb9 #define SECOND_BYTE_OPCODE_UD2 0x0b @@ -26,6 +27,7 @@ #define BUG_UD1 0xfffd #define BUG_UD1_UBSAN 0xfffc #define BUG_EA 0xffea +#define BUG_LOCK 0xfff0 =20 #ifdef CONFIG_GENERIC_BUG =20 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -97,6 +97,7 @@ __always_inline int is_valid_bugaddr(uns * If it's a UD1, further decode to determine its use: * * FineIBT: ea (bad) + * FineIBT: 0f 75 f9 lock jne . - 6 * UBSan{0}: 67 0f b9 00 ud1 (%eax),%eax * UBSan{10}: 67 0f b9 40 10 ud1 0x10(%eax),%eax * static_call: 0f b9 cc ud1 %esp,%ecx @@ -106,6 +107,7 @@ __always_inline int is_valid_bugaddr(uns __always_inline int decode_bug(unsigned long addr, s32 *imm, int *len) { unsigned long start =3D addr; + bool lock =3D false; u8 v; =20 if (addr < TASK_SIZE_MAX) @@ -114,12 +116,29 @@ __always_inline int decode_bug(unsigned v =3D *(u8 *)(addr++); if (v =3D=3D INSN_ASOP) v =3D *(u8 *)(addr++); - if (v =3D=3D 0xea) { + + if (v =3D=3D INSN_LOCK) { + lock =3D true; + v =3D *(u8 *)(addr++); + } + + switch (v) { + case 0x70 ... 0x7f: /* Jcc.d8 */ + addr +=3D 1; /* d8 */ + *len =3D addr - start; + WARN_ON_ONCE(!lock); + return BUG_LOCK; + + case 0xea: *len =3D addr - start; return BUG_EA; - } - if (v !=3D OPCODE_ESCAPE) + + case OPCODE_ESCAPE: + break; + + default: return BUG_NONE; + } =20 v =3D *(u8 *)(addr++); if (v =3D=3D SECOND_BYTE_OPCODE_UD2) { @@ -322,6 +341,7 @@ static noinstr bool handle_bug(struct pt fallthrough; =20 case BUG_EA: + case BUG_LOCK: if (handle_cfi_failure(regs) =3D=3D BUG_TRAP_TYPE_WARN) { handled =3D true; break; From nobody Wed Dec 17 07:17:03 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50BD3219313 for ; Mon, 24 Feb 2025 12:46:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=MF/oQWyM9SECgqV8FmzNCqbAl7OMf7DLLjnQq1rWtyWRMKCq43NsT0pifxQJdcAwbl6SS28Z88K/bDo02yvGL64vuS3+dn7UoVJ3Xiy8LA0naRRMTihMYtOLvXcDhg+kNkzV+x2fESTHctwoaWEnBbHN6PMXa8mQZIab33PZgvI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=2WOjVB/m85Pf5JtgvnyWsbcTlhIvf4IqqclserNk4uI=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=VybxuuE7EYxGnsJus/bAopCtF6pNXh6JUgT7u3nv2boM14uXeHVQHCWGP2gFrizWVHDz5lVmau6Bg7alEb9OsMVf2QaZiqFR9w4aw6NNIlswWjrtHvOMMJUftal4N4LeebaSzJbq+71T2b1ca60zKse0QpMMM9RWX37AsblSfo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=mGzkeCjJ; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="mGzkeCjJ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=mAH9uOTqSKYPwdncWlgBev8lQzyWNgnGRmhsPRQnO9M=; b=mGzkeCjJXnZPchO82t89jinQ3W fE28fYLcmaTbhqVJITMjIu5soUytYbQfNw7+QzkacCFhViQ/Ox8XOjV14O5WQhPJngKki1a7xQkf/ xTzLnD7kxbs7Ml4G1MS2Xt90bc7pdLkON4EEoXkLAED2RZ9/YBpnjX+ndoX81luXMBBYhQ5FvT/IA jiOGqhHgqUHji54qjZwb804yS288TXH9g6CBMlNxQljKcbjnwrpfRrCmn9i574H5Tk4uRJ8ikdX1y qUnAqd+0fdd1pSoScedvS7i/Gnly1+HxnYjLoyq/OPOlHdM6JgKoJ5XcDauTt5x9LoUkzDzJJlH+9 HDiKQJKg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqN-00000003C2V-0clZ; Mon, 24 Feb 2025 12:45:55 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 2BEB53078A4; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.598033084@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:10 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 07/10] x86/ibt: Add paranoid FineIBT mode References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Due to concerns about circumvention attacks against FineIBT on 'naked' ENDBR, add an additional caller side hash check to FineIBT. This should make it impossible to pivot over such a 'naked' ENDBR instruction at the cost of an additional load. The specific pivot reported was against the SYSCALL entry site and FRED will have all those holes fixed up. https://lore.kernel.org/linux-hardening/Z60NwR4w%2F28Z7XUa@ubun/ This specific fineibt_paranoid_start[] sequence was concocted by Scott. Reported-by: Jennifer Miller Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/kernel/alternative.c | 144 +++++++++++++++++++++++++++++++++++++= +++-- 1 file changed, 138 insertions(+), 6 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -741,6 +741,11 @@ void __init_or_module noinline apply_ret op2 =3D insn.opcode.bytes[1]; =20 switch (op1) { + case 0x70 ... 0x7f: /* Jcc.d8 */ + /* See cfi_paranoid. */ + WARN_ON_ONCE(cfi_mode !=3D CFI_FINEIBT); + continue; + case CALL_INSN_OPCODE: case JMP32_INSN_OPCODE: break; @@ -994,6 +999,8 @@ u32 cfi_get_func_hash(void *func) static bool cfi_rand __ro_after_init =3D true; static u32 cfi_seed __ro_after_init; =20 +static bool cfi_paranoid __ro_after_init =3D false; + /* * Re-hash the CFI hash with a boot-time seed while making sure the result= is * not a valid ENDBR instruction. @@ -1036,6 +1043,12 @@ static __init int cfi_parse_cmdline(char } else if (!strcmp(str, "warn")) { pr_alert("CFI mismatch non-fatal!\n"); cfi_warn =3D true; + } else if (!strcmp(str, "paranoid")) { + if (cfi_mode =3D=3D CFI_FINEIBT) { + cfi_paranoid =3D true; + } else { + pr_err("Ignoring paranoid; depends on fineibt.\n"); + } } else { pr_err("Ignoring unknown cfi option (%s).", str); } @@ -1124,6 +1137,52 @@ extern u8 fineibt_caller_end[]; =20 #define fineibt_caller_jmp (fineibt_caller_size - 2) =20 +/* + * Since FineIBT does hash validation on the callee side it is prone to + * circumvention attacks where a 'naked' ENDBR instruction exists that + * is not part of the fineibt_preamble sequence. + * + * Notably the x86 entry points must be ENDBR and equally cannot be + * fineibt_preamble. + * + * The fineibt_paranoid caller sequence adds additional caller side + * hash validation. This stops such circumvention attacks dead, but at the= cost + * of adding a load. + * + * : + * 0: 41 ba 78 56 34 12 mov $0x12345678, %r10d + * 6: 45 3b 53 f7 cmp -0x9(%r11), %r10d + * a: 4d 8d 5b lea -0x10(%r11), %r11 + * e: 75 fd jne d + * 10: 41 ff d3 call *%r11 + * 13: 90 nop + * + * Notably LEA does not modify flags and can be reordered with the CMP, + * avoiding a dependency. Again, using a non-taken (backwards) branch + * for the failure case, abusing LEA's immediate 0xf0 as LOCK prefix for t= he + * Jcc.d8, causing #UD. + */ +asm( ".pushsection .rodata \n" + "fineibt_paranoid_start: \n" + " movl $0x12345678, %r10d \n" + " cmpl -9(%r11), %r10d \n" + " lea -0x10(%r11), %r11 \n" + " jne fineibt_paranoid_start+0xd \n" + "fineibt_paranoid_ind: \n" + " call *%r11 \n" + " nop \n" + "fineibt_paranoid_end: \n" + ".popsection \n" +); + +extern u8 fineibt_paranoid_start[]; +extern u8 fineibt_paranoid_ind[]; +extern u8 fineibt_paranoid_end[]; + +#define fineibt_paranoid_size (fineibt_paranoid_end - fineibt_paranoid_sta= rt) +#define fineibt_paranoid_ind (fineibt_paranoid_ind - fineibt_paranoid_sta= rt) +#define fineibt_paranoid_ud 0xd + static u32 decode_preamble_hash(void *addr) { u8 *p =3D addr; @@ -1287,18 +1346,48 @@ static int cfi_rewrite_callers(s32 *star { s32 *s; =20 + BUG_ON(fineibt_paranoid_size !=3D 20); + for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; + struct insn insn; + u8 bytes[20]; u32 hash; + int ret; + u8 op; =20 addr -=3D fineibt_caller_size; hash =3D decode_caller_hash(addr); - if (hash) { + if (!hash) + continue; + + if (!cfi_paranoid) { text_poke_early(addr, fineibt_caller_start, fineibt_caller_size); WARN_ON(*(u32 *)(addr + fineibt_caller_hash) !=3D 0x12345678); text_poke_early(addr + fineibt_caller_hash, &hash, 4); + /* rely on apply_retpolines() */ + continue; + } + + /* cfi_paranoid */ + ret =3D insn_decode_kernel(&insn, addr + fineibt_caller_size); + if (WARN_ON_ONCE(ret < 0)) + continue; + + op =3D insn.opcode.bytes[0]; + if (op !=3D CALL_INSN_OPCODE && op !=3D JMP32_INSN_OPCODE) { + WARN_ON_ONCE(1); + continue; } - /* rely on apply_retpolines() */ + + memcpy(bytes, fineibt_paranoid_start, fineibt_paranoid_size); + memcpy(bytes + fineibt_caller_hash, &hash, 4); + + ret =3D emit_indirect(op, 11, bytes + fineibt_paranoid_ind); + if (WARN_ON_ONCE(ret !=3D 3)) + continue; + + text_poke_early(addr, bytes, fineibt_paranoid_size); } =20 return 0; @@ -1315,8 +1404,15 @@ static void __apply_fineibt(s32 *start_r =20 if (cfi_mode =3D=3D CFI_AUTO) { cfi_mode =3D CFI_KCFI; - if (HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT)) + if (HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT)) { + /* + * FRED has much saner context on exception entry and + * is less easy to take advantage of. + */ + if (!cpu_feature_enabled(X86_FEATURE_FRED)) + cfi_paranoid =3D true; cfi_mode =3D CFI_FINEIBT; + } } =20 /* @@ -1373,8 +1469,10 @@ static void __apply_fineibt(s32 *start_r /* now that nobody targets func()+0, remove ENDBR there */ cfi_rewrite_endbr(start_cfi, end_cfi); =20 - if (builtin) - pr_info("Using FineIBT CFI\n"); + if (builtin) { + pr_info("Using %sFineIBT CFI\n", + cfi_paranoid ? "paranoid " : ""); + } return; =20 default: @@ -1447,7 +1545,7 @@ static void poison_cfi(void *addr) * We check the preamble by checking for the ENDBR instruction relative to= the * 0xEA instruction. */ -bool decode_fineibt_insn(struct pt_regs *regs, unsigned long *target, u32 = *type) +static bool decode_fineibt_preamble(struct pt_regs *regs, unsigned long *t= arget, u32 *type) { unsigned long addr =3D regs->ip - fineibt_preamble_ud; u32 hash; @@ -1472,6 +1570,40 @@ bool decode_fineibt_insn(struct pt_regs return false; } =20 +/* + * regs->ip points to a LOCK Jcc.d8 instruction from the fineibt_paranoid_= start[] + * sequence. + */ +static bool decode_fineibt_paranoid(struct pt_regs *regs, unsigned long *t= arget, u32 *type) +{ + unsigned long addr =3D regs->ip - fineibt_paranoid_ud; + u32 hash; + + if (!cfi_paranoid || !is_cfi_trap(addr + fineibt_caller_size - LEN_UD2)) + return false; + + __get_kernel_nofault(&hash, addr + fineibt_caller_hash, u32, Efault); + *target =3D regs->r11 + fineibt_preamble_size; + *type =3D regs->r10; + + /* + * Since the trapping instruction is the exact, but LOCK prefixed, + * Jcc.d8 that got us here, the normal fixup will work. + */ + return true; + +Efault: + return false; +} + +bool decode_fineibt_insn(struct pt_regs *regs, unsigned long *target, u32 = *type) +{ + if (decode_fineibt_paranoid(regs, target, type)) + return true; + + return decode_fineibt_preamble(regs, target, type); +} + #else =20 static void __apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, From nobody Wed Dec 17 07:17:03 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50B6C21930E for ; Mon, 24 Feb 2025 12:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401164; cv=none; b=tJ9qwlKo/P0rOmyfMK+tE5L6ZOq68TUp9BJXpPQIhG1tSb6YflL9RKvRz6epruHIRhYLqGEyJrOWLsEDxC0IIa+q+7pC6u9qgdvXEFcLokJozO5dWBRmcBf9Ab0mv+I0gmAsW9tgYQPbdZykSr9Eec+I3N5UOmMi0ugeeZGFRx4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401164; c=relaxed/simple; bh=fWc+Fuw0k4sUEWKZxXxObCDs5KXrWXOOIuRvUtq9krw=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=Y1J9nGTFsJZkyhhDtLlfZMfkpv019Gq6tDTlwiiHutzdygy1Wm6MNyzbV21/GP+fC6kCju6yVLd+Q/nD2zLiZukSfvWYj9lwh3A1Omsb5A761WBpPI4YGlkM+Ufqd1R225Ka0Tm6Y73HG9HATkheM047fAvDYTThNkMx78rE+jQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=NlYyERSp; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="NlYyERSp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=94t6o8AkyshOtxSEKb/7p5vb1+FDXIF2IGvFBSAPtx4=; b=NlYyERSpwa78hnW4eZ8alLeCqf xNA0TMispOirO5dHtwA0gWVbX/Jjc5iSRDmGSNQeLTlltvna81qzIRWJRLu6eYCqm6Nx7xG5vy9zH nxuawC/cbyL0uQBTKqWTRABlZlEIR4VrgAsk7WBHFDbLnWRR27uCeURgBE3+5WN+6r31MA9QpnPdP 3vmp/EFtJ+nPHwuJweerx286cOFWJLyHo1nGxnT3ABltYLSX9roDn+WgulFjl4drrtBXTjCbZovco tUmGeDp9La3aIrGH6w1rv45OrJ5qXz9Ex9mveTpMVxeXy7MyEm4Ku85OTdKvUyYu5WG8NgIWR6h+f PxrvOU+A==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqN-000000079ml-0jab; Mon, 24 Feb 2025 12:45:55 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 2EF61307F02; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.717378681@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:11 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 08/10] x86: BHI stubs References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/include/asm/cfi.h | 4 + arch/x86/lib/Makefile | 3=20 arch/x86/lib/bhi.S | 146 ++++++++++++++++++++++++++++++++++++++++= +++++ 3 files changed, 152 insertions(+), 1 deletion(-) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -101,6 +101,10 @@ enum cfi_mode { =20 extern enum cfi_mode cfi_mode; =20 +typedef u8 bhi_thunk[32]; +extern bhi_thunk __bhi_args[]; +extern bhi_thunk __bhi_args_end[]; + struct pt_regs; =20 #ifdef CONFIG_CFI_CLANG --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -66,5 +66,6 @@ endif lib-y +=3D clear_page_64.o copy_page_64.o lib-y +=3D memmove_64.o memset_64.o lib-y +=3D copy_user_64.o copy_user_uncached_64.o - lib-y +=3D cmpxchg16b_emu.o + lib-y +=3D cmpxchg16b_emu.o + lib-y +=3D bhi.o endif --- /dev/null +++ b/arch/x86/lib/bhi.S @@ -0,0 +1,146 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include + +/* + * Notably, the FineIBT preamble calling these will have ZF set and r10 ze= ro. + * + * The very last element is in fact larger than 32 bytes, but since its the + * last element, this does not matter, + * + * There are 2 #UD sites, located between 0,1-2,3 and 4,5-6,7 such that th= ey + * can be reached using Jcc.d8, these elements (1 and 5) have sufficiently + * big alignment holes for this to not stagger the array. + */ + +.pushsection .noinstr.text, "ax" + + .align 32 +SYM_CODE_START(__bhi_args) + +#ifdef CONFIG_FINEIBT_BHI + + .align 32 +SYM_INNER_LABEL(__bhi_args_0, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_1 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_1, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_1 + cmovne %r10, %rdi + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 8 + ANNOTATE_REACHABLE +.Lud_1: ud2 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_2, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_1 + cmovne %r10, %rdi + cmovne %r10, %rsi + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_3, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_1 + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_4, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_2 + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_5, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_2 + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 8 + ANNOTATE_REACHABLE +.Lud_2: ud2 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_6, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_2 + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + ANNOTATE_UNRET_SAFE + ret + int3 + + .align 32 +SYM_INNER_LABEL(__bhi_args_7, SYM_L_LOCAL) + ANNOTATE_NOENDBR + UNWIND_HINT_FUNC + jne .Lud_2 + cmovne %r10, %rdi + cmovne %r10, %rsi + cmovne %r10, %rdx + cmovne %r10, %rcx + cmovne %r10, %r8 + cmovne %r10, %r9 + cmovne %r10, %rsp + ANNOTATE_UNRET_SAFE + ret + int3 + +#endif /* CONFIG_FINEIBT_BHI */ + + .align 32 +SYM_INNER_LABEL(__bhi_args_end, SYM_L_GLOBAL) + ANNOTATE_NOENDBR +SYM_CODE_END(__bhi_args) + +.popsection From nobody Wed Dec 17 07:17:03 2025 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E035D1519A1 for ; Mon, 24 Feb 2025 12:46:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=siyYj1wEDJ19GYiJc0XPzEB9+7YHVBnne76KZHLKZXaudSjKiJS8swKo5+b+wt5W8vLOxy4/tWUpficDrsWboM0H3k0EqnPeSk7kardQzSMN0y7Nr5WHPp6k6rfc2kDPPsjIwCg96ojsdPRuCjJHlYyzqYgIjfo/iCpDKmk5q1Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=dmnvdlhWgDbi0hbRIMm67vsauH0yLfpdhG3wN4mRELw=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=W+EErwjL3tC8yz0HVfRsYooAa8rLaC2+TZdQxdcDeYkYWTb8MwmeqpXHq7iceg2ee4q/Bb9+Urts/W5Yg4c5mAlItKCf0Zc7ht+G4xjrPrp1bzM4tDS+6JlK/up4YcJiiFrcyq6HmRZ+ttx83gEAeNAzOSiDCSB02wFx7IFRpU4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=gWXkAWk7; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="gWXkAWk7" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=+q7hDJj0ov0CGa+ulzmF01G7YVJQ+Z0Z7dP4W5sCcZQ=; b=gWXkAWk7YWuYOxPSyxU2L0q6up DP9/Sk0zc/kP4mvTMJjl4YNtfpQ+jj2B/bpshFsisvebih2BPH/eIcQpLcAyGihb7VlzMMvQ2nA7L WP7bg4OGyCy8rPHatDEvN6eoYPgZijGIw0TB1jS+LqMHZ1NQK6OOBksyp13gn9IyWgpjKpLEN+CiY imRnIROmFPyl7xLYPNiJgzI1dr83GZ8AIXLja39czqeHhYz+NMivdiOCCC5Liy1wGpXp4/CRLU/v7 YEMelmmPIt0PFTrzBt2QQ8XQUourScSQPHB5rRjN2KbVWrTMxrqLcHQELqtTTWH+fZsaYL+67H1ZX TvXQ9ePQ==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqN-00000003C2c-1pnH; Mon, 24 Feb 2025 12:45:55 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 327AA307FAE; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.820402212@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:12 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 09/10] x86/ibt: Implement FineIBT-BHI mitigation References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" While WAIT_FOR_ENDBR is specified to be a full speculation stop; it has been shown that some implementations are 'leaky' to such an extend that speculation can escape even the FineIBT preamble. To deal with this, add additional hardening to the FineIBT preamble. Notably, using a new LLVM feature: https://github.com/llvm/llvm-project/commit/e223485c9b38a5579991b8cebb6a2= 00153eee245 which encodes the number of arguments in the kCFI preamble's register. Using this register<->arity mapping, have the FineIBT preamble CALL into a stub clobbering the relevant argument registers in the speculative case. (This is where Scott goes and gives more details...) Suggested-by: Scott Constable Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- Makefile | 3 + arch/x86/Kconfig | 8 +++ arch/x86/include/asm/cfi.h | 6 ++ arch/x86/kernel/alternative.c | 107 +++++++++++++++++++++++++++++++++++++= +---- arch/x86/net/bpf_jit_comp.c | 29 +++++++---- 5 files changed, 134 insertions(+), 19 deletions(-) --- a/Makefile +++ b/Makefile @@ -1014,6 +1014,9 @@ CC_FLAGS_CFI :=3D -fsanitize=3Dkcfi ifdef CONFIG_CFI_ICALL_NORMALIZE_INTEGERS CC_FLAGS_CFI +=3D -fsanitize-cfi-icall-experimental-normalize-integers endif +ifdef CONFIG_FINEIBT_BHI + CC_FLAGS_CFI +=3D -fsanitize-kcfi-arity +endif ifdef CONFIG_RUST # Always pass -Zsanitizer-cfi-normalize-integers as CONFIG_RUST selects # CONFIG_CFI_ICALL_NORMALIZE_INTEGERS. --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2473,6 +2473,10 @@ config CC_HAS_RETURN_THUNK config CC_HAS_ENTRY_PADDING def_bool $(cc-option,-fpatchable-function-entry=3D16,16) =20 +config CC_HAS_KCFI_ARITY + def_bool $(cc-option,-fsanitize=3Dkcfi -fsanitize-kcfi-arity) + depends on CC_IS_CLANG && !RUST + config FUNCTION_PADDING_CFI int default 59 if FUNCTION_ALIGNMENT_64B @@ -2498,6 +2502,10 @@ config FINEIBT depends on X86_KERNEL_IBT && CFI_CLANG && MITIGATION_RETPOLINE select CALL_PADDING =20 +config FINEIBT_BHI + def_bool y + depends on FINEIBT && CC_HAS_KCFI_ARITY + config HAVE_CALL_THUNKS def_bool y depends on CC_HAS_ENTRY_PADDING && MITIGATION_RETHUNK && OBJTOOL --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -100,6 +100,7 @@ enum cfi_mode { }; =20 extern enum cfi_mode cfi_mode; +extern bool cfi_bhi; =20 typedef u8 bhi_thunk[32]; extern bhi_thunk __bhi_args[]; @@ -129,6 +130,7 @@ static inline int cfi_get_offset(void) #define cfi_get_offset cfi_get_offset =20 extern u32 cfi_get_func_hash(void *func); +extern int cfi_get_func_arity(void *func); =20 #ifdef CONFIG_FINEIBT extern bool decode_fineibt_insn(struct pt_regs *regs, unsigned long *targe= t, u32 *type); @@ -152,6 +154,10 @@ static inline u32 cfi_get_func_hash(void { return 0; } +static inline int cfi_get_func_arity(void *func) +{ + return 0; +} #endif /* CONFIG_CFI_CLANG */ =20 #if HAS_KERNEL_IBT =3D=3D 1 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -932,6 +932,7 @@ void __init_or_module apply_seal_endbr(s #endif =20 enum cfi_mode cfi_mode __ro_after_init =3D __CFI_DEFAULT; +bool cfi_bhi __ro_after_init =3D false; =20 #ifdef CONFIG_CFI_CLANG struct bpf_insn; @@ -992,6 +993,21 @@ u32 cfi_get_func_hash(void *func) =20 return hash; } + +int cfi_get_func_arity(void *func) +{ + bhi_thunk *target; + s32 disp; + + if (cfi_mode !=3D CFI_FINEIBT && !cfi_bhi) + return 0; + + if (get_kernel_nofault(disp, func - 4)) + return 0; + + target =3D func + disp; + return target - __bhi_args; +} #endif =20 #ifdef CONFIG_FINEIBT @@ -1049,6 +1065,12 @@ static __init int cfi_parse_cmdline(char } else { pr_err("Ignoring paranoid; depends on fineibt.\n"); } + } else if (!strcmp(str, "bhi")) { + if (cfi_mode =3D=3D CFI_FINEIBT) { + cfi_bhi =3D true; + } else { + pr_err("Ignoring bhi; depends on fineibt.\n"); + } } else { pr_err("Ignoring unknown cfi option (%s).", str); } @@ -1101,6 +1123,7 @@ asm( ".pushsection .rodata \n" "fineibt_preamble_start: \n" " endbr64 \n" " subl $0x12345678, %r10d \n" + "fineibt_preamble_bhi: \n" " jne fineibt_preamble_start+6 \n" ASM_NOP3 "fineibt_preamble_end: \n" @@ -1108,9 +1131,11 @@ asm( ".pushsection .rodata \n" ); =20 extern u8 fineibt_preamble_start[]; +extern u8 fineibt_preamble_bhi[]; extern u8 fineibt_preamble_end[]; =20 #define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_sta= rt) +#define fineibt_preamble_bhi (fineibt_preamble_bhi - fineibt_preamble_sta= rt) #define fineibt_preamble_ud 6 #define fineibt_preamble_hash 7 =20 @@ -1183,13 +1208,16 @@ extern u8 fineibt_paranoid_end[]; #define fineibt_paranoid_ind (fineibt_paranoid_ind - fineibt_paranoid_sta= rt) #define fineibt_paranoid_ud 0xd =20 -static u32 decode_preamble_hash(void *addr) +static u32 decode_preamble_hash(void *addr, int *reg) { u8 *p =3D addr; =20 - /* b8 78 56 34 12 mov $0x12345678,%eax */ - if (p[0] =3D=3D 0xb8) + /* b8+reg 78 56 34 12 movl $0x12345678,\reg */ + if (p[0] >=3D 0xb8 && p[0] < 0xc0) { + if (reg) + *reg =3D p[0] - 0xb8; return *(u32 *)(addr + 1); + } =20 return 0; /* invalid hash value */ } @@ -1198,11 +1226,11 @@ static u32 decode_caller_hash(void *addr { u8 *p =3D addr; =20 - /* 41 ba 78 56 34 12 mov $0x12345678,%r10d */ + /* 41 ba 88 a9 cb ed mov $(-0x12345678),%r10d */ if (p[0] =3D=3D 0x41 && p[1] =3D=3D 0xba) return -*(u32 *)(addr + 2); =20 - /* e8 0c 78 56 34 12 jmp.d8 +12 */ + /* e8 0c 88 a9 cb ed jmp.d8 +12 */ if (p[0] =3D=3D JMP8_INSN_OPCODE && p[1] =3D=3D fineibt_caller_jmp) return -*(u32 *)(addr + 2); =20 @@ -1267,7 +1295,7 @@ static int cfi_rand_preamble(s32 *start, void *addr =3D (void *)s + *s; u32 hash; =20 - hash =3D decode_preamble_hash(addr); + hash =3D decode_preamble_hash(addr, NULL); if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", addr, addr, 5, addr)) return -EINVAL; @@ -1285,6 +1313,7 @@ static int cfi_rewrite_preamble(s32 *sta =20 for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; + int arity; u32 hash; =20 /* @@ -1295,7 +1324,7 @@ static int cfi_rewrite_preamble(s32 *sta if (!is_endbr(addr + 16)) continue; =20 - hash =3D decode_preamble_hash(addr); + hash =3D decode_preamble_hash(addr, &arity); if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", addr, addr, 5, addr)) return -EINVAL; @@ -1303,6 +1332,19 @@ static int cfi_rewrite_preamble(s32 *sta text_poke_early(addr, fineibt_preamble_start, fineibt_preamble_size); WARN_ON(*(u32 *)(addr + fineibt_preamble_hash) !=3D 0x12345678); text_poke_early(addr + fineibt_preamble_hash, &hash, 4); + + WARN_ONCE(!IS_ENABLED(CONFIG_FINEIBT_BHI) && arity, + "kCFI preamble has wrong register at: %pS %*ph\n", + addr, 5, addr); + + if (!cfi_bhi || !arity) + continue; + + text_poke_early(addr + fineibt_preamble_bhi, + text_gen_insn(CALL_INSN_OPCODE, + addr + fineibt_preamble_bhi, + __bhi_args[arity]), + CALL_INSN_SIZE); } =20 return 0; @@ -1470,8 +1512,9 @@ static void __apply_fineibt(s32 *start_r cfi_rewrite_endbr(start_cfi, end_cfi); =20 if (builtin) { - pr_info("Using %sFineIBT CFI\n", - cfi_paranoid ? "paranoid " : ""); + pr_info("Using %sFineIBT%s CFI\n", + cfi_paranoid ? "paranoid " : "", + cfi_bhi ? "+BHI" : ""); } return; =20 @@ -1522,7 +1565,7 @@ static void poison_cfi(void *addr) /* * kCFI prefix should start with a valid hash. */ - if (!decode_preamble_hash(addr)) + if (!decode_preamble_hash(addr, NULL)) break; =20 /* @@ -1571,6 +1614,47 @@ static bool decode_fineibt_preamble(stru } =20 /* + * regs->ip points to one of the UD2 in __bhi_args[]. + */ +static bool decode_fineibt_bhi(struct pt_regs *regs, unsigned long *target= , u32 *type) +{ + unsigned long addr; + u32 hash; + + if (!cfi_bhi) + return false; + + if (regs->ip < (unsigned long)__bhi_args || + regs->ip >=3D (unsigned long)__bhi_args_end) + return false; + + /* + * Fetch the return address from the stack, this points to the + * FineIBT preamble. Since the CALL instruction is in the 5 last + * bytes of the preamble, the return address is in fact the target + * address. + */ + __get_kernel_nofault(&addr, regs->sp, unsigned long, Efault); + *target =3D addr; + + addr -=3D fineibt_preamble_size; + if (!exact_endbr((void *)addr)) + return false; + + __get_kernel_nofault(&hash, addr + fineibt_preamble_hash, u32, Efault); + *type =3D (u32)regs->r10 + hash; + + /* + * The UD2 sites are constructed with a RET immediately following, + * as such the non-fatal case can use the regular fixup. + */ + return true; + +Efault: + return false; +} + +/* * regs->ip points to a LOCK Jcc.d8 instruction from the fineibt_paranoid_= start[] * sequence. */ @@ -1601,6 +1685,9 @@ bool decode_fineibt_insn(struct pt_regs if (decode_fineibt_paranoid(regs, target, type)) return true; =20 + if (decode_fineibt_bhi(regs, target, type)) + return true; + return decode_fineibt_preamble(regs, target, type); } =20 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -410,15 +410,20 @@ static void emit_nops(u8 **pprog, int le * Emit the various CFI preambles, see asm/cfi.h and the comments about Fi= neIBT * in arch/x86/kernel/alternative.c */ +static int emit_call(u8 **prog, void *func, void *ip); =20 -static void emit_fineibt(u8 **pprog, u32 hash) +static void emit_fineibt(u8 **pprog, u8 *ip, u32 hash, int arity) { u8 *prog =3D *pprog; =20 EMIT_ENDBR(); EMIT3_off32(0x41, 0x81, 0xea, hash); /* subl $hash, %r10d */ - EMIT2(0x75, 0xf9); /* jne.d8 .-7 */ - EMIT3(0x0f, 0x1f, 0x00); /* nop3 */ + if (cfi_bhi) { + emit_call(&prog, __bhi_args[arity], ip + 11); + } else { + EMIT2(0x75, 0xf9); /* jne.d8 .-7 */ + EMIT3(0x0f, 0x1f, 0x00); /* nop3 */ + } EMIT_ENDBR_POISON(); =20 *pprog =3D prog; @@ -447,13 +452,13 @@ static void emit_kcfi(u8 **pprog, u32 ha *pprog =3D prog; } =20 -static void emit_cfi(u8 **pprog, u32 hash) +static void emit_cfi(u8 **pprog, u8 *ip, u32 hash, int arity) { u8 *prog =3D *pprog; =20 switch (cfi_mode) { case CFI_FINEIBT: - emit_fineibt(&prog, hash); + emit_fineibt(&prog, ip, hash, arity); break; =20 case CFI_KCFI: @@ -504,13 +509,17 @@ static void emit_prologue_tail_call(u8 * * bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes * while jumping to another program */ -static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf, +static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_f= rom_cbpf, bool tail_call_reachable, bool is_subprog, bool is_exception_cb) { u8 *prog =3D *pprog; =20 - emit_cfi(&prog, is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash); + if (is_subprog) { + emit_cfi(&prog, ip, cfi_bpf_subprog_hash, 5); + } else { + emit_cfi(&prog, ip, cfi_bpf_hash, 1); + } /* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ @@ -1479,7 +1488,7 @@ static int do_jit(struct bpf_prog *bpf_p =20 detect_reg_usage(insn, insn_cnt, callee_regs_used); =20 - emit_prologue(&prog, stack_depth, + emit_prologue(&prog, image, stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb); /* Exception callback will clobber callee regs for its own use, and @@ -3046,7 +3055,9 @@ static int __arch_prepare_bpf_trampoline /* * Indirect call for bpf_struct_ops */ - emit_cfi(&prog, cfi_get_func_hash(func_addr)); + emit_cfi(&prog, image, + cfi_get_func_hash(func_addr), + cfi_get_func_arity(func_addr)); } else { /* * Direct-call fentry stub, as such it needs accounting for the From nobody Wed Dec 17 07:17:03 2025 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E701C163 for ; Mon, 24 Feb 2025 12:46:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; cv=none; b=bEZypZr0576R8fZkyxttWwJj5cwE2aC8rOW2UxNRGgShLBKSNtusqNebCpGrxnD6F5AnD3XS/xgnmKcOZ6NUoixXX0xYzbO/JRwaIwMjMVNreoAixGdWMVJz519wDU2Wl1Cp1ZQljhfZC/jZhnNRIF0sSLR74DODxq/fz+RyKMM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740401163; c=relaxed/simple; bh=pDgaArRbB9J1a9tmEa0hR/h0yVxFlmcKaFGE2f+f6Pg=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=b4uffJ6H8yGnakH+1pfRuMS4k5sUv45S2/7ZDc1BH/vrFK9sntRH3IsUaiX6Bn7BloTQ2cB9ZJLGaCeJf2nib8Tu5rPhbuhDPtRaz3KtlxdjzoIouyIindCeAMrKNmVtegV3nzLL4obFaRH+LAXeOkyfPonU/djPKyhFC2Fg8Ls= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=PRMkQEJ8; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="PRMkQEJ8" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=YqfLieYxAGFlflkWnT6nHogORevj322hTYMScSLzSlo=; b=PRMkQEJ8kid/DriIa1Z+F78wW2 3Vk/AznrgtdqTqStRzJZ6rM63JevBtnolHf8iNeg1fGctAN77qjKhWdcUjujgAM0/hv7jcF0zNXlB cu4Y05Qe44dv5gwofHQSswvymW3uHoA0WJOCeVmLHJxcXIUelz+8FZ09zRND5QCf4sNwd96IP0uSa 95twV2UnDpbptKKhDah8inu7FI1fvtYnUy6QMD+C1cYGmamTT1OpKDrWQIXy3itKvslxd2xf3wH0X HjO1YUq8DSBEW26xIV9aiAlNYtZL8g4FeaeYNhhMqG50r7OgPpOc2NT3xZo7C4e+2/hSG1ElP/EeY moXNI//Q==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tmXqN-000000079mm-1xXd; Mon, 24 Feb 2025 12:45:55 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 360E73080BA; Mon, 24 Feb 2025 13:45:54 +0100 (CET) Message-ID: <20250224124200.927885784@infradead.org> User-Agent: quilt/0.66 Date: Mon, 24 Feb 2025 13:37:13 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, alyssa.milburn@intel.com, scott.d.constable@intel.com, joao@overdrivepizza.com, andrew.cooper3@citrix.com, jpoimboe@kernel.org, jose.marchesi@oracle.com, hjl.tools@gmail.com, ndesaulniers@google.com, samitolvanen@google.com, nathan@kernel.org, ojeda@kernel.org, kees@kernel.org, alexei.starovoitov@gmail.com, mhiramat@kernel.org, jmill@asu.edu Subject: [PATCH v4 10/10] x86/ibt: Optimize fineibt-bhi arity 1 case References: <20250224123703.843199044@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Saves a CALL to an out-of-line thunk for the common case of 1 argument. Suggested-by: Scott Constable Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/include/asm/ibt.h | 4 ++ arch/x86/kernel/alternative.c | 59 +++++++++++++++++++++++++++++++++++--= ----- 2 files changed, 54 insertions(+), 9 deletions(-) --- a/arch/x86/include/asm/ibt.h +++ b/arch/x86/include/asm/ibt.h @@ -70,6 +70,10 @@ static inline bool __is_endbr(u32 val) if (val =3D=3D gen_endbr_poison()) return true; =20 + /* See cfi_fineibt_bhi_preamble() */ + if (IS_ENABLED(CONFIG_FINEIBT_BHI) && val =3D=3D 0x001f0ff5) + return true; + val &=3D ~0x01000000U; /* ENDBR32 -> ENDBR64 */ return val =3D=3D gen_endbr(); } --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1307,6 +1307,53 @@ static int cfi_rand_preamble(s32 *start, return 0; } =20 +static void cfi_fineibt_bhi_preamble(void *addr, int arity) +{ + if (!arity) + return; + + if (!cfi_warn && arity =3D=3D 1) { + /* + * Crazy scheme to allow arity-1 inline: + * + * __cfi_foo: + * 0: f3 0f 1e fa endbr64 + * 4: 41 81 78 56 34 12 sub 0x12345678, %r10d + * b: 49 0f 45 fa cmovne %r10, %rdi + * f: 75 f5 jne __cfi_foo+6 + * 11: 0f 1f 00 nopl (%rax) + * + * Code that direct calls to foo()+0, decodes the tail end as: + * + * foo: + * 0: f5 cmc + * 1: 0f 1f 00 nopl (%rax) + * + * which clobbers CF, but does not affect anything ABI + * wise. + * + * Notably, this scheme is incompatible with permissive CFI + * because the CMOVcc is unconditional and RDI will have been + * clobbered. + */ + const u8 magic[9] =3D { + 0x49, 0x0f, 0x45, 0xfa, + 0x75, 0xf5, + BYTES_NOP3, + }; + + text_poke_early(addr + fineibt_preamble_bhi, magic, 9); + + return; + } + + text_poke_early(addr + fineibt_preamble_bhi, + text_gen_insn(CALL_INSN_OPCODE, + addr + fineibt_preamble_bhi, + __bhi_args[arity]), + CALL_INSN_SIZE); +} + static int cfi_rewrite_preamble(s32 *start, s32 *end) { s32 *s; @@ -1337,14 +1384,8 @@ static int cfi_rewrite_preamble(s32 *sta "kCFI preamble has wrong register at: %pS %*ph\n", addr, 5, addr); =20 - if (!cfi_bhi || !arity) - continue; - - text_poke_early(addr + fineibt_preamble_bhi, - text_gen_insn(CALL_INSN_OPCODE, - addr + fineibt_preamble_bhi, - __bhi_args[arity]), - CALL_INSN_SIZE); + if (cfi_bhi) + cfi_fineibt_bhi_preamble(addr, arity); } =20 return 0; @@ -1357,7 +1398,7 @@ static void cfi_rewrite_endbr(s32 *start for (s =3D start; s < end; s++) { void *addr =3D (void *)s + *s; =20 - if (!is_endbr(addr + 16)) + if (!exact_endbr(addr + 16)) continue; =20 poison_endbr(addr + 16);