From nobody Sat Feb 7 17:19:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9612BEB64DA for ; Thu, 22 Jun 2023 14:51:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231927AbjFVOu6 (ORCPT ); Thu, 22 Jun 2023 10:50:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231866AbjFVOug (ORCPT ); Thu, 22 Jun 2023 10:50:36 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0A891710 for ; Thu, 22 Jun 2023 07:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=FI1Ilf3TB6X2/hXhpNF04O2UYw1z64psa92hXMDi+uU=; b=P9y0b72/KnuWV4rOI284V4niL6 bwMBr6txDcKTZfI2iEfHs2xGzc/eWJhA03uX85c+n4Sb3ygdtxh8QExp0RxlQWMU3nN2/c7E6uHxN 0cAxuLq/lVGkHYAfET2O28dvzeaBT0DCD81LWUCzsKOrZ74nu8pXX5duHV5Z/kPxiuWpWgMeGcpkR qyPHy47E7o92Ur/nuv4xbRwe8CKPNBfiKe58MOg3+FjTG7c6on+cMT0MVSSrK4KtsOqqlzTj6oCk2 CpZP4Gy9bIwXpX9Wn9vM0leOnIODoXcP15tNloD2H9pvJEoEMrxo5N1ky7wGY9WfED8XDWAtxW8AM onUcA9jw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qCLdh-00FgfA-Rv; Thu, 22 Jun 2023 14:50:26 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2C40D300338; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 15905209D8B39; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Message-ID: <20230622144321.360957723@infradead.org> User-Agent: quilt/0.66 Date: Thu, 22 Jun 2023 16:42:19 +0200 From: Peter Zijlstra To: x86@kernel.org, alyssa.milburn@linux.intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, samitolvanen@google.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, brgerst@gmail.com Subject: [PATCH v2 1/6] x86/cfi: Extend {JMP,CAKK}_NOSPEC comment References: <20230622144218.860926475@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With the introduction of kCFI these helpers are no longer equivalent to C indirect calls and should be used with care. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook Reviewed-by: Sami Tolvanen --- arch/x86/include/asm/nospec-branch.h | 4 ++++ 1 file changed, 4 insertions(+) --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -234,6 +234,10 @@ * JMP_NOSPEC and CALL_NOSPEC macros can be used instead of a simple * indirect jmp/call which may be susceptible to the Spectre variant 2 * attack. + * + * NOTE: these do not take kCFI into account and are thus not comparable t= o C + * indirect calls, take care when using. The target of these should be an = ENDBR + * instruction irrespective of kCFI. */ .macro JMP_NOSPEC reg:req #ifdef CONFIG_RETPOLINE From nobody Sat Feb 7 17:19:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55402EB64DA for ; Thu, 22 Jun 2023 14:50:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231603AbjFVOuu (ORCPT ); Thu, 22 Jun 2023 10:50:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231403AbjFVOug (ORCPT ); Thu, 22 Jun 2023 10:50:36 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F16FA1995 for ; Thu, 22 Jun 2023 07:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=BTT+KoTXqCaNq9y13aV/Jw9DZObN70x0nFDB9z9hNio=; b=fbvfH2/HrnNfAVJ69LLHLM/iwi JHopjHpVTm/xSNYjjJbBfkdnDXhp1u+5qYEZbc5IkZjky+2ZsJGY+iFaRB3cXGtzB+VRbinoPb7Ae NQU4erYGvI7cq2fINx7mr3lkMpGzJ/3wAR8sVN/Nq1GFMaaQPd2uR1B6G5aU3fAuCpyFWsr2bVeso fjY6sp+vZe4i1aUc0irIpwEe/7HnqQMnIF83vXK6a624TlQCjR0K3GZlmpoRDbovKdlWqoPrT8sFw cGZaF1/rXWTilayNT6o95K6uZgbEyAyC1YhZ0I9+TsNHp7x9jrSlBYiUNTiPsEIrQGVdXu77Da++h jYYnKxvA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qCLdh-00FgfB-Rc; Thu, 22 Jun 2023 14:50:26 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 309B0300427; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 17D91209D8B3B; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Message-ID: <20230622144321.427441595@infradead.org> User-Agent: quilt/0.66 Date: Thu, 22 Jun 2023 16:42:20 +0200 From: Peter Zijlstra To: x86@kernel.org, alyssa.milburn@linux.intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, samitolvanen@google.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, brgerst@gmail.com Subject: [PATCH v2 2/6] x86/alternative: Rename apply_ibt_endbr() References: <20230622144218.860926475@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current name doesn't reflect what it does very well. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook Reviewed-by: Sami Tolvanen --- arch/um/kernel/um_arch.c | 2 +- arch/x86/include/asm/alternative.h | 2 +- arch/x86/include/asm/ibt.h | 2 +- arch/x86/kernel/alternative.c | 9 ++++++--- arch/x86/kernel/module.c | 2 +- 5 files changed, 10 insertions(+), 7 deletions(-) --- a/arch/um/kernel/um_arch.c +++ b/arch/um/kernel/um_arch.c @@ -437,7 +437,7 @@ void __init arch_cpu_finalize_init(void) os_check_bugs(); } =20 -void apply_ibt_endbr(s32 *start, s32 *end) +void apply_seal_endbr(s32 *start, s32 *end) { } =20 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -96,7 +96,7 @@ extern void alternative_instructions(voi extern void apply_alternatives(struct alt_instr *start, struct alt_instr *= end); extern void apply_retpolines(s32 *start, s32 *end); extern void apply_returns(s32 *start, s32 *end); -extern void apply_ibt_endbr(s32 *start, s32 *end); +extern void apply_seal_endbr(s32 *start, s32 *end); extern void apply_fineibt(s32 *start_retpoline, s32 *end_retpoine, s32 *start_cfi, s32 *end_cfi); =20 --- a/arch/x86/include/asm/ibt.h +++ b/arch/x86/include/asm/ibt.h @@ -34,7 +34,7 @@ /* * Create a dummy function pointer reference to prevent objtool from marki= ng * the function as needing to be "sealed" (i.e. ENDBR converted to NOP by - * apply_ibt_endbr()). + * apply_seal_endbr()). */ #define IBT_NOSEAL(fname) \ ".pushsection .discard.ibt_endbr_noseal\n\t" \ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -803,7 +803,7 @@ static void __init_or_module poison_endb /* * Generated by: objtool --ibt */ -void __init_or_module noinline apply_ibt_endbr(s32 *start, s32 *end) +void __init_or_module noinline apply_seal_endbr(s32 *start, s32 *end) { s32 *s; =20 @@ -818,7 +818,7 @@ void __init_or_module noinline apply_ibt =20 #else =20 -void __init_or_module apply_ibt_endbr(s32 *start, s32 *end) { } +void __init_or_module apply_seal_endbr(s32 *start, s32 *end) { } =20 #endif /* CONFIG_X86_KERNEL_IBT */ =20 @@ -1565,7 +1565,10 @@ void __init alternative_instructions(voi */ callthunks_patch_builtin_calls(); =20 - apply_ibt_endbr(__ibt_endbr_seal, __ibt_endbr_seal_end); + /* + * Seal all functions that do not have their address taken. + */ + apply_seal_endbr(__ibt_endbr_seal, __ibt_endbr_seal_end); =20 #ifdef CONFIG_SMP /* Patch to UP if other cpus not imminent. */ --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -358,7 +358,7 @@ int module_finalize(const Elf_Ehdr *hdr, } if (ibt_endbr) { void *iseg =3D (void *)ibt_endbr->sh_addr; - apply_ibt_endbr(iseg, iseg + ibt_endbr->sh_size); + apply_seal_endbr(iseg, iseg + ibt_endbr->sh_size); } if (locks) { void *lseg =3D (void *)locks->sh_addr; From nobody Sat Feb 7 17:19:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FBC9EB64D8 for ; Thu, 22 Jun 2023 14:51:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231974AbjFVOvC (ORCPT ); Thu, 22 Jun 2023 10:51:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231881AbjFVOug (ORCPT ); Thu, 22 Jun 2023 10:50:36 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BACD19B7 for ; Thu, 22 Jun 2023 07:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=QJieZYDWzFCRvSf0wOeMBiP6eKWnKstpX1vra/Nb2RE=; b=t5FAyWdSCA8MhmTF70mRlJ8+bF PtTAVofzOV0VUJ4IiR6Mj3h2bermXWeNMQu6CTVlEUAHj7O9WATPVBktgOnDycDJWu9ZAh6nY7JXL Kci/BqJJMsb5tUV3YqtNL2Q/x7x0IZFV0DMecRpsvSf127dAPsa0FsRHtSaRyTX6RVBa49KwKbFkU f5rNyS8G8XvzWrM82W1/Q2pUh6uZ7SSeSYKMOKNY9ugxyEXvyyV/d2AZHHsChCjBAMmsdYuasMPTp Vl5cd+fqKKEiEuTe9gWblRPAtTOjqmUdvLcptOBXcBHu2oGQxRO2B6ohEXYnT7IbnHN/4TQrCzJnr kwc3bnog==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qCLdh-00FgfC-Ru; Thu, 22 Jun 2023 14:50:26 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3390C3006D5; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 1C78A209D8B3D; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Message-ID: <20230622144321.494426891@infradead.org> User-Agent: quilt/0.66 Date: Thu, 22 Jun 2023 16:42:21 +0200 From: Peter Zijlstra To: x86@kernel.org, alyssa.milburn@linux.intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, samitolvanen@google.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, brgerst@gmail.com Subject: [PATCH v2 3/6] x86/cfi: Extend ENDBR sealing to kCFI References: <20230622144218.860926475@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kees noted that IBT sealing could be extended to kCFI. Fundamentally it is the list of functions that do not have their address taken and are thus never called indirectly. It doesn't matter that objtool uses IBT infrastructure to determine this list, once we have it it can also be used to clobber kCFI hashes and seal kCFI indirect calls. Suggested-by: Kees Cook Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook Reviewed-by: Sami Tolvanen --- arch/x86/kernel/alternative.c | 44 +++++++++++++++++++++++++++++++++++++= ++++- 1 file changed, 43 insertions(+), 1 deletion(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -778,6 +778,8 @@ void __init_or_module noinline apply_ret =20 #ifdef CONFIG_X86_KERNEL_IBT =20 +static void poison_cfi(void *addr); + static void __init_or_module poison_endbr(void *addr, bool warn) { u32 endbr, poison =3D gen_endbr_poison(); @@ -802,6 +804,9 @@ static void __init_or_module poison_endb =20 /* * Generated by: objtool --ibt + * + * Seal the functions for indirect calls by clobbering the ENDBR instructi= ons + * and the kCFI hash value. */ void __init_or_module noinline apply_seal_endbr(s32 *start, s32 *end) { @@ -812,7 +817,7 @@ void __init_or_module noinline apply_sea =20 poison_endbr(addr, true); if (IS_ENABLED(CONFIG_FINEIBT)) - poison_endbr(addr - 16, false); + poison_cfi(addr - 16); } } =20 @@ -1177,6 +1182,41 @@ static void __apply_fineibt(s32 *start_r pr_err("Something went horribly wrong trying to rewrite the CFI implement= ation.\n"); } =20 +static inline void poison_hash(void *addr) +{ + *(u32 *)addr =3D 0; +} + +static void poison_cfi(void *addr) +{ + switch (cfi_mode) { + case CFI_FINEIBT: + /* + * __cfi_\func: + * osp nopl (%rax) + * subl $0, %r10d + * jz 1f + * ud2 + * 1: nop + */ + poison_endbr(addr, false); + poison_hash(addr + fineibt_preamble_hash); + break; + + case CFI_KCFI: + /* + * __cfi_\func: + * movl $0, %eax + * .skip 11, 0x90 + */ + poison_hash(addr + 1); + break; + + default: + break; + } +} + #else =20 static void __apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, @@ -1184,6 +1224,8 @@ static void __apply_fineibt(s32 *start_r { } =20 +static void poison_cfi(void *addr) { } + #endif =20 void apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, From nobody Sat Feb 7 17:19:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63388EB64D8 for ; Thu, 22 Jun 2023 14:50:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231905AbjFVOui (ORCPT ); Thu, 22 Jun 2023 10:50:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231661AbjFVOuf (ORCPT ); Thu, 22 Jun 2023 10:50:35 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A88CAE57 for ; Thu, 22 Jun 2023 07:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LGsdeHWEpXVTOOsTrfyXxkvr7QjEX8WnxEDVS86hVu0=; b=Vc1PCzlgv53QVpsR+WDW+RQqb6 hK7KNjrcjl+OM0sSOaXWHo1TLrdWF6Ft+sYwXsr6/tvfbojEmVncYsiUxhSWg/ccffx5Vp5h+E65l iTOfakYBU57ARcTK5pLhxLX1TwdYs+p+rSAi35g9L7X/LffRI+NRdtOrMWDspVfMgzmS1h0vWfayf 92OwwNzhL4uiLlFDCTrtaT0x3lah4hMBUKxB/5tLAOxbTKp31oapEesD7UR1+ROiQbXFVLC2HgORi SQp/YPBNcgr0rmbQmi37UXBKkT/rDdl8fS3inb9MhS5FPUIHGwQtdpek14S6NsTFIbr4tFvPPh9nA mhbSdVtQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qCLdh-001CzP-1B; Thu, 22 Jun 2023 14:50:25 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3999330075E; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 20EF6209D8B3E; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Message-ID: <20230622144321.561264520@infradead.org> User-Agent: quilt/0.66 Date: Thu, 22 Jun 2023 16:42:22 +0200 From: Peter Zijlstra To: x86@kernel.org, alyssa.milburn@linux.intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, samitolvanen@google.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, brgerst@gmail.com Subject: [PATCH v2 4/6] x86/32: Remove schedule_tail_wrapper() References: <20230622144218.860926475@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Brian Gerst The unwinder expects a return address at the very top of the kernel stack just below pt_regs and before any stack frame is created. Instead of calling a wrapper, set up a return address as if ret_from_fork() was called from the syscall entry code. Signed-off-by: Brian Gerst Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20230622120750.5549-2-brgerst@gmail.com Reviewed-by: Kees Cook Reviewed-by: Sami Tolvanen --- arch/x86/entry/entry_32.S | 32 ++++++++++---------------------- 1 file changed, 10 insertions(+), 22 deletions(-) --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -720,26 +720,6 @@ SYM_CODE_END(__switch_to_asm) .popsection =20 /* - * The unwinder expects the last frame on the stack to always be at the sa= me - * offset from the end of the page, which allows it to validate the stack. - * Calling schedule_tail() directly would break that convention because it= s an - * asmlinkage function so its argument has to be pushed on the stack. This - * wrapper creates a proper "end of stack" frame header before the call. - */ -.pushsection .text, "ax" -SYM_FUNC_START(schedule_tail_wrapper) - FRAME_BEGIN - - pushl %eax - call schedule_tail - popl %eax - - FRAME_END - RET -SYM_FUNC_END(schedule_tail_wrapper) -.popsection - -/* * A newly forked process directly context switches into this address. * * eax: prev task we switched from @@ -748,7 +728,13 @@ SYM_FUNC_END(schedule_tail_wrapper) */ .pushsection .text, "ax" SYM_CODE_START(ret_from_fork) - call schedule_tail_wrapper + /* return address for the stack unwinder */ + pushl $.Lsyscall_32_done + FRAME_BEGIN + + pushl %eax + call schedule_tail + addl $4, %esp =20 testl %ebx, %ebx jnz 1f /* kernel threads are uncommon */ @@ -757,7 +743,9 @@ SYM_CODE_START(ret_from_fork) /* When we fork, we trace the syscall return in the child, too. */ movl %esp, %eax call syscall_exit_to_user_mode - jmp .Lsyscall_32_done + + FRAME_END + RET =20 /* kernel thread */ 1: movl %edi, %eax From nobody Sat Feb 7 17:19:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F2E5EB64D8 for ; Thu, 22 Jun 2023 14:50:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231931AbjFVOun (ORCPT ); Thu, 22 Jun 2023 10:50:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231689AbjFVOuf (ORCPT ); Thu, 22 Jun 2023 10:50:35 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1C68E7E for ; Thu, 22 Jun 2023 07:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=MiQXciH6QFUFwWGYzaBS9NluMPpHBbDeC6AdeQsFuMU=; b=PtHif4EwoZQlLcVEpXtJWFpyNU GdRXNWxLm1HEbFx6U8QdziOOM12seJlecoI+n0+mMvvz+/ztcAigWigR+dwcmRxzZck1qORCqq8sT q9LWokguxzNnDdk1UiG+CModRKKD0c4DKPlJaFTWzm0C7l8IHM6hisuT5OhYEdnDwAXkW9f17rvQy mAu1O26d9XVgA8jeuPMbkBDAH+tKiaHWJLB1ejxjnjq2fbgxHbC/WyILXuGZu5R/oWOTCNeJ50V4x JeDs/HkDewn8XeSyWsT/9xXKCO8zkVNfxVgYDAq4vLVGHbgUSgZ5QalZci7uTz6TTRu6jQRYTgYgs UHklGcLg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qCLdh-001CzU-36; Thu, 22 Jun 2023 14:50:26 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 771CB300794; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 24D9B209D8B3C; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Message-ID: <20230622144321.629918727@infradead.org> User-Agent: quilt/0.66 Date: Thu, 22 Jun 2023 16:42:23 +0200 From: Peter Zijlstra To: x86@kernel.org, alyssa.milburn@linux.intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, samitolvanen@google.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, brgerst@gmail.com Subject: [PATCH v2 5/6] x86: Rewrite ret_from_fork() in C References: <20230622144218.860926475@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Brian Gerst When kCFI is enabled, special handling is needed for the indirect call to the kernel thread function. Rewrite the ret_from_fork() function in C so that the compiler can properly handle the indirect call. Suggested-by: Peter Zijlstra (Intel) Signed-off-by: Brian Gerst Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/20230622120750.5549-3-brgerst@gmail.com Reviewed-by: Kees Cook Reviewed-by: Sami Tolvanen --- arch/x86/entry/entry_32.S | 30 +++++++----------------------- arch/x86/entry/entry_64.S | 35 ++++++++++------------------------- arch/x86/include/asm/switch_to.h | 4 +++- arch/x86/kernel/process.c | 22 +++++++++++++++++++++- 4 files changed, 41 insertions(+), 50 deletions(-) --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -727,37 +727,21 @@ SYM_CODE_END(__switch_to_asm) * edi: kernel thread arg */ .pushsection .text, "ax" -SYM_CODE_START(ret_from_fork) +SYM_CODE_START(ret_from_fork_asm) /* return address for the stack unwinder */ pushl $.Lsyscall_32_done FRAME_BEGIN =20 - pushl %eax - call schedule_tail + /* prev already in EAX */ + movl %esp, %edx /* regs */ + movl %ebx, %ecx /* fn */ + pushl %edi /* fn_arg */ + call ret_from_fork addl $4, %esp =20 - testl %ebx, %ebx - jnz 1f /* kernel threads are uncommon */ - -2: - /* When we fork, we trace the syscall return in the child, too. */ - movl %esp, %eax - call syscall_exit_to_user_mode - FRAME_END RET - - /* kernel thread */ -1: movl %edi, %eax - CALL_NOSPEC ebx - /* - * A kernel thread is allowed to return here after successfully - * calling kernel_execve(). Exit to userspace to complete the execve() - * syscall. - */ - movl $0, PT_EAX(%esp) - jmp 2b -SYM_CODE_END(ret_from_fork) +SYM_CODE_END(ret_from_fork_asm) .popsection =20 SYM_ENTRY(__begin_SYSENTER_singlestep_region, SYM_L_GLOBAL, SYM_A_NONE) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -284,36 +284,21 @@ SYM_FUNC_END(__switch_to_asm) * r12: kernel thread arg */ .pushsection .text, "ax" - __FUNC_ALIGN -SYM_CODE_START_NOALIGN(ret_from_fork) +SYM_CODE_START(ret_from_fork_asm) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR // copy_thread CALL_DEPTH_ACCOUNT - movq %rax, %rdi - call schedule_tail /* rdi: 'prev' task parameter */ =20 - testq %rbx, %rbx /* from kernel_thread? */ - jnz 1f /* kernel threads are uncommon */ + /* return address for the stack unwinder */ + pushq $swapgs_restore_regs_and_return_to_usermode + UNWIND_HINT_FUNC =20 -2: - UNWIND_HINT_REGS - movq %rsp, %rdi - call syscall_exit_to_user_mode /* returns with IRQs disabled */ - jmp swapgs_restore_regs_and_return_to_usermode - -1: - /* kernel thread */ - UNWIND_HINT_END_OF_STACK - movq %r12, %rdi - CALL_NOSPEC rbx - /* - * A kernel thread is allowed to return here after successfully - * calling kernel_execve(). Exit to userspace to complete the execve() - * syscall. - */ - movq $0, RAX(%rsp) - jmp 2b -SYM_CODE_END(ret_from_fork) + movq %rax, %rdi /* prev */ + movq %rsp, %rsi /* regs */ + movq %rbx, %rdx /* fn */ + movq %r12, %rcx /* fn_arg */ + jmp ret_from_fork +SYM_CODE_END(ret_from_fork_asm) .popsection =20 .macro DEBUG_ENTRY_ASSERT_IRQS_OFF --- a/arch/x86/include/asm/switch_to.h +++ b/arch/x86/include/asm/switch_to.h @@ -12,7 +12,9 @@ struct task_struct *__switch_to_asm(stru __visible struct task_struct *__switch_to(struct task_struct *prev, struct task_struct *next); =20 -asmlinkage void ret_from_fork(void); +asmlinkage void ret_from_fork_asm(void); +__visible void ret_from_fork(struct task_struct *prev, struct pt_regs *reg= s, + int (*fn)(void *), void *fn_arg); =20 /* * This is the structure pointed to by thread.sp for an inactive task. The --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -134,6 +135,25 @@ static int set_new_tls(struct task_struc return do_set_thread_area_64(p, ARCH_SET_FS, tls); } =20 +__visible void ret_from_fork(struct task_struct *prev, struct pt_regs *reg= s, + int (*fn)(void *), void *fn_arg) +{ + schedule_tail(prev); + + /* Is this a kernel thread? */ + if (unlikely(fn)) { + fn(fn_arg); + /* + * A kernel thread is allowed to return here after successfully + * calling kernel_execve(). Exit to userspace to complete the + * execve() syscall. + */ + regs->ax =3D 0; + } + + syscall_exit_to_user_mode(regs); +} + int copy_thread(struct task_struct *p, const struct kernel_clone_args *arg= s) { unsigned long clone_flags =3D args->flags; @@ -149,7 +169,7 @@ int copy_thread(struct task_struct *p, c frame =3D &fork_frame->frame; =20 frame->bp =3D encode_frame_pointer(childregs); - frame->ret_addr =3D (unsigned long) ret_from_fork; + frame->ret_addr =3D (unsigned long) ret_from_fork_asm; p->thread.sp =3D (unsigned long) fork_frame; p->thread.io_bitmap =3D NULL; p->thread.iopl_warn =3D 0; From nobody Sat Feb 7 17:19:37 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A197EB64D8 for ; Thu, 22 Jun 2023 14:51:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231977AbjFVOvH (ORCPT ); Thu, 22 Jun 2023 10:51:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231913AbjFVOuj (ORCPT ); Thu, 22 Jun 2023 10:50:39 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2462FE7E for ; Thu, 22 Jun 2023 07:50:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=r2emQb1KTou2w3E9LTe7OUggXTTKEQJMuYJi6cUiILo=; b=ojMRDX4ZBEo3NUsxlNBhbZLvLt fkA47p2jhqe8lzrjurlZD2De4pLm2TykWc791yVF/zL0xFJOdfWxq++Veod7TQ+XhUdasBxrjX8nN OPAOuz0MCUzYEi5uA94oEvnGBs/JxIEIewilpDBPNdDupmEPS2IQkauXWZ1vv9BQ0B4FqjdrHr9/k uz//mQ/fof1UKUzFvRn3jAOYUN6rNV20xLBwELPIXwHorcoG27Mi7SDZOq2BCt3HWvJBLHzwNlkWM jBEnghKq3fIWQ5vkBpYn7CMxzJOb13s5sOeop9IS55tSD/6RdDNLVPyBCkK7oQ7E5V7V/TqIP49tf wSh7KL6A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qCLdh-00FgfE-V5; Thu, 22 Jun 2023 14:50:26 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 7716F300790; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 29E62209FB79B; Thu, 22 Jun 2023 16:50:24 +0200 (CEST) Message-ID: <20230622144321.696656240@infradead.org> User-Agent: quilt/0.66 Date: Thu, 22 Jun 2023 16:42:24 +0200 From: Peter Zijlstra To: x86@kernel.org, alyssa.milburn@linux.intel.com Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, samitolvanen@google.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, brgerst@gmail.com, "Milburn, Alyssa" Subject: [PATCH v2 6/6] x86/fineibt: Poison ENDBR at +0 References: <20230622144218.860926475@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Alyssa noticed that when building the kernel with CFI_CLANG+IBT and booting on IBT enabled hardware to obtain FineIBT, the indirect functions look like: __cfi_foo: endbr64 subl $hash, %r10d jz 1f ud2 nop 1: foo: endbr64 This is because the compiler generates code for kCFI+IBT. In that case the caller does the hash check and will jump to +0, so there must be an ENDBR there. The compiler doesn't know about FineIBT at all; also it is possible to actually use kCFI+IBT when booting with 'cfi=3Dkcfi' on IBT enabled hardware. Having this second ENDBR however makes it possible to elide the CFI check. Therefore, we should poison this second ENDBR when switching to FineIBT mode. Fixes: 931ab63664f0 ("x86/ibt: Implement FineIBT") Reported-by: "Milburn, Alyssa" Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Sami Tolvanen Acked-by: Kees Cook Link: https://lore.kernel.org/r/20230615193722.194131053@infradead.org Reviewed-by: Kees Cook --- arch/x86/kernel/alternative.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1063,6 +1063,17 @@ static int cfi_rewrite_preamble(s32 *sta return 0; } =20 +static void cfi_rewrite_endbr(s32 *start, s32 *end) +{ + s32 *s; + + for (s =3D start; s < end; s++) { + void *addr =3D (void *)s + *s; + + poison_endbr(addr+16, false); + } +} + /* .retpoline_sites */ static int cfi_rand_callers(s32 *start, s32 *end) { @@ -1157,14 +1168,19 @@ static void __apply_fineibt(s32 *start_r return; =20 case CFI_FINEIBT: + /* place the FineIBT preamble at func()-16 */ ret =3D cfi_rewrite_preamble(start_cfi, end_cfi); if (ret) goto err; =20 + /* rewrite the callers to target func()-16 */ ret =3D cfi_rewrite_callers(start_retpoline, end_retpoline); if (ret) goto err; =20 + /* now that nobody targets func()+0, remove ENDBR there */ + cfi_rewrite_endbr(start_cfi, end_cfi); + if (builtin) pr_info("Using FineIBT CFI\n"); return;