From nobody Thu Dec 18 07:09:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86377C2BB3F for ; Mon, 20 Nov 2023 14:38:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233479AbjKTOiK (ORCPT ); Mon, 20 Nov 2023 09:38:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229823AbjKTOh7 (ORCPT ); Mon, 20 Nov 2023 09:37:59 -0500 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A3EA112 for ; Mon, 20 Nov 2023 06:37:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=d5XfGL+IrDak//xTR1WiSXXdLz5PwCMSAL7n3tlPN8o=; b=SJQGJytk1fKS8n/+Ta28Kl7DBA N6y0osVoOkoHKZ9ODHFh0IrauX/uyuoK0Mvx+Fg+VQta3h4oGY2eJj3oU2kZ8IMLtBoLCwKLDkV7q qGHIjHN4vhag1fD48fsJXZLRCnjA3lb6oQWp2s9PM2TY9cmYVxtjWQzd902OrY8NxVdFY0FFpi5q6 r3xWR3NBT7C4+ofHYBbLPXxui+N7CDGcUjHwFgwtCYSWxROfPxTQpvRfUku79ETTBNg9imBw2o8z4 8OmqmYTZxN2HFJhhSvqmFsxBI3raHZN8WKN11TYINAyeD+oixoLP+zri+BP1FZ+m7KM6vLSor8hzd efcttDsw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1r55PM-00B09z-1p; Mon, 20 Nov 2023 14:37:52 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D107B3004E3; Mon, 20 Nov 2023 15:37:51 +0100 (CET) Message-Id: <20231120143626.638107480@infradead.org> User-Agent: quilt/0.65 Date: Mon, 20 Nov 2023 15:33:45 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org Subject: [PATCH 1/2] x86/entry: Optimize common_interrupt_return() References: <20231120143344.584345121@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The code in common_interrupt_return() does a bunch of unconditional work that is really only needed on PTI kernels. Specifically it unconditionally copies the IRET frame back onto the entry stack, swizzles onto the entry stack and does IRET from there. However, without PTI we can simply IRET from whatever stack we're on. ivb-ep, mitigations=3Doff, gettid-1m: PRE: 140,118,538 cycles:k = ( +- 0.01% ) 236,692,878 instructions:k # 1.69 insn per cycle= ( +- 0.00% ) POST: 140,026,608 cycles:k = ( +- 0.01% ) 236,696,176 instructions:k # 1.69 insn per cycle= ( +- 0.00% ) (this is with --repeat 100 and the run-to-run variance is bigger than the difference shown) Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Thomas Gleixner --- arch/x86/entry/calling.h | 12 +++++++++--- arch/x86/entry/entry_64.S | 17 +++++++++++++++-- 2 files changed, 24 insertions(+), 5 deletions(-) --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -175,8 +175,7 @@ For 32-bit we have the following convent #define THIS_CPU_user_pcid_flush_mask \ PER_CPU_VAR(cpu_tlbstate + TLB_STATE_user_pcid_flush_mask) =20 -.macro SWITCH_TO_USER_CR3_NOSTACK scratch_reg:req scratch_reg2:req - ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI +.macro SWITCH_TO_USER_CR3 scratch_reg:req scratch_reg2:req mov %cr3, \scratch_reg =20 ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID @@ -206,13 +205,20 @@ For 32-bit we have the following convent /* Flip the PGD to the user version */ orq $(PTI_USER_PGTABLE_MASK), \scratch_reg mov \scratch_reg, %cr3 +.endm + +.macro SWITCH_TO_USER_CR3_NOSTACK scratch_reg:req scratch_reg2:req + ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI + SWITCH_TO_USER_CR3 \scratch_reg \scratch_reg2 .Lend_\@: .endm =20 .macro SWITCH_TO_USER_CR3_STACK scratch_reg:req + ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI pushq %rax - SWITCH_TO_USER_CR3_NOSTACK scratch_reg=3D\scratch_reg scratch_reg2=3D%rax + SWITCH_TO_USER_CR3 scratch_reg=3D\scratch_reg scratch_reg2=3D%rax popq %rax +.Lend_\@: .endm =20 .macro SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg:req save_reg:req --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -569,7 +569,18 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_ #ifdef CONFIG_XEN_PV ALTERNATIVE "", "jmp xenpv_restore_regs_and_return_to_usermode", X86_FEAT= URE_XENPV #endif +#ifdef CONFIG_PAGE_TABLE_ISOLATION + ALTERNATIVE "", "jmp .Lpti_restore_regs_and_return_to_usermode", X86_FEAT= URE_PTI +#endif + + STACKLEAK_ERASE + POP_REGS + add $8, %rsp /* orig_ax */ + swapgs + jmp .Lnative_iret =20 +#ifdef CONFIG_PAGE_TABLE_ISOLATION +.Lpti_restore_regs_and_return_to_usermode: POP_REGS pop_rdi=3D0 =20 /* @@ -596,13 +607,15 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_ */ STACKLEAK_ERASE_NOCLOBBER =20 - SWITCH_TO_USER_CR3_STACK scratch_reg=3D%rdi + push %rax + SWITCH_TO_USER_CR3 scratch_reg=3D%rdi scratch_reg2=3D%rax + pop %rax =20 /* Restore RDI. */ popq %rdi swapgs jmp .Lnative_iret - +#endif =20 SYM_INNER_LABEL(restore_regs_and_return_to_kernel, SYM_L_GLOBAL) #ifdef CONFIG_DEBUG_ENTRY From nobody Thu Dec 18 07:09:30 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFF54C2BB3F for ; Mon, 20 Nov 2023 14:38:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233191AbjKTOiD (ORCPT ); Mon, 20 Nov 2023 09:38:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232227AbjKTOh6 (ORCPT ); Mon, 20 Nov 2023 09:37:58 -0500 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AC2A110 for ; Mon, 20 Nov 2023 06:37:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Uh5WSMt/ka3TMy8zfdv/20yKPi+OXOk7N9AxmMFqyLk=; b=Fb2800RwQnF0EXjquKGxGkXtLR buQDZLc3i7Quns6BiCSAbKLaIzRh/62W+0H+UD2s9lUyo9zsVc9FGWQgs7sczVSoR5/4EbDxfSxub zBgjVIHN1j+6e0o+1V6d5otGsRZkP93CATfK1t5IHWSrN2zevwhDjBiKuKxF462aWnnXsVZ8lwhSL zfeCb1E6/aPx6/2F3Z82zjiJQH/z8xOehDfIH3VoYII8ArEVHg0kBMpSm/nor+6Hi9Yfg1Jbu391T b4L4cUB/yOuLxuG3D5RM0H3s61BJMotnPWgR1hcVGCKqo2qjO5//RNgoPthOeRpTu1sBpRES2i9I+ lAyCpCkw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1r55PM-00B0A0-1p; Mon, 20 Nov 2023 14:37:52 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D60333007C8; Mon, 20 Nov 2023 15:37:51 +0100 (CET) Message-Id: <20231120143626.753200755@infradead.org> User-Agent: quilt/0.65 Date: Mon, 20 Nov 2023 15:33:46 +0100 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org Subject: [PATCH 2/2] x86/entry: Harden return-to-user References: <20231120143344.584345121@infradead.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make the DEBUG_ENTRY check that validates CS is a user segment unconditional and move it nearer to IRET. PRE: 140,026,608 cycles:k = ( +- 0.01% ) 236,696,176 instructions:k # 1.69 insn per cycle= ( +- 0.00% ) POST: 139,957,681 cycles:k = ( +- 0.01% ) 236,681,819 instructions:k # 1.69 insn per cycle= ( +- 0.00% ) (this is with --repeat 100 and the run-to-run variance is bigger than the difference shown) Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Thomas Gleixner --- arch/x86/entry/entry_64.S | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -559,13 +559,6 @@ SYM_CODE_END(\asmsym) SYM_CODE_START_LOCAL(common_interrupt_return) SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) IBRS_EXIT -#ifdef CONFIG_DEBUG_ENTRY - /* Assert that pt_regs indicates user mode. */ - testb $3, CS(%rsp) - jnz 1f - ud2 -1: -#endif #ifdef CONFIG_XEN_PV ALTERNATIVE "", "jmp xenpv_restore_regs_and_return_to_usermode", X86_FEAT= URE_XENPV #endif @@ -576,8 +569,14 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_ STACKLEAK_ERASE POP_REGS add $8, %rsp /* orig_ax */ + UNWIND_HINT_IRET_REGS + +.Lswapgs_and_iret: swapgs - jmp .Lnative_iret + /* Assert that the IRET frame indicates user mode. */ + testb $3, 8(%rsp) + jnz .Lnative_iret + ud2 =20 #ifdef CONFIG_PAGE_TABLE_ISOLATION .Lpti_restore_regs_and_return_to_usermode: @@ -613,8 +612,7 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_ =20 /* Restore RDI. */ popq %rdi - swapgs - jmp .Lnative_iret + jmp .Lswapgs_and_iret #endif =20 SYM_INNER_LABEL(restore_regs_and_return_to_kernel, SYM_L_GLOBAL)