From nobody Mon Jun 22 23:55:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13B5FC433EF for ; Tue, 15 Mar 2022 07:39:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345526AbiCOHkg (ORCPT ); Tue, 15 Mar 2022 03:40:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345470AbiCOHka (ORCPT ); Tue, 15 Mar 2022 03:40:30 -0400 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EED74B429 for ; Tue, 15 Mar 2022 00:39:18 -0700 (PDT) Received: by mail-pf1-x42a.google.com with SMTP id g19so18286733pfc.9 for ; Tue, 15 Mar 2022 00:39:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8NPns1qVGLmvkhCunAKngrwBXpGFAkcIYE6CmNe5pkc=; b=JGe/DLSG6eqrdvTG8fcOiSIyn82CHHbL8KhES5Nq0BTlkRWtKZz42wqPmuADIR42qh 5y1sps6OKB2V1Jn60/6fCDVBBvJLT/0yJoD53gKSpw6kcR9AqMTDBAV9+rSnbvo20Srq QVFtdGCzG3u1FRhlyrDVIf4Pt+58MZdsvmF0If1msEvVkdYtqd8T+ORCp8Vy+BVp1LnS nPWttS4NNHsvW4SkUeL/hZRl6SiLhDZvrc3KIqAJkL0CjU1Okk4ieA24Q99GFcZYjIIS jV2k8QuoHxS9SRSCMvJ+lvD9x0zvYdaT6HgC61uxbwHNhxZ0lwoVJrpXNLS2E3f8oOKD GLAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8NPns1qVGLmvkhCunAKngrwBXpGFAkcIYE6CmNe5pkc=; b=3rdc9bbvvJlLkk/SFeKA5vbWvjrbaRrQwXpvxBwkQG6JOD98LYgIZk98fwuczTccG/ Upt1BGYparCt2ZetZkMm+KIfaKZcnHhbb9LwA84EVJxeiGMagsYxjZpMQjOds0oVyDBG CpYIhNBAJxFikmM7ewGRrtgjvVJGRQ62UxVR9WAqcFibsVa419omHaNIW/UwbWxoalFQ 9yn6owNuHRFOaPCG88DT9LXgthSONVxjvI4t2HCoHFyDgR8ylGvq/CK3oK/6DLwaxYvy rYnKP6B6o55mmGrbbexKQXqBcGWf12ZeCLoRrTAeEgQVGRPa46nx/bZDCEXufNM0BA5+ u5mA== X-Gm-Message-State: AOAM53218Dc30ThRFEpBxjvso3815jETvZ2/znogwB5Irke3K5B34fmd cibfep521vohpN3BbbzQ7aENUdzEw7c= X-Google-Smtp-Source: ABdhPJzNzO6S1o6kWfSIb/cwHRV1pPwYVnGSB4Humpl+LZqgbQmqIhVIHG0oH9LtIKGsapzvbyZupA== X-Received: by 2002:a05:6a00:1350:b0:4f7:8c4f:cfca with SMTP id k16-20020a056a00135000b004f78c4fcfcamr20267920pfu.45.1647329957729; Tue, 15 Mar 2022 00:39:17 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id b17-20020a056a000a9100b004e1b7cdb8fdsm24617305pfl.70.2022.03.15.00.39.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2022 00:39:17 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Borislav Petkov , Peter Zijlstra , Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Dave Hansen , "H. Peter Anvin" , Josh Poimboeuf , Joerg Roedel , "Chang S. Bae" , Jan Kiszka Subject: [PATCH V3 1/7] x86/entry: Use idtentry macro for entry_INT80_compat Date: Tue, 15 Mar 2022 15:39:43 +0800 Message-Id: <20220315073949.7541-2-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220315073949.7541-1-jiangshanlai@gmail.com> References: <20220315073949.7541-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan entry_INT80_compat is identical to idtentry macro except a special handling for %rax in the prolog. Add the prolog to idtentry and use idtentry for entry_INT80_compat. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 18 ++++++ arch/x86/entry/entry_64_compat.S | 103 ------------------------------- arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/proto.h | 4 -- 4 files changed, 65 insertions(+), 107 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index cb5b7348b9b8..47ed97cbf46c 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -360,6 +360,24 @@ SYM_CODE_START(\asmsym) pushq $-1 /* ORIG_RAX: no syscall to restart */ .endif =20 + .if \vector =3D=3D IA32_SYSCALL_VECTOR + /* + * User tracing code (ptrace or signal handlers) might assume + * that the saved RAX contains a 32-bit number when we're + * invoking a 32-bit syscall. Just in case the high bits are + * nonzero, zero-extend the syscall number. (This could almost + * certainly be deleted with no ill effects.) + */ + movl %eax, %eax + + /* + * do_int80_syscall_32() expects regs->orig_ax to be user ax, + * and regs->ax to be $-ENOSYS. + */ + movq %rax, (%rsp) + movq $-ENOSYS, %rax + .endif + .if \vector =3D=3D X86_TRAP_BP /* * If coming from kernel space, create a 6-word gap to allow the diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_com= pat.S index 4fdb007cddbd..2b7496a36f65 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -315,106 +315,3 @@ sysret32_from_system_call: swapgs sysretl SYM_CODE_END(entry_SYSCALL_compat) - -/* - * 32-bit legacy system call entry. - * - * 32-bit x86 Linux system calls traditionally used the INT $0x80 - * instruction. INT $0x80 lands here. - * - * This entry point can be used by 32-bit and 64-bit programs to perform - * 32-bit system calls. Instances of INT $0x80 can be found inline in - * various programs and libraries. It is also used by the vDSO's - * __kernel_vsyscall fallback for hardware that doesn't support a faster - * entry method. Restarted 32-bit system calls also fall back to INT - * $0x80 regardless of what instruction was originally used to do the - * system call. - * - * This is considered a slow path. It is not used by most libc - * implementations on modern hardware except during process startup. - * - * Arguments: - * eax system call number - * ebx arg1 - * ecx arg2 - * edx arg3 - * esi arg4 - * edi arg5 - * ebp arg6 - */ -SYM_CODE_START(entry_INT80_compat) - UNWIND_HINT_EMPTY - ENDBR - /* - * Interrupts are off on entry. - */ - ASM_CLAC /* Do this early to minimize exposure */ - SWAPGS - - /* - * User tracing code (ptrace or signal handlers) might assume that - * the saved RAX contains a 32-bit number when we're invoking a 32-bit - * syscall. Just in case the high bits are nonzero, zero-extend - * the syscall number. (This could almost certainly be deleted - * with no ill effects.) - */ - movl %eax, %eax - - /* switch to thread stack expects orig_ax and rdi to be pushed */ - pushq %rax /* pt_regs->orig_ax */ - pushq %rdi /* pt_regs->di */ - - /* Need to switch before accessing the thread stack. */ - SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdi - - /* In the Xen PV case we already run on the thread stack. */ - ALTERNATIVE "", "jmp .Lint80_keep_stack", X86_FEATURE_XENPV - - movq %rsp, %rdi - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - - pushq 6*8(%rdi) /* regs->ss */ - pushq 5*8(%rdi) /* regs->rsp */ - pushq 4*8(%rdi) /* regs->eflags */ - pushq 3*8(%rdi) /* regs->cs */ - pushq 2*8(%rdi) /* regs->ip */ - pushq 1*8(%rdi) /* regs->orig_ax */ - pushq (%rdi) /* pt_regs->di */ -.Lint80_keep_stack: - - pushq %rsi /* pt_regs->si */ - xorl %esi, %esi /* nospec si */ - pushq %rdx /* pt_regs->dx */ - xorl %edx, %edx /* nospec dx */ - pushq %rcx /* pt_regs->cx */ - xorl %ecx, %ecx /* nospec cx */ - pushq $-ENOSYS /* pt_regs->ax */ - pushq %r8 /* pt_regs->r8 */ - xorl %r8d, %r8d /* nospec r8 */ - pushq %r9 /* pt_regs->r9 */ - xorl %r9d, %r9d /* nospec r9 */ - pushq %r10 /* pt_regs->r10*/ - xorl %r10d, %r10d /* nospec r10 */ - pushq %r11 /* pt_regs->r11 */ - xorl %r11d, %r11d /* nospec r11 */ - pushq %rbx /* pt_regs->rbx */ - xorl %ebx, %ebx /* nospec rbx */ - pushq %rbp /* pt_regs->rbp */ - xorl %ebp, %ebp /* nospec rbp */ - pushq %r12 /* pt_regs->r12 */ - xorl %r12d, %r12d /* nospec r12 */ - pushq %r13 /* pt_regs->r13 */ - xorl %r13d, %r13d /* nospec r13 */ - pushq %r14 /* pt_regs->r14 */ - xorl %r14d, %r14d /* nospec r14 */ - pushq %r15 /* pt_regs->r15 */ - xorl %r15d, %r15d /* nospec r15 */ - - UNWIND_HINT_REGS - - cld - - movq %rsp, %rdi - call do_int80_syscall_32 - jmp swapgs_restore_regs_and_return_to_usermode -SYM_CODE_END(entry_INT80_compat) diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentr= y.h index 7924f27f5c8b..fac5db38c895 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -206,6 +206,20 @@ __visible noinstr void func(struct pt_regs *regs, \ \ static noinline void __##func(struct pt_regs *regs, u32 vector) =20 +/** + * DECLARE_IDTENTRY_IA32_EMULATION - Declare functions for int80 + * @vector: Vector number (ignored for C) + * @asm_func: Function name of the entry point + * @cfunc: The C handler called from the ASM entry point (ignored for C) + * + * Declares two functions: + * - The ASM entry point: asm_func + * - The XEN PV trap entry point: xen_##asm_func (maybe unused) + */ +#define DECLARE_IDTENTRY_IA32_EMULATION(vector, asm_func, cfunc) \ + asmlinkage void asm_func(void); \ + asmlinkage void xen_##asm_func(void) + /** * DECLARE_IDTENTRY_SYSVEC - Declare functions for system vector entry poi= nts * @vector: Vector number (ignored for C) @@ -432,6 +446,35 @@ __visible noinstr void func(struct pt_regs *regs, \ #define DECLARE_IDTENTRY_ERRORCODE(vector, func) \ idtentry vector asm_##func func has_error_code=3D1 =20 +/* + * 32-bit legacy system call entry. + * + * 32-bit x86 Linux system calls traditionally used the INT $0x80 + * instruction. INT $0x80 lands here. + * + * This entry point can be used by 32-bit and 64-bit programs to perform + * 32-bit system calls. Instances of INT $0x80 can be found inline in + * various programs and libraries. It is also used by the vDSO's + * __kernel_vsyscall fallback for hardware that doesn't support a faster + * entry method. Restarted 32-bit system calls also fall back to INT + * $0x80 regardless of what instruction was originally used to do the + * system call. + * + * This is considered a slow path. It is not used by most libc + * implementations on modern hardware except during process startup. + * + * Arguments: + * eax system call number + * ebx arg1 + * ecx arg2 + * edx arg3 + * esi arg4 + * edi arg5 + * ebp arg6 + */ +#define DECLARE_IDTENTRY_IA32_EMULATION(vector, asm_func, cfunc) \ + idtentry vector asm_func cfunc has_error_code=3D0 + /* Special case for 32bit IRET 'trap'. Do not emit ASM code */ #define DECLARE_IDTENTRY_SW(vector, func) =20 @@ -638,6 +681,10 @@ DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER, common_interrupt); DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER, spurious_interrupt); #endif =20 +#ifdef CONFIG_IA32_EMULATION +DECLARE_IDTENTRY_IA32_EMULATION(IA32_SYSCALL_VECTOR, entry_INT80_compat, d= o_int80_syscall_32); +#endif + /* System vector entry points */ #ifdef CONFIG_X86_LOCAL_APIC DECLARE_IDTENTRY_SYSVEC(ERROR_APIC_VECTOR, sysvec_error_interrupt); diff --git a/arch/x86/include/asm/proto.h b/arch/x86/include/asm/proto.h index feed36d44d04..c4d331fe65ff 100644 --- a/arch/x86/include/asm/proto.h +++ b/arch/x86/include/asm/proto.h @@ -28,10 +28,6 @@ void entry_SYSENTER_compat(void); void __end_entry_SYSENTER_compat(void); void entry_SYSCALL_compat(void); void entry_SYSCALL_compat_safe_stack(void); -void entry_INT80_compat(void); -#ifdef CONFIG_XEN_PV -void xen_entry_INT80_compat(void); -#endif #endif =20 void x86_configure_nx(void); --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 23:55:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81DFDC433F5 for ; Tue, 15 Mar 2022 07:39:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345540AbiCOHkr (ORCPT ); Tue, 15 Mar 2022 03:40:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345543AbiCOHkm (ORCPT ); Tue, 15 Mar 2022 03:40:42 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AEDA4B84F for ; Tue, 15 Mar 2022 00:39:29 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id u2so581107ple.10 for ; Tue, 15 Mar 2022 00:39:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yOyx9E8tfL4n6GqsVbEY5LNO7I69yL8/5cHJdSeUlTc=; b=A37fPEIAk9vlJKK8W1H1KcJvNWGCYODPXC+G8E1xOjSCPugsbEaIR3GXp8YgdYULCm CsKJFVU/2Zj2Y9G0lyfgorVMQ5A6b/iuUi7oM6r9oiHtBgWoIjkFdHzGGjNb2wDwtexN RxhdoaYc/Bam9UU8SQkKC5PyNZmuCGOOy4E/cbKVPfTmzik35OVCoDQ+tneYR58DfD7f Kcn35qwCOMXkwWjaJhgE8jxMEn42pmfEOjavLGktAJSf22XJg1QvvRdLrAdfw3FDvHpN XK8dooK7uFgWGqaZPwMX6EypS2c/1kVrBIc1ntucJqh+V5ngqQPcX4KGP6xPngEZT2cj VOMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yOyx9E8tfL4n6GqsVbEY5LNO7I69yL8/5cHJdSeUlTc=; b=LgSKK2Uq0geRCKb7NSaHFL4+RYw2UH0E7p66bwlRxVIN7XB3jnb8nPviGbxYqBCA+z IJNAoiEvkr7188DfbkX/uy+oFv1ZxAqzSaggEHuCjOKqYJ0VwEfyEYlx3UZfPN8vHF+E mDfMpw4Sr+1Ip1C1HGZTbi4XHhytLatWWzKv+ydN0rd2EfJpsh+2++NQouqB6uEYmTbn +34s2XFYUlPa64fONhW1U7l41sq2Mn7nqhctKCcZoC1KZr07+QIql9z2ObCO9Sxx4AGu oZuI2PVKY46EtlQneNbyBp3CBCrw4fMpfq6J15lo7X8IySy4HfeKtL7aTh4/Np5deZZ2 U+2g== X-Gm-Message-State: AOAM530EvqCi9aZwP0+Nw5jKMnpotudxCQW419AmNZnjEgD31SRpyn8e qO+0kgl8DhBQuxOZjWWRFpacdrdGEiI= X-Google-Smtp-Source: ABdhPJyL/ka9uuvEAKTGTaOjlN+mcOao1BFGRvE5QVkhp1rC74coMo131TFXLbXJNB/MrTyISoT82g== X-Received: by 2002:a17:902:8492:b0:14d:5ddc:9df6 with SMTP id c18-20020a170902849200b0014d5ddc9df6mr26482713plo.15.1647329968339; Tue, 15 Mar 2022 00:39:28 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id j8-20020a056a00174800b004f6f3bfd054sm24443381pfc.94.2022.03.15.00.39.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2022 00:39:28 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Borislav Petkov , Peter Zijlstra , Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Dave Hansen , "H. Peter Anvin" , Josh Poimboeuf , Fenghua Yu , Thomas Tai , "Chang S. Bae" , Masami Hiramatsu Subject: [PATCH V3 2/7] x86/traps: Move pt_regs only in fixup_bad_iret() Date: Tue, 15 Mar 2022 15:39:44 +0800 Message-Id: <20220315073949.7541-3-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220315073949.7541-1-jiangshanlai@gmail.com> References: <20220315073949.7541-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan fixup_bad_iret() and sync_regs() have similar arguments and do similar work that copies full or partial pt_regs to a place and switches stack after return. They are quite the same, but fixup_bad_iret() not only copies the pt_regs but also the return address of error_entry() while sync_regs() copies the pt_regs only and the return address of error_entry() was preserved and handled in ASM code. This patch makes fixup_bad_iret() work like sync_regs() and the handling of the return address of error_entry() is moved in ASM code. It removes the need to use the struct bad_iret_stack, simplifies fixup_bad_iret() and makes the ASM error_entry() call fixup_bad_iret() as the same as calling sync_regs() which adds readability because the calling patterns are exactly the same. It is prepared for later patch to do the stack switch after the error_entry() which simplifies the code further. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 5 ++++- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 17 ++++++----------- 3 files changed, 11 insertions(+), 13 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 47ed97cbf46c..37505331b7f1 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1073,9 +1073,12 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - mov %rsp, %rdi + popq %r12 /* save return addr in %12 */ + movq %rsp, %rdi /* arg0 =3D pt_regs pointer */ call fixup_bad_iret mov %rax, %rsp + ENCODE_FRAME_POINTER + pushq %r12 jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry) =20 diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 35317c5c551d..47ecfff2c83d 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -13,7 +13,7 @@ #ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *ere= gs); asmlinkage __visible notrace -struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s); +struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs); void __init trap_init(void); asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_r= egs *eregs); #endif diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 1563fb995005..9fe9cd9d3eeb 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -892,13 +892,8 @@ asmlinkage __visible noinstr struct pt_regs *vc_switch= _off_ist(struct pt_regs *r } #endif =20 -struct bad_iret_stack { - void *error_entry_ret; - struct pt_regs regs; -}; - asmlinkage __visible noinstr -struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s) +struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs) { /* * This is called from entry_64.S early in handling a fault @@ -908,19 +903,19 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret= _stack *s) * just below the IRET frame) and we want to pretend that the * exception came from the IRET target. */ - struct bad_iret_stack tmp, *new_stack =3D - (struct bad_iret_stack *)__this_cpu_read(cpu_tss_rw.x86_tss.sp0) - 1; + struct pt_regs tmp, *new_stack =3D + (struct pt_regs *)__this_cpu_read(cpu_tss_rw.x86_tss.sp0) - 1; =20 /* Copy the IRET target to the temporary storage. */ - __memcpy(&tmp.regs.ip, (void *)s->regs.sp, 5*8); + __memcpy(&tmp.ip, (void *)bad_regs->sp, 5*8); =20 /* Copy the remainder of the stack from the current stack. */ - __memcpy(&tmp, s, offsetof(struct bad_iret_stack, regs.ip)); + __memcpy(&tmp, bad_regs, offsetof(struct pt_regs, ip)); =20 /* Update the entry stack */ __memcpy(new_stack, &tmp, sizeof(tmp)); =20 - BUG_ON(!user_mode(&new_stack->regs)); + BUG_ON(!user_mode(new_stack)); return new_stack; } #endif --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 23:55:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93025C433EF for ; Tue, 15 Mar 2022 07:39:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345550AbiCOHkw (ORCPT ); Tue, 15 Mar 2022 03:40:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345537AbiCOHkq (ORCPT ); Tue, 15 Mar 2022 03:40:46 -0400 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DFE64B436 for ; Tue, 15 Mar 2022 00:39:35 -0700 (PDT) Received: by mail-pl1-x62d.google.com with SMTP id d18so435506plr.6 for ; Tue, 15 Mar 2022 00:39:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J0gDUXrqZ+nZHPdsNhy8A8CfRPW/VJ+6u8PwNWHKT0g=; b=gzDJU2XyoAB7vgBY0WkAweA4XmIRJiiuz9wDjw5nnuFd3BB3OH6IZiTEnC5ZlYvHkv QJrHKsEjrStcKjUkI0bOCnFB2UJ34eK1vlp9a5RhxXosHaFewLS5D5Gfu2LA0qd64UuB /0xAr/oL5X/bESct9B8KapvEzJMT/87Kah+VaKAmWhYKQTvzy/LWBU7mRDxmsK8IpskV EozS+jKKPYtF6Wx9msUbs16zg6TnBvADPHlA75h5FyLUnCxMs7UZwjJ7t1L76AgjHNdQ XfBy9hGgxsffWioSrPTaQIKdL0ncbQBRCBr4rmcbUp7pEnQFUAANnV8F4CS1FiamkJiX 8lxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J0gDUXrqZ+nZHPdsNhy8A8CfRPW/VJ+6u8PwNWHKT0g=; b=epjxblG9l3m43fb7UVnu7JDRM2hI2HYKks39ATx4rxHA/bg6/MnT4kL783qnIJi4r3 VL+1impAHW8l7EtcNZdKmCai0C9UUS6t1Ahs6gUVFZMxQNW2ko5RlGfokSlHnSZ6Rp6T qNAOqcxGzBSw/YvHpbGdz72lXOxGUmukNKylmWlNpW+4B28Vqi7MAX1E8U1Gqb7KQwCo oGMYToJb4AT6IHjKaG8fmcjZ9Nsrz4me+lmAump8KmbpTrUTgQxYebb9KSRN6MncxxJd Epm0POdEWpnTGmrS6JzaDOUZSSNpcKOl2bSsyj92+W+6SYoMMhe+0X+O/YYPlfv8E3Uf 7Obg== X-Gm-Message-State: AOAM533/GtMlLdWhEGLhLaQzHEECdTGNfO62eng4SCMePR7o/+rHbIES NaSCQJ/EMszS3+hnlvSty7J/sQGGBag= X-Google-Smtp-Source: ABdhPJwM5pMM7p6QT9TCfZfJ5nqAqoYAOrmb1vwPZzsp0gHOvvoR/3Qy1mlHRG8/JWxl93tlpe/G3Q== X-Received: by 2002:a17:90b:1d07:b0:1c6:35d1:4416 with SMTP id on7-20020a17090b1d0700b001c635d14416mr2357341pjb.210.1647329974849; Tue, 15 Mar 2022 00:39:34 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id p186-20020a62d0c3000000b004f6fa49c4b9sm21834511pfg.218.2022.03.15.00.39.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2022 00:39:34 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Borislav Petkov , Peter Zijlstra , Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V3 3/7] x86/entry: Switch the stack after error_entry() returns Date: Tue, 15 Mar 2022 15:39:45 +0800 Message-Id: <20220315073949.7541-4-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220315073949.7541-1-jiangshanlai@gmail.com> References: <20220315073949.7541-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan error_entry() calls sync_regs() to settle/copy the pt_regs and switches the stack directly after sync_regs(). But error_entry() itself is also a function call, the switching has to handle the return address of it together, which causes the work complicated and tangly. Switching to the stack after error_entry() makes the code simpler and intuitive. The behavior/logic is unchanged: 1) (opt) feed fixup_bad_iret() with the pt_regs pushed by ASM code 2) (opt) fixup_bad_iret() moves the partial pt_regs up 3) feed sync_regs() with the pt_regs pushed by ASM code or returned by fixup_bad_iret() 4) sync_regs() copies the whole pt_regs to kernel stack if needed 5) after error_entry() and switching %rsp, it is in kernel stack with the pt_regs Changes only in calling: Old code switches to copied pt_regs immediately twice in error_entry() while new code switches to the copied pt_regs only once after error_entry() returns. It is correct since sync_regs() doesn't need to be called close to the pt_regs it handles. Old code stashes the return-address of error_entry() in a scratch register and new code doesn't stash it. It relies on the fact that fixup_bad_iret() and sync_regs() don't corrupt the return-address of error_entry() on the stack. But even the old code also relies on the fact that fixup_bad_iret() and sync_regs() don't corrupt the return-address of themselves. They are the same reliances and are assured. After this change, error_entry() will not do fancy things with the stack except when in the prolog which will be fixed in the next patch ("move PUSH_AND_CLEAR_REGS out of error_entry"). This patch and the next patch can't be swapped because the next patch relies on this patch's stopping fiddling with the return-address of error_entry(), otherwise the objtool would complain. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 37505331b7f1..7768cdd0c7ed 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -326,6 +326,8 @@ SYM_CODE_END(ret_from_fork) .macro idtentry_body cfunc has_error_code:req =20 call error_entry + movq %rax, %rsp /* switch stack settled by sync_regs() */ + ENCODE_FRAME_POINTER UNWIND_HINT_REGS =20 movq %rsp, %rdi /* pt_regs pointer into 1st argument*/ @@ -1014,14 +1016,10 @@ SYM_CODE_START_LOCAL(error_entry) /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax =20 + leaq 8(%rsp), %rdi /* arg0 =3D pt_regs pointer */ .Lerror_entry_from_usermode_after_swapgs: /* Put us onto the real thread stack. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 =3D pt_regs pointer */ call sync_regs - movq %rax, %rsp /* switch stack */ - ENCODE_FRAME_POINTER - pushq %r12 RET =20 /* @@ -1053,6 +1051,7 @@ SYM_CODE_START_LOCAL(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + leaq 8(%rsp), %rax /* return pt_regs pointer */ RET =20 .Lbstep_iret: @@ -1073,12 +1072,9 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 =3D pt_regs pointer */ + leaq 8(%rsp), %rdi /* arg0 =3D pt_regs pointer */ call fixup_bad_iret - mov %rax, %rsp - ENCODE_FRAME_POINTER - pushq %r12 + mov %rax, %rdi jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry) =20 --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 23:55:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF907C433EF for ; Tue, 15 Mar 2022 07:39:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345561AbiCOHlB (ORCPT ); Tue, 15 Mar 2022 03:41:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345553AbiCOHkx (ORCPT ); Tue, 15 Mar 2022 03:40:53 -0400 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E529A4B851 for ; Tue, 15 Mar 2022 00:39:41 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id h2so14296743pfh.6 for ; Tue, 15 Mar 2022 00:39:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eTZW2d3yB5B8DbC38GNqbyFxqNMHtItePfoOGslF2jk=; b=mq+zMg5Y0ewmVVR1gqqxY755c3OnalZcRpjWYTE+TdrrWHYDOR6Elc0Jah1kopG8nv AMvvr9MECUGJm4o//KpujMMtL5KjmwuDrXzgZ1bkkrinkwshNDxrTH1Ftyn8FAC8Urgm 3MutFxWv4ij79GG6UTIuqRz262OENhcgy5wkycNgh3Q+jogCQVeidS3cTnwDAOPCcgA/ 9w2i17sRx5eyukUC8BbsohdPQ4ybovIswXRjyvoBDZv2KdHub2CUqxvRd82Aw4nsRrRB veRBYRyQ+5aAvt8sMTLOyQnWszKjdjJlRiRBtJBtfkV1fle/UDEhAKZtd7xdjcWDMxGy gnoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eTZW2d3yB5B8DbC38GNqbyFxqNMHtItePfoOGslF2jk=; b=lhSOkHqXl8Ye3BAYYqmYa8FlonMrzsH0XTJUsk3oKhtjMiCjNtKCVxm/YV+PHAaZnC VnntDgfWafPXPUdB7hJ6diBPZ/t+NnIOypZg3UCNYaHUbemq/i3qNa7tbJ8jdmai4ohq 4G95Y7HHuTLFgsiuizfA+Xs2MFs4n2tBlkz1YXTYxMxP+dIPUEpc6MD4mdTPqpXI3KEa ZtVDu8qDT6+3PO4Fe57m8RjgBW3PhsIUlO4bqjuSzU8Ci7xSNQibD/Jsno9m8W0dsSJm PAlSZ4kI4B2PKF3eylPkXfv4DG2tmFjG5X/GjGvIbxKglyfoUkCvJ0vDFlik2myyo88A b5YQ== X-Gm-Message-State: AOAM533fKjng52DTFa4fBS8RAy+8NJ12R5ow6H6HY1ial+FDfzBx3iin kk+HpNeN/jeg+TuCyfQJ3mBU751fZRU= X-Google-Smtp-Source: ABdhPJzHVTdtdIpMM5t5w4t0iJJrZA9zZRLjh0DhvYt3ZgCmfQmyNakpfba3S8i5UcwrV3U62ZZx5A== X-Received: by 2002:a05:6a00:22c1:b0:4f7:3e72:d99b with SMTP id f1-20020a056a0022c100b004f73e72d99bmr27663638pfj.79.1647329981058; Tue, 15 Mar 2022 00:39:41 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id m11-20020a17090a3f8b00b001bc299e0aefsm1795812pjc.56.2022.03.15.00.39.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2022 00:39:40 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Borislav Petkov , Peter Zijlstra , Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V3 4/7] x86/entry: move PUSH_AND_CLEAR_REGS out of error_entry Date: Tue, 15 Mar 2022 15:39:46 +0800 Message-Id: <20220315073949.7541-5-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220315073949.7541-1-jiangshanlai@gmail.com> References: <20220315073949.7541-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Moving PUSH_AND_CLEAR_REGS out of error_entry doesn't change any functionality. It makes error_entry() do not fiddle with the stack. It will enlarge the size: size arch/x86/entry/entry_64.o.before: text data bss dec hex filename 17916 384 0 18300 477c arch/x86/entry/entry_64.o size --format=3DSysV arch/x86/entry/entry_64.o.before: .entry.text 5528 0 .orc_unwind 6456 0 .orc_unwind_ip 4304 0 size arch/x86/entry/entry_64.o.after: text data bss dec hex filename 26868 384 0 27252 6a74 arch/x86/entry/entry_64.o size --format=3DSysV arch/x86/entry/entry_64.o.after: .entry.text 8200 0 .orc_unwind 10224 0 .orc_unwind_ip 6816 0 But .entry.text in x86_64 is 2M aligned, enlarging it to 8.2k doesn't enlarge the final text size. The tables .orc_unwind[_ip] are enlarged due to it adds many pushes. It is prepared for not calling error_entry() from XENPV in later patch and for future converting the whole error_entry into C code. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 7768cdd0c7ed..903da9119e7a 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -325,6 +325,9 @@ SYM_CODE_END(ret_from_fork) */ .macro idtentry_body cfunc has_error_code:req =20 + PUSH_AND_CLEAR_REGS + ENCODE_FRAME_POINTER + call error_entry movq %rax, %rsp /* switch stack settled by sync_regs() */ ENCODE_FRAME_POINTER @@ -1002,8 +1005,6 @@ SYM_CODE_END(paranoid_exit) SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC cld - PUSH_AND_CLEAR_REGS save_ret=3D1 - ENCODE_FRAME_POINTER 8 testb $3, CS+8(%rsp) jz .Lerror_kernelspace =20 --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 23:55:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D41C433F5 for ; Tue, 15 Mar 2022 07:40:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345546AbiCOHlL (ORCPT ); Tue, 15 Mar 2022 03:41:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345589AbiCOHlD (ORCPT ); Tue, 15 Mar 2022 03:41:03 -0400 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BD1B4B853 for ; Tue, 15 Mar 2022 00:39:47 -0700 (PDT) Received: by mail-pf1-x436.google.com with SMTP id u17so16124485pfk.11 for ; Tue, 15 Mar 2022 00:39:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DWK/XTWgXvQZrveqfBhyRmFNTbe4j+HdjdwbIGTRxxk=; b=mfDT8vFJmN9G2/6m4sTwQrGlFRG4zcNQd/FkhNb0O1AvNwki5P+PMvvXcq7j1tFa/t D+hebPoBq+BNdUSQKw5S3kW+Os5FnlW95ewCsYVaQPACovR6wSB7+Q5pS8BQqDKS1CC/ DXNWIVNczWAU3T2BhZJTjqCrWdJ3ZQQBUdzrabFezsBA5eIaOlZo0wwFG4AxExkAFq8S hZSLd2O1tS34WIzD0BVXIxF4kxC4+RxPdEJVU5N05XHVAK+t0TqHpPsgv7jBDXqmrt/b oAkUhd1Dt1r3rNl7KELlmLp+JTHpdUeCDFkPEKoRwMUp9Ic4Cn2E49XPnU3Zh9ziLq+E +LUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DWK/XTWgXvQZrveqfBhyRmFNTbe4j+HdjdwbIGTRxxk=; b=6NG6WFmWb2GjV6wSc8tWGn1ZbpcK6gZMYQ3h2KGx+lN0wsTbQXlNhwlCOFC6pe6YJ4 DFAiF79dhKkn2nxmSsgXpJswCzWrvB21uKa+UvGkjNHXMHAjNRnXnbABQJHOECdwmcHa OBzuvFHw9NGFBSxIJdHHGMCYu3N8YH28R2j7zBB6jcUpyPzTzZssxR6Dm3QPeb61mVCY HdYYu/PwlbMh46eOqzpvuRrhTFGD3YKE01A5Gv23OEvMmoe1Oiu6EdTqGyg8vOnugim4 2gEdZBrfz4mecIwMi71Zo9bO1fiid9iJuZya60ZtgS22rrFi0G+ObM5bIeue4iYy+WF7 o1zg== X-Gm-Message-State: AOAM533/8ld0osu3/9pT+0yBDChIB8acEGaULho1gds/9ardEL65pXj4 eYCweL7Gzzu8wzEJBvA2n1KiqVaacQw= X-Google-Smtp-Source: ABdhPJyQvrgJWZ2unWQKfqZsh9FFF4as/wfSFGiIyM/g/Vt+U3MG3RkDLCSD42eyczsYZXs6klc1vQ== X-Received: by 2002:a05:6a00:2311:b0:4e1:52bf:e466 with SMTP id h17-20020a056a00231100b004e152bfe466mr27189712pfh.77.1647329987055; Tue, 15 Mar 2022 00:39:47 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id ay5-20020a056a00300500b004f6d510af4asm20992174pfb.124.2022.03.15.00.39.46 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2022 00:39:46 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Borislav Petkov , Peter Zijlstra , Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V3 5/7] x86/entry: Move cld to the start of idtentry Date: Tue, 15 Mar 2022 15:39:47 +0800 Message-Id: <20220315073949.7541-6-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220315073949.7541-1-jiangshanlai@gmail.com> References: <20220315073949.7541-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Make it next to CLAC Suggested-by: Peter Zijlstra Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 903da9119e7a..e4a07276fd1c 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -360,6 +360,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=3D\has_error_code*8 ENDBR ASM_CLAC + cld =20 .if \has_error_code =3D=3D 0 pushq $-1 /* ORIG_RAX: no syscall to restart */ @@ -446,6 +447,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld =20 pushq $-1 /* ORIG_RAX: no syscall to restart */ =20 @@ -502,6 +504,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld =20 /* * If the entry is from userspace, switch stacks and treat it as @@ -564,6 +567,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=3D8 ENDBR ASM_CLAC + cld =20 /* paranoid_entry returns GS information for paranoid_exit in EBX. */ call paranoid_entry @@ -886,7 +890,6 @@ SYM_CODE_END(xen_failsafe_callback) */ SYM_CODE_START_LOCAL(paranoid_entry) UNWIND_HINT_FUNC - cld PUSH_AND_CLEAR_REGS save_ret=3D1 ENCODE_FRAME_POINTER 8 =20 @@ -1004,7 +1007,6 @@ SYM_CODE_END(paranoid_exit) */ SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC - cld testb $3, CS+8(%rsp) jz .Lerror_kernelspace =20 @@ -1138,6 +1140,7 @@ SYM_CODE_START(asm_exc_nmi) */ =20 ASM_CLAC + cld =20 /* Use %rdx as our temp variable throughout */ pushq %rdx @@ -1157,7 +1160,6 @@ SYM_CODE_START(asm_exc_nmi) */ =20 swapgs - cld FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdx movq %rsp, %rdx --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 23:55:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D422AC433F5 for ; Tue, 15 Mar 2022 07:40:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345565AbiCOHlQ (ORCPT ); Tue, 15 Mar 2022 03:41:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345595AbiCOHlE (ORCPT ); Tue, 15 Mar 2022 03:41:04 -0400 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 899874B436 for ; Tue, 15 Mar 2022 00:39:53 -0700 (PDT) Received: by mail-pf1-x432.google.com with SMTP id s8so18262617pfk.12 for ; Tue, 15 Mar 2022 00:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=F47NijhW68SKqc1BYAUjtn551pZ33aNYyOQgE9AVSDc=; b=fqKS6i+KlVpg6+nqo4tBqUZmUTLYnarlbnbmEjnoWzXn9GwJstTgZCknaGOx0PdjEf jAOWt58SOCSf/dlNN+cNPCdi3/TUlf1dG5l3j4Tq0Ja67LcW63n3+LvdwiOrbaRu5Go6 RlqrmRY1QmqEwJNxk3yORt44s5yulSzTXPS5O7MZOLkG5lh3p+k7gdcy7e976Xw9GbZ1 JfaYFiT3S1Qw3j3CJeORhWvy7UMzBlDir1ONT7MshL87BlqPSEE7qMVq+mBohBs65avh yWokbe4qjXxzhv8FDIER5vrn/3ge7Lt/bqkfFsILk3qRwXcrl550gp3dgrvVZ+VYxWwu ZaAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=F47NijhW68SKqc1BYAUjtn551pZ33aNYyOQgE9AVSDc=; b=DiC9N8Qxr3T5kT5SP8S8K++mz6FK0D1ZGTysGusBIHPfnDrpoeyMiE7U+BGP+6ZkGG 8IA0Wt4HJyA6pwlxEKnkwYgFrJb/Y6VjsMN+fFZ0gKrYeiiMN6h19ns2d/ddoODrQ7MA O3e7WGVp66RFaxqUuyXzn+RpJPHujzaq3bQyODtgVfKBM2YvKN+QxZCeOmDTVi+tIVgy FpiO85JYTZ2tRZmL2D6cvnBU2f0p/w5Q+rljRW7SEPf8EoPMPLL78GXwlPJBpQdjtV9U XzTiU0a0N+5zdHUb16UH4Wi7/Hnv6Vs6H61zoq2VabyUpaAtxZUTbDhThQUEbaYDZLpl jJoQ== X-Gm-Message-State: AOAM532xWDek2cBDbkygJhmqXpByWIrBUbVILU32WHrl0wfNIu1BOKq3 5nvrDYJ1bOb9MiP2CBLSt9Xh7roq/bE= X-Google-Smtp-Source: ABdhPJwaLUUAyeBa1DbtwyyQJmWkAUxohn03gbxPjCJO9/soJ6AfgeEnYpzW2l0yw0aBE5ap2madoQ== X-Received: by 2002:a65:6794:0:b0:36c:460e:858d with SMTP id e20-20020a656794000000b0036c460e858dmr22949141pgr.418.1647329992887; Tue, 15 Mar 2022 00:39:52 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id k14-20020a056a00134e00b004f83f05608esm4044796pfu.31.2022.03.15.00.39.51 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2022 00:39:52 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Borislav Petkov , Peter Zijlstra , Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V3 6/7] x86/entry: Don't call error_entry for XENPV Date: Tue, 15 Mar 2022 15:39:48 +0800 Message-Id: <20220315073949.7541-7-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220315073949.7541-1-jiangshanlai@gmail.com> References: <20220315073949.7541-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan When in XENPV, it is already in the task stack, and it can't fault for native_iret() nor native_load_gs_index() since XENPV uses its own pvops for iret and load_gs_index(). And it doesn't need to switch CR3. So there is no reason to call error_entry() in XENPV. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index e4a07276fd1c..ec885c2107de 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -328,8 +328,17 @@ SYM_CODE_END(ret_from_fork) PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER =20 - call error_entry - movq %rax, %rsp /* switch stack settled by sync_regs() */ + /* + * Call error_entry and switch stack settled by sync_regs(). + * + * When in XENPV, it is already in the task stack, and it can't fault + * for native_iret() nor native_load_gs_index() since XENPV uses its + * own pvops for iret and load_gs_index(). And it doesn't need to + * switch CR3. So it can skip invoking error_entry(). + */ + ALTERNATIVE "call error_entry; movq %rax, %rsp", \ + "", X86_FEATURE_XENPV + ENCODE_FRAME_POINTER UNWIND_HINT_REGS =20 --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 23:55:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13034C433EF for ; Tue, 15 Mar 2022 07:40:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345568AbiCOHlU (ORCPT ); Tue, 15 Mar 2022 03:41:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345555AbiCOHlN (ORCPT ); Tue, 15 Mar 2022 03:41:13 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 919674B436 for ; Tue, 15 Mar 2022 00:40:00 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id l8so7805123pfu.1 for ; Tue, 15 Mar 2022 00:40:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=amuoYMRnXO7FCaFWCvrHLYYAmtkhMgtyIXyJeMSribs=; b=O5UI23mfzMei9S129vyjFDLG5nlO/dxGYSigcINb8ZWYFkp8iYSf3JJP2pQEYs0hJR u4SwAUUjcqDr0sWCjyyNLO4OpVRaA7V19OO2Ao3fYrwA0qD85PiXZ7VmFEGwSLdG1Wlf 8xYuS4NNf98Q4xA2bQ5WEWTykfCxgsxLJFmuPmoYNSYaBelPoJiEyvPXLRrTuqvD3i3w AfLDQ87l4AnrCiOqMCo6/qUmN3ihK3MSAr0/gHz8JaZtdEZ4L/Bx3s7+dFarq2B+Lsgb Xs7Byn0XczyYJhnwF/ZtoWtHQ38jmRFv64GoNgLklZeifMtk/mhPp1P0mc9T8/NtLdq0 J37w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=amuoYMRnXO7FCaFWCvrHLYYAmtkhMgtyIXyJeMSribs=; b=2vvaSOthBr5XG9AyccD+AuCHa5wbDG1haRgaRmu6m0K8KeZF+3NF9WDm/hW+IUP4Qi ZLD68e1mg/SpAB68sQAG/ErpM2NinN1n571CDQy4JKCEUKY+Udwo0bymo5/MIxRgKFJ2 s/1Zt+LizxuOKglORB65tKHAIfCDdXaP5rkW2ENz93AQK+Nc3Ny+VI06OypafHMp9qqg fJloWoRUR9sa/t15HuTjuZpLrDRGxZNsz/u88c1gxqglbhWDN1W3mjtLfwUIcSVMrQlg JKSm5u9jV0TWOp8U3cDXknAFFVfWg71PK/JVRIZNr3i2ocwvP+eT/QDc+5kR60qraduO TNoA== X-Gm-Message-State: AOAM530wng/P1b1btDxuXChEU7K8ioTSeFM+cj6W9hxjyaEUdNqGJBDe zG1+ir2UGYNXYRgdYCb91zTtdNTXAvk= X-Google-Smtp-Source: ABdhPJwzhabjS0Ov7qsHNlSu6iIov9cD4PDgTrfp6hOiSS66sxdu4O4BqOgK8qLDhEXWkyeEcYbf1A== X-Received: by 2002:a63:d44c:0:b0:380:8c48:e040 with SMTP id i12-20020a63d44c000000b003808c48e040mr23572043pgj.14.1647329999895; Tue, 15 Mar 2022 00:39:59 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id u17-20020a056a00159100b004f763ac761fsm22306203pfk.33.2022.03.15.00.39.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2022 00:39:59 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Borislav Petkov , Peter Zijlstra , Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Dave Hansen , "H. Peter Anvin" , "Kirill A. Shutemov" , Josh Poimboeuf Subject: [PATCH V3 7/7] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS Date: Tue, 15 Mar 2022 15:39:49 +0800 Message-Id: <20220315073949.7541-8-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220315073949.7541-1-jiangshanlai@gmail.com> References: <20220315073949.7541-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan XENPV doesn't use swapgs_restore_regs_and_return_to_usermode(), error_entry() and entry_SYSENTER_compat(), so the PV-awared SWAPGS in them can be changed to swapgs. There is no user of the SWAPGS anymore after this change. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 2 +- arch/x86/include/asm/irqflags.h | 8 -------- 3 files changed, 4 insertions(+), 12 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index ec885c2107de..e20eabaa56b8 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1023,7 +1023,7 @@ SYM_CODE_START_LOCAL(error_entry) * We entered from user mode or we're pretending to have entered * from user mode due to an IRET fault. */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax @@ -1055,7 +1055,7 @@ SYM_CODE_START_LOCAL(error_entry) * gsbase and proceed. We'll fix up the exception and land in * .Lgs_change's error handler with kernel gsbase. */ - SWAPGS + swapgs =20 /* * Issue an LFENCE to prevent GS speculation, regardless of whether it is= a @@ -1076,7 +1076,7 @@ SYM_CODE_START_LOCAL(error_entry) * We came from an IRET to user mode, so we have user * gsbase and CR3. Switch to kernel gsbase and CR3: */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax =20 diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_com= pat.S index 2b7496a36f65..6866151bbef3 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY ENDBR /* Interrupts are off on entry. */ - SWAPGS + swapgs =20 pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax diff --git a/arch/x86/include/asm/irqflags.h b/arch/x86/include/asm/irqflag= s.h index 111104d1c2cd..7793e52d6237 100644 --- a/arch/x86/include/asm/irqflags.h +++ b/arch/x86/include/asm/irqflags.h @@ -137,14 +137,6 @@ static __always_inline void arch_local_irq_restore(uns= igned long flags) if (!arch_irqs_disabled_flags(flags)) arch_local_irq_enable(); } -#else -#ifdef CONFIG_X86_64 -#ifdef CONFIG_XEN_PV -#define SWAPGS ALTERNATIVE "swapgs", "", X86_FEATURE_XENPV -#else -#define SWAPGS swapgs -#endif -#endif #endif /* !__ASSEMBLY__ */ =20 #endif --=20 2.19.1.6.gb485710b