From nobody Mon Jun 22 18:10:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9915C433EF for ; Fri, 18 Mar 2022 14:29:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237272AbiCRObL (ORCPT ); Fri, 18 Mar 2022 10:31:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237265AbiCRObG (ORCPT ); Fri, 18 Mar 2022 10:31:06 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B1D555370F for ; Fri, 18 Mar 2022 07:29:46 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id n18so7086712plg.5 for ; Fri, 18 Mar 2022 07:29:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0Gsw2FtcQ99RccR2kNPSNup46xZKy3zNYSA/XBeg9M0=; b=WFWQjDi7YrzVgEMTXGrmg6GVxuQBb/zYntzqeASVfz+KB7+74aoNpFur1pcm5jVyY2 L7s0dMH2wmv/lIwX6E+qCLGbhn5Uutz7GPuWTnVRq4q0pOq+xbJkYdxGFRpZyEaiE1/T 9glm7MW/hBpt0O2hFMJjrKQEJa3amST6oWeBwrKIsr/8DnYSGCmUuFz1tARXZjmT5SM0 WeR+z9IzHFKNc3n9gBXczYts3Btod7rZ3oJYEG/WtJVwQj6YagU+TwHuIasvrpnpSCQ/ DABJnzVtJaSTYLx9h0BR118/kQ28K4DZ4ZdjYGe70lE+hxqmRoL3gkm2FMI5tsceVs66 3iCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0Gsw2FtcQ99RccR2kNPSNup46xZKy3zNYSA/XBeg9M0=; b=MTj2S0NOyVXEExQD8Vdc/m7gNFkm1wbsVm3SvOF+0V+dW3C55g4ZOki9lRSuaiXhWr xS52XQtmlB6efjWXxpqToclV94Mv0IRe5TAEACsl34wGpL3n0xp+/EX2zUl0Embm/CKX WS3/GWpTx4hAKADOeUD02sh7yPAeHDS427nokdqqikTZ+28xARuVfm8VxFpQYQwIDeLX k1FiUd16b556nuCw6GgrzIleiYQhawaYjHfXNoP2mUachQa3m78peDqgW3ZIaZak+hFc BAHFj3+P54PtN6P2HuqQfPUfhDO05hPbWEfC+GOAcg9RDJB/+fiabr8ZlQxTIDpAyi6y hAFw== X-Gm-Message-State: AOAM533tJJyNv1GOygsFZvbUnF3DUkM/ysaJF9mKy0F7TvYJ36PGwxCb urcb0mT83+oYcmxd2Ou0CuXyUrqrWgk= X-Google-Smtp-Source: ABdhPJyF/GlEAMrjIGf6kttwu/emtz8/+lAzt9WHKWTY0aCzRq/U0OoYC4CylaUYEIwSpsZHEvw0gw== X-Received: by 2002:a17:902:e5cc:b0:154:1c96:2e5b with SMTP id u12-20020a170902e5cc00b001541c962e5bmr1410821plf.94.1647613786038; Fri, 18 Mar 2022 07:29:46 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id kk12-20020a17090b4a0c00b001bed1ff3717sm8452612pjb.6.2022.03.18.07.29.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2022 07:29:45 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, Lai Jiangshan , Ingo Molnar , Dave Hansen , "H. Peter Anvin" , Fenghua Yu , Thomas Tai , "Chang S. Bae" , Masami Hiramatsu Subject: [PATCH V4 1/7] x86/traps: Move pt_regs only in fixup_bad_iret() Date: Fri, 18 Mar 2022 22:30:10 +0800 Message-Id: <20220318143016.124387-2-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220318143016.124387-1-jiangshanlai@gmail.com> References: <20220318143016.124387-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan fixup_bad_iret() and sync_regs() have similar arguments and do similar work that copies full or partial pt_regs to a place and switches stack after return. They are quite the same, but fixup_bad_iret() not only copies the pt_regs but also the return address of error_entry() while sync_regs() copies the pt_regs only and the return address of error_entry() was preserved and handled in ASM code. This patch makes fixup_bad_iret() work like sync_regs() and the handling of the return address of error_entry() is moved in ASM code. It removes the need to use the struct bad_iret_stack, simplifies fixup_bad_iret() and makes the ASM error_entry() call fixup_bad_iret() as the same as calling sync_regs() which adds readability because the calling patterns are exactly the same. It is prepared for later patch to do the stack switch after the error_entry() which simplifies the code further. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 5 ++++- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 17 ++++++----------- 3 files changed, 11 insertions(+), 13 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 4faac48ebec5..e9d896717ab4 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1058,9 +1058,12 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - mov %rsp, %rdi + popq %r12 /* save return addr in %12 */ + movq %rsp, %rdi /* arg0 =3D pt_regs pointer */ call fixup_bad_iret mov %rax, %rsp + ENCODE_FRAME_POINTER + pushq %r12 jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry) =20 diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 35317c5c551d..47ecfff2c83d 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -13,7 +13,7 @@ #ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *ere= gs); asmlinkage __visible notrace -struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s); +struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs); void __init trap_init(void); asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_r= egs *eregs); #endif diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 1563fb995005..9fe9cd9d3eeb 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -892,13 +892,8 @@ asmlinkage __visible noinstr struct pt_regs *vc_switch= _off_ist(struct pt_regs *r } #endif =20 -struct bad_iret_stack { - void *error_entry_ret; - struct pt_regs regs; -}; - asmlinkage __visible noinstr -struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s) +struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs) { /* * This is called from entry_64.S early in handling a fault @@ -908,19 +903,19 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret= _stack *s) * just below the IRET frame) and we want to pretend that the * exception came from the IRET target. */ - struct bad_iret_stack tmp, *new_stack =3D - (struct bad_iret_stack *)__this_cpu_read(cpu_tss_rw.x86_tss.sp0) - 1; + struct pt_regs tmp, *new_stack =3D + (struct pt_regs *)__this_cpu_read(cpu_tss_rw.x86_tss.sp0) - 1; =20 /* Copy the IRET target to the temporary storage. */ - __memcpy(&tmp.regs.ip, (void *)s->regs.sp, 5*8); + __memcpy(&tmp.ip, (void *)bad_regs->sp, 5*8); =20 /* Copy the remainder of the stack from the current stack. */ - __memcpy(&tmp, s, offsetof(struct bad_iret_stack, regs.ip)); + __memcpy(&tmp, bad_regs, offsetof(struct pt_regs, ip)); =20 /* Update the entry stack */ __memcpy(new_stack, &tmp, sizeof(tmp)); =20 - BUG_ON(!user_mode(&new_stack->regs)); + BUG_ON(!user_mode(new_stack)); return new_stack; } #endif --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 18:10:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD93BC433EF for ; Fri, 18 Mar 2022 14:30:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237282AbiCRObR (ORCPT ); Fri, 18 Mar 2022 10:31:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237265AbiCRObM (ORCPT ); Fri, 18 Mar 2022 10:31:12 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC5AE54190 for ; Fri, 18 Mar 2022 07:29:52 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id n15so7100570plh.2 for ; Fri, 18 Mar 2022 07:29:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=54VbEK3zT/tw2zu0eSh8jL8lMaaZrl/LxM/7ysPzMto=; b=NpCNdcyetSIK9dwAjgNBvUltYDUF0AvVnVrqyAb5hLNDvZBYAk87F9B1X1X6m9bp+h NIZug8FiE3gqmtxFXJQ6yLgV34XsFSzgHnJw5NhEwce6exm+S1aSiD7NnXaDrtzumyZ1 1e7u4J5ekDxAGGJ1TniU9iopv+2VuvoB38XdN2Dyl/CahFLEZYNFiFRjPNDCw2376A9g 3ItRuHsOU1babWjsmwQMV1lFHmZgVpR6RDMxLqeiOxyTqLXqa+jLxCTowkFVXwVni6T6 Lpnj+XcCeuGM3bmJUQsa3elGqhHcHNwMfsHC44uR9iqPSm7yu63UGEGafDWDj0s9tLcj Qzgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=54VbEK3zT/tw2zu0eSh8jL8lMaaZrl/LxM/7ysPzMto=; b=Uhlt3jHJz4CAh2J8mP06tJFVHL7BIIozqFD+SvPse9Dd9hVT5Q63VURkV6u+v94ewq 6feSenouPFDvo01Bj48+Ai466hptxLfLyiYGCF2lKkEIScfcHOTBOYgi/dcyS16K3PgV q0IyHfmMWs92ze8KyZ5YL9KPEctfT00PKEEycNWlTZQ+H7vgaAKSblx68foDaTz89rmn wR1lHw0V1hI3ym3or5dVgaRsqjlK4Feylf/ZYA8lEpW3SyXN1O+rl/7yE7XUIXQg9UCW goza1U7LKK0X4u/pUN5VxRRUpW/nZv3DTJHWQ65ppi//+kfFDoL9sQrhjNCd+DH8OCPu viiQ== X-Gm-Message-State: AOAM5316TR/vuXpKxPe/zK1gXnhG3ysroTgjjMbN/cYhQHAwdtIKZGng QeEOCyFMUzzCtmXVnD+5B1rVgCLoPyk= X-Google-Smtp-Source: ABdhPJyz4sNXr2jwVkwN9Bi1CZU8BenYDOHQHWQhPSPW0RFSs6JnJDqXIowzVeT3/+b07qK+FfwMbg== X-Received: by 2002:a17:902:dad2:b0:151:f895:9c31 with SMTP id q18-20020a170902dad200b00151f8959c31mr10510832plx.93.1647613792214; Fri, 18 Mar 2022 07:29:52 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id t8-20020a056a00138800b004f724e9fb3esm10100869pfg.89.2022.03.18.07.29.51 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2022 07:29:51 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, Lai Jiangshan , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V4 2/7] x86/entry: Switch the stack after error_entry() returns Date: Fri, 18 Mar 2022 22:30:11 +0800 Message-Id: <20220318143016.124387-3-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220318143016.124387-1-jiangshanlai@gmail.com> References: <20220318143016.124387-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan error_entry() calls sync_regs() to settle/copy the pt_regs and switches the stack directly after sync_regs(). But error_entry() itself is also a function call, the switching has to handle the return address of it together, which causes the work complicated and tangly. Switching to the stack after error_entry() makes the code simpler and intuitive. The behavior/logic is unchanged: 1) (opt) feed fixup_bad_iret() with the pt_regs pushed by ASM code 2) (opt) fixup_bad_iret() moves the partial pt_regs up 3) feed sync_regs() with the pt_regs pushed by ASM code or returned by fixup_bad_iret() 4) sync_regs() copies the whole pt_regs to kernel stack if needed 5) after error_entry() and switching %rsp, it is in kernel stack with the pt_regs Changes only in calling: Old code switches to copied pt_regs immediately twice in error_entry() while new code switches to the copied pt_regs only once after error_entry() returns. It is correct since sync_regs() doesn't need to be called close to the pt_regs it handles. Old code stashes the return-address of error_entry() in a scratch register and new code doesn't stash it. It relies on the fact that fixup_bad_iret() and sync_regs() don't corrupt the return-address of error_entry() on the stack. But even the old code also relies on the fact that fixup_bad_iret() and sync_regs() don't corrupt the return-address of themselves. They are the same reliances and are assured. After this change, error_entry() will not do fancy things with the stack except when in the prolog which will be fixed in the next patch ("move PUSH_AND_CLEAR_REGS out of error_entry"). This patch and the next patch can't be swapped because the next patch relies on this patch's stopping fiddling with the return-address of error_entry(), otherwise the objtool would complain. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index e9d896717ab4..8eff3e6b1687 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -326,6 +326,8 @@ SYM_CODE_END(ret_from_fork) .macro idtentry_body cfunc has_error_code:req =20 call error_entry + movq %rax, %rsp /* switch stack settled by sync_regs() */ + ENCODE_FRAME_POINTER UNWIND_HINT_REGS =20 movq %rsp, %rdi /* pt_regs pointer into 1st argument*/ @@ -999,14 +1001,10 @@ SYM_CODE_START_LOCAL(error_entry) /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax =20 + leaq 8(%rsp), %rdi /* arg0 =3D pt_regs pointer */ .Lerror_entry_from_usermode_after_swapgs: /* Put us onto the real thread stack. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 =3D pt_regs pointer */ call sync_regs - movq %rax, %rsp /* switch stack */ - ENCODE_FRAME_POINTER - pushq %r12 RET =20 /* @@ -1038,6 +1036,7 @@ SYM_CODE_START_LOCAL(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + leaq 8(%rsp), %rax /* return pt_regs pointer */ RET =20 .Lbstep_iret: @@ -1058,12 +1057,9 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 =3D pt_regs pointer */ + leaq 8(%rsp), %rdi /* arg0 =3D pt_regs pointer */ call fixup_bad_iret - mov %rax, %rsp - ENCODE_FRAME_POINTER - pushq %r12 + mov %rax, %rdi jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry) =20 --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 18:10:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFF6DC433F5 for ; Fri, 18 Mar 2022 14:30:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237275AbiCRObY (ORCPT ); Fri, 18 Mar 2022 10:31:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbiCRObU (ORCPT ); Fri, 18 Mar 2022 10:31:20 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEB2E5A096 for ; Fri, 18 Mar 2022 07:29:58 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id c2so5169353pga.10 for ; Fri, 18 Mar 2022 07:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/GaABf2FhAaYX7fHVjFklFYbgdZLkCz9CVYEgXadTOI=; b=UB/s7/kZkVmWLUOGRm7bH6et+9alTXL5loAN6hOCbKJVARGm3Apx6/O4vNJGCCgtt7 Q/uAPciwpPksxFzEYFjH7yQjxUKwffr0MRcGVrxAminTQXr3gee58Q3cHMNsZTxrpsh+ RGx56SXFCpuXgvFyWejBs635Q64ZXoSDSTgpNVDrTmueCUHLAREHLuQaB2njgVKMiHM2 Qj/AfL7pUjQYBifNYVAcoUF9olckXIjwDR4n71Xqip/bflnp4LWieb56AFaqo3lKz/6V scCLZjA0+UPC6hhD8qWk/vxQY/lmuZ/jB4PrhF4Zpck6ugMiBuVgRIf1uqtXWz6OyGxa 3Y7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/GaABf2FhAaYX7fHVjFklFYbgdZLkCz9CVYEgXadTOI=; b=6spxrbFDEatlmLXDzrfLYI30bqH/llOOy5VxYrEpf++aaewr8/R52sCVWhjhqjRvHy VVeu1DzpNdM+rIOqVpaJjyXmT8/fFoXtjdby9FtnphciN5kAB+9RpziMubRBC8nU4Hqz CrhURMXoUDJVrSBgXc0s0moJsee62USDJXd0OugM4LHP0jZHsShS01Q0hLcdPyHWTfiy V+JBmVJJlLW4DJyPqMRPnx93KaAiqLh1eQ8TsIe+VkMdvQ8u5eKI6lKvrLunpyXm5R4B ++ACx+xsrWCHgczTJ2lky3FtkTRHZEsKfk49xon2lPheUIvHyV3wafBsCBW9DP6tAIAI krZg== X-Gm-Message-State: AOAM531RqplNnQIZuG+zICIkOH6mmkmJFjti3XMsmBt38yNu7x2FZL3N Oif3JEu5NN7De+65EjEgwMuSBaF9CtM= X-Google-Smtp-Source: ABdhPJze1CY3rPNfpk71PsO2hS+2RSknDLzAuRnkIbiLvvkePvcFOH1dZU9kIMqLsm0x+/pxmDq6Ig== X-Received: by 2002:a63:4b1e:0:b0:365:8bc:6665 with SMTP id y30-20020a634b1e000000b0036508bc6665mr8194126pga.445.1647613798292; Fri, 18 Mar 2022 07:29:58 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id lb1-20020a17090b4a4100b001bfb76e56d1sm12923865pjb.36.2022.03.18.07.29.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2022 07:29:57 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, Lai Jiangshan , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V4 3/7] x86/entry: move PUSH_AND_CLEAR_REGS out of error_entry Date: Fri, 18 Mar 2022 22:30:12 +0800 Message-Id: <20220318143016.124387-4-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220318143016.124387-1-jiangshanlai@gmail.com> References: <20220318143016.124387-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Moving PUSH_AND_CLEAR_REGS out of error_entry doesn't change any functionality. It makes error_entry() do not fiddle with the stack. It will enlarge the size: size arch/x86/entry/entry_64.o.before: text data bss dec hex filename 17916 384 0 18300 477c arch/x86/entry/entry_64.o size --format=3DSysV arch/x86/entry/entry_64.o.before: .entry.text 5528 0 .orc_unwind 6456 0 .orc_unwind_ip 4304 0 size arch/x86/entry/entry_64.o.after: text data bss dec hex filename 26868 384 0 27252 6a74 arch/x86/entry/entry_64.o size --format=3DSysV arch/x86/entry/entry_64.o.after: .entry.text 8200 0 .orc_unwind 10224 0 .orc_unwind_ip 6816 0 But .entry.text in x86_64 is 2M aligned, enlarging it to 8.2k doesn't enlarge the final text size. The tables .orc_unwind[_ip] are enlarged due to it adds many pushes. It is prepared for not calling error_entry() from XENPV in later patch and for future converting the whole error_entry into C code. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 8eff3e6b1687..666109d56f6b 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -325,6 +325,9 @@ SYM_CODE_END(ret_from_fork) */ .macro idtentry_body cfunc has_error_code:req =20 + PUSH_AND_CLEAR_REGS + ENCODE_FRAME_POINTER + call error_entry movq %rax, %rsp /* switch stack settled by sync_regs() */ ENCODE_FRAME_POINTER @@ -987,8 +990,6 @@ SYM_CODE_END(paranoid_exit) SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC cld - PUSH_AND_CLEAR_REGS save_ret=3D1 - ENCODE_FRAME_POINTER 8 testb $3, CS+8(%rsp) jz .Lerror_kernelspace =20 --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 18:10:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CBA0C433EF for ; Fri, 18 Mar 2022 14:30:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237292AbiCROb3 (ORCPT ); Fri, 18 Mar 2022 10:31:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237265AbiCRObX (ORCPT ); Fri, 18 Mar 2022 10:31:23 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4986055762 for ; Fri, 18 Mar 2022 07:30:05 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id q11so7063031pln.11 for ; Fri, 18 Mar 2022 07:30:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WziS7bteCF6OfCEG8hp7WEVcZ4348Ta3b20GTAdH5KI=; b=ZEhWZACLHxqTLH9utNg7Z8OomwzUzraDSzrpwJkJIgzkNvWGAIhv7fkLT21sXh3BJW jdCGsUvbEHpcweTLQWRqXg5KK6NEWO3c5aCDip+yn8LvGIyy1ZV7pN45eycvqSgoo2cU 71QT9HlpYH7sYUa8GrguzxJtUuwWl4XvpgTlmr7sEMQEcseY+7jUFdtVg7+JnpWUmdjM 6/wU2v55PUVh9LNJuzCSzR6rH7eeixldZ0I+tl8sTq5STShvjUCt1wg94dpFAaj3x4zn oTzfmVtrQIrkRkQOjtBoSdAtujsdoq1tKzjmX+KsoF9F84n3WLALnP5MARtn6w11ZVRu vuCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WziS7bteCF6OfCEG8hp7WEVcZ4348Ta3b20GTAdH5KI=; b=4VGI5PXlWniOUOjVexCi0biXQqamU7OCv2/MKTZN5RMSMg7zgBvJr4UUqOt32igNI+ LeWoU7lnr3kEpah8vdfNH8x1wQYrECkNUvZmdPn728l+c1RoDTBPvMJVlPnr5s0ucOto tY5l2jBe5orwIxvvKQ8TwnAqOjCdm8n7HETN+znsrztQMgfLwaciGtGUJZbzO35Ux9EC M+d3ha8OWp7P2r4tQt7J/R9eaugtI5JeheL0za9QpY5mhOVNNFOA58i6TymqctZR5ghu sSQ9XZkkpI0bc7iYYf1tK3M7EgfrNpLYb+q8wVk2bpaCj7MQPiQL2BBy378QFvmi9OTH w5IA== X-Gm-Message-State: AOAM530CBWnaHuSu74c836WfOma9Lt1dGpXewMG17vA7be9Dvw83UReP /ldBw24issOosMtPbWSaJusUDPU+b1s= X-Google-Smtp-Source: ABdhPJw0EGjT0/O0eTquz0IWAjnnJetq7f59azUestepDvLoL7Jg/QcGTDMQhIM9DQEcAq7Ab0oCWg== X-Received: by 2002:a17:90b:4b0d:b0:1bc:4cdb:ebe3 with SMTP id lx13-20020a17090b4b0d00b001bc4cdbebe3mr11495485pjb.176.1647613804470; Fri, 18 Mar 2022 07:30:04 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id o27-20020a63731b000000b0038232af858esm1349379pgc.65.2022.03.18.07.30.03 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2022 07:30:04 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, Lai Jiangshan , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V4 4/7] x86/entry: Move cld to the start of idtentry Date: Fri, 18 Mar 2022 22:30:13 +0800 Message-Id: <20220318143016.124387-5-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220318143016.124387-1-jiangshanlai@gmail.com> References: <20220318143016.124387-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Make it next to CLAC Suggested-by: Peter Zijlstra Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 666109d56f6b..8121b9f3fceb 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -360,6 +360,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=3D\has_error_code*8 ENDBR ASM_CLAC + cld =20 .if \has_error_code =3D=3D 0 pushq $-1 /* ORIG_RAX: no syscall to restart */ @@ -428,6 +429,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld =20 pushq $-1 /* ORIG_RAX: no syscall to restart */ =20 @@ -484,6 +486,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld =20 /* * If the entry is from userspace, switch stacks and treat it as @@ -546,6 +549,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=3D8 ENDBR ASM_CLAC + cld =20 /* paranoid_entry returns GS information for paranoid_exit in EBX. */ call paranoid_entry @@ -871,7 +875,6 @@ SYM_CODE_END(xen_failsafe_callback) */ SYM_CODE_START_LOCAL(paranoid_entry) UNWIND_HINT_FUNC - cld PUSH_AND_CLEAR_REGS save_ret=3D1 ENCODE_FRAME_POINTER 8 =20 @@ -989,7 +992,6 @@ SYM_CODE_END(paranoid_exit) */ SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC - cld testb $3, CS+8(%rsp) jz .Lerror_kernelspace =20 @@ -1123,6 +1125,7 @@ SYM_CODE_START(asm_exc_nmi) */ =20 ASM_CLAC + cld =20 /* Use %rdx as our temp variable throughout */ pushq %rdx @@ -1142,7 +1145,6 @@ SYM_CODE_START(asm_exc_nmi) */ =20 swapgs - cld FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdx movq %rsp, %rdx --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 18:10:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EF9EC433F5 for ; Fri, 18 Mar 2022 14:30:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237304AbiCRObf (ORCPT ); Fri, 18 Mar 2022 10:31:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237284AbiCRObb (ORCPT ); Fri, 18 Mar 2022 10:31:31 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D74F55DE55 for ; Fri, 18 Mar 2022 07:30:11 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id bx5so7537158pjb.3 for ; Fri, 18 Mar 2022 07:30:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=p16mVRwfAs4lOC/jhpdduor9Cpu9c9v3C71RdCE1eKU=; b=k2UBtOMTP5XnazjZlWUlQAkBTrclhFjgpu6Vn/6fC4rZIygIgiqhuzR2tVVQH9IxHY 5/MeBbTVOEBy8HQw3QWQ9Hy8yjCJzPAO+Slf8ZYit9ppQutXpc0WiK7QcYqfUFzBUgaI 9ZMn8G6rPEj3ilRFdhDI7uCT/rAX7i3K1+XLcQ9capSr7Kwo1/43480eAaroz/hUVbci Tn3WS23QreUG2SZO7DGzP5WEkkxvlv9GaiWMi3pNEi91FNgyALnje0STgu/sv9v8H+an iC3Gcn6tXUvgTeTORHegDDbEZwi8F+gEsDR3GJ+aBlcLyMWRLk10JyyVhy08B09NZkF1 BQ4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=p16mVRwfAs4lOC/jhpdduor9Cpu9c9v3C71RdCE1eKU=; b=jQR/0dr7dz0gUINpXFHJoJUfWqZ7+ENULIjUBbOstuHFpf+3xP0zO3E9L3nXClLfj2 6GFWcd9QUeM7EDvyewBGQjq0FmjJeO8QlfSccEPxsoOw5gxbayegzrRhis4HzJiS6d9X c0DfkmQlXNDBMOfpWL86xRYBv/KWNAkauOMsq+oUhULo+7sV2Pb4EL7NcQMDngeIwAVq n7nCFMaZEPVaYgmE/3X8T4kj71zoleZXOwzi+bvc5TAnQRIPbnkmYReAEFPmQejsdnom T/nTNo23nAJf5NbnYR8g+Ltt4IS0WdOwSELdRx6lnPic/ZRx4N/US9XKzmbSO+sEXwTV +W8A== X-Gm-Message-State: AOAM531HyukQGeDXhVP/UYVHjpi8oZeAvp2kQCR0FGuBRekWNRqL0pf6 xYDqVvhQSB/SWKVSmSI3BCQkBTq5MQw= X-Google-Smtp-Source: ABdhPJyrKXLLP4SUQc0RG8A6ybc/o7QcU+F9ALNyMcXTy9LLeuHTseNT4hg5cloDQXs885RIuUgoxQ== X-Received: by 2002:a17:902:d2cc:b0:152:fda8:f3ff with SMTP id n12-20020a170902d2cc00b00152fda8f3ffmr10465772plc.92.1647613810403; Fri, 18 Mar 2022 07:30:10 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id z5-20020a056a00240500b004e15d39f15fsm9982929pfh.83.2022.03.18.07.30.09 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2022 07:30:10 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, Lai Jiangshan , Ingo Molnar , Dave Hansen , "H. Peter Anvin" Subject: [PATCH V4 5/7] x86/entry: Don't call error_entry for XENPV Date: Fri, 18 Mar 2022 22:30:14 +0800 Message-Id: <20220318143016.124387-6-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220318143016.124387-1-jiangshanlai@gmail.com> References: <20220318143016.124387-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan When in XENPV, it is already in the task stack, and it can't fault for native_iret() nor native_load_gs_index() since XENPV uses its own pvops for iret and load_gs_index(). And it doesn't need to switch CR3. So there is no reason to call error_entry() in XENPV. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 8121b9f3fceb..e9fe9f00d17c 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -328,8 +328,17 @@ SYM_CODE_END(ret_from_fork) PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER =20 - call error_entry - movq %rax, %rsp /* switch stack settled by sync_regs() */ + /* + * Call error_entry and switch stack settled by sync_regs(). + * + * When in XENPV, it is already in the task stack, and it can't fault + * for native_iret() nor native_load_gs_index() since XENPV uses its + * own pvops for iret and load_gs_index(). And it doesn't need to + * switch CR3. So it can skip invoking error_entry(). + */ + ALTERNATIVE "call error_entry; movq %rax, %rsp", \ + "", X86_FEATURE_XENPV + ENCODE_FRAME_POINTER UNWIND_HINT_REGS =20 --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 18:10:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7C26C433F5 for ; Fri, 18 Mar 2022 14:30:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237318AbiCRObr (ORCPT ); Fri, 18 Mar 2022 10:31:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237313AbiCRObn (ORCPT ); Fri, 18 Mar 2022 10:31:43 -0400 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 681236515A for ; Fri, 18 Mar 2022 07:30:18 -0700 (PDT) Received: by mail-pl1-x62b.google.com with SMTP id t22so7119364plo.0 for ; Fri, 18 Mar 2022 07:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YRZsn+I4L3PF7mITZVsUF7pl5hGeYOR3Ot1UI7N3MG4=; b=AzpVs+jr/74rDuMGwlZ4YEaixxx5I3+qierhYh7q4Zq5O2OJALwT4Tkj1KUx8PnfZl fV4aiGNL0gdSfnFpSnjkC3ckbjrtGnffZvpVvBb8x38uwx+56hAq0XXyfn/xw8yN99gp oqR/JC+042XVxBqSXK/0V+jfP0h30JOCvHEP0RnYGycGQTGPkhtdKP9KPGaZVmC0YWRZ LBwLJv4U0k7yO+vmnRcFuDbSZsEZmvmFuQkOct3pED/YlZsHYEayqOaeIAFHSHnTseA0 F+5YU6QR8JB867flZCtHwYGC20XJbs0YMOld1o8Co7rLy8qQZF0G/0UX+rR9BW2vy3cH Eb2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YRZsn+I4L3PF7mITZVsUF7pl5hGeYOR3Ot1UI7N3MG4=; b=t6yqUDgiN4luxQR805kKAvBpRhyiymOFmEthpyg7bOuVxmfWjzKCOjp9FWdx7jSXjI fSwfhFssytQK+CIWX9GoGWObr+AOhz1LbfGcO3BAAb1RTCdZ6+u5ICzk5cId7ix04jR/ HHwSga3Z7gtwzStzfEsab6/15Rxte4DX0QJk5+VrodqOSkFQfk4foJPzGI656Tuw6J9j AMyQaFTGiKKlEicsk1PRSw5T427i1PvXOXYr+9ld6jqTUWYMOYyCKjG+jMLeljfsRVom x5wlW9y7TTfX1e+PHQWlLgk9tO8u2m1cF4UBsIH04Tvi6fl+C3vsKw+ojQ8UfZ+9S5b+ Dqmw== X-Gm-Message-State: AOAM530MoHKYzoG9ItfTGz22uSwsV8/JAtT1+Hg3wb390976jcOqb1Q3 3twom3xHij+5RhDQl7q1wYsWjKDFhb4= X-Google-Smtp-Source: ABdhPJyzCUjk2o8N2j8v4BfuvQQHAVverf7ShHV7YJrcXxY7IMRYoxQSd/GK2leaFIgsJ0ippUf8Nw== X-Received: by 2002:a17:90a:bf91:b0:1b9:bda3:10ff with SMTP id d17-20020a17090abf9100b001b9bda310ffmr11395652pjs.38.1647613817769; Fri, 18 Mar 2022 07:30:17 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id oj2-20020a17090b4d8200b001bef79ea006sm13187606pjb.29.2022.03.18.07.30.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2022 07:30:17 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, Lai Jiangshan , Ingo Molnar , Dave Hansen , "H. Peter Anvin" , "Kirill A. Shutemov" Subject: [PATCH V4 6/7] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS Date: Fri, 18 Mar 2022 22:30:15 +0800 Message-Id: <20220318143016.124387-7-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220318143016.124387-1-jiangshanlai@gmail.com> References: <20220318143016.124387-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan XENPV doesn't use swapgs_restore_regs_and_return_to_usermode(), error_entry() and entry_SYSENTER_compat(), so the PV-awared SWAPGS in them can be changed to swapgs. There is no user of the SWAPGS anymore after this change. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 2 +- arch/x86/include/asm/irqflags.h | 8 -------- 3 files changed, 4 insertions(+), 12 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index e9fe9f00d17c..9e8d0e259a7d 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1008,7 +1008,7 @@ SYM_CODE_START_LOCAL(error_entry) * We entered from user mode or we're pretending to have entered * from user mode due to an IRET fault. */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax @@ -1040,7 +1040,7 @@ SYM_CODE_START_LOCAL(error_entry) * gsbase and proceed. We'll fix up the exception and land in * .Lgs_change's error handler with kernel gsbase. */ - SWAPGS + swapgs =20 /* * Issue an LFENCE to prevent GS speculation, regardless of whether it is= a @@ -1061,7 +1061,7 @@ SYM_CODE_START_LOCAL(error_entry) * We came from an IRET to user mode, so we have user * gsbase and CR3. Switch to kernel gsbase and CR3: */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax =20 diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_com= pat.S index 4fdb007cddbd..c5aeb0819707 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY ENDBR /* Interrupts are off on entry. */ - SWAPGS + swapgs =20 pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax diff --git a/arch/x86/include/asm/irqflags.h b/arch/x86/include/asm/irqflag= s.h index 111104d1c2cd..7793e52d6237 100644 --- a/arch/x86/include/asm/irqflags.h +++ b/arch/x86/include/asm/irqflags.h @@ -137,14 +137,6 @@ static __always_inline void arch_local_irq_restore(uns= igned long flags) if (!arch_irqs_disabled_flags(flags)) arch_local_irq_enable(); } -#else -#ifdef CONFIG_X86_64 -#ifdef CONFIG_XEN_PV -#define SWAPGS ALTERNATIVE "swapgs", "", X86_FEATURE_XENPV -#else -#define SWAPGS swapgs -#endif -#endif #endif /* !__ASSEMBLY__ */ =20 #endif --=20 2.19.1.6.gb485710b From nobody Mon Jun 22 18:10:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D180C433F5 for ; Fri, 18 Mar 2022 14:30:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237321AbiCRObt (ORCPT ); Fri, 18 Mar 2022 10:31:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237316AbiCRObq (ORCPT ); Fri, 18 Mar 2022 10:31:46 -0400 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F759606CB for ; Fri, 18 Mar 2022 07:30:27 -0700 (PDT) Received: by mail-pf1-x436.google.com with SMTP id p8so9568279pfh.8 for ; Fri, 18 Mar 2022 07:30:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y21L5VYRRuO7Md0Sv1CNvpdJBGvYW0pBnpZZyTAuB+I=; b=gfMVLDcrUOultXZfP4iYKKNXaGn4fUrTsNS0wemUZM6OlNhqFkIBw9o2PzYaAw2Y+/ UJr9hZQ/2R5x4OFNui1SMEaFJ7yYERnPzD3ZBaD+7RK0WSSICwvqpwfyQu8deT2h5Ofq vqqvFj1jXxvV7zpC3CriMsN3K8pXJBlMVZJ74u7U4WIG3P49Gb4/LC8nwn8OxUYgXasM DWmh26Tvoi8lfhyfGpb59hcR68MsiA4SC4EMGXcXDUemBeWuewd6wSzb+HQsIqAfRZlD CoH3NQSdS1/vQ3t9xYJMFCEyfOTbE9ZjKk8nsXiqRyTvTeMtzqO/rP6P8a/sVU1oJ1vX xHfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y21L5VYRRuO7Md0Sv1CNvpdJBGvYW0pBnpZZyTAuB+I=; b=mo6LCffUptITnZnWVMQdSSl16IKOiZAuWhukJjRKa4m0cqhjx7fD4A0Qd+9gNIc1Yu +PSwhGIyPDooT9HBKY5NV+bZbQ6iZ1cT3K8W/y23v9IjKQR58T+W82Kw1Uyvy1ZH4xVc AFRpObe5cz7cuoh45MppqAXBS/F/1BUQkTpCYGEZV0TgyAE8RZKgh51WAzfnXgoHE3KE V0KVXH3/jKsSi1AxkGcsH3XjwzJQU0ssYfOa4PHSsrCak+GhiKmk3d7ilsqxJih12knW KuaOLu9zTodojlw3Ar3J9NcvTVPGXki1eyCyK08WYpN6+qafVMfhC2hP0FFMlhxiYNBg w4FQ== X-Gm-Message-State: AOAM533DPsdMrgNAxZMqNv/0lDLEXPb7rWK/OU0I9CEWVOz5QJJZDokU zRpkFOSmYmHCiMr8+BZHA/+yVyWulhg= X-Google-Smtp-Source: ABdhPJwKqVkrvhpqqBFV/EKEKO+kyStwmxQdU1LZm3biPo3uhf6hWIx8112T/IDcFFNiA10I52Djcg== X-Received: by 2002:a05:6a00:2168:b0:4f6:eb64:71dd with SMTP id r8-20020a056a00216800b004f6eb6471ddmr10077292pff.40.1647613826332; Fri, 18 Mar 2022 07:30:26 -0700 (PDT) Received: from localhost ([47.251.4.198]) by smtp.gmail.com with ESMTPSA id d15-20020a056a00198f00b004f7109da1c4sm9714669pfl.205.2022.03.18.07.30.25 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2022 07:30:26 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Borislav Petkov , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, Lai Jiangshan , Ingo Molnar , Dave Hansen , "H. Peter Anvin" , Joerg Roedel , "Chang S. Bae" , Jan Kiszka Subject: [PATCH V4 7/7] x86/entry: Use idtentry macro for entry_INT80_compat Date: Fri, 18 Mar 2022 22:30:16 +0800 Message-Id: <20220318143016.124387-8-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20220318143016.124387-1-jiangshanlai@gmail.com> References: <20220318143016.124387-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan entry_INT80_compat is identical to idtentry macro except a special handling for %rax in the prolog. Add the prolog to idtentry and use idtentry for entry_INT80_compat. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry_64.S | 18 ++++++ arch/x86/entry/entry_64_compat.S | 103 ------------------------------- arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/proto.h | 4 -- 4 files changed, 65 insertions(+), 107 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 9e8d0e259a7d..6ac070378d8b 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -375,6 +375,24 @@ SYM_CODE_START(\asmsym) pushq $-1 /* ORIG_RAX: no syscall to restart */ .endif =20 + .if \vector =3D=3D IA32_SYSCALL_VECTOR + /* + * User tracing code (ptrace or signal handlers) might assume + * that the saved RAX contains a 32-bit number when we're + * invoking a 32-bit syscall. Just in case the high bits are + * nonzero, zero-extend the syscall number. (This could almost + * certainly be deleted with no ill effects.) + */ + movl %eax, %eax + + /* + * do_int80_syscall_32() expects regs->orig_ax to be user ax, + * and regs->ax to be $-ENOSYS. + */ + movq %rax, (%rsp) + movq $-ENOSYS, %rax + .endif + .if \vector =3D=3D X86_TRAP_BP /* * If coming from kernel space, create a 6-word gap to allow the diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_com= pat.S index c5aeb0819707..6866151bbef3 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -315,106 +315,3 @@ sysret32_from_system_call: swapgs sysretl SYM_CODE_END(entry_SYSCALL_compat) - -/* - * 32-bit legacy system call entry. - * - * 32-bit x86 Linux system calls traditionally used the INT $0x80 - * instruction. INT $0x80 lands here. - * - * This entry point can be used by 32-bit and 64-bit programs to perform - * 32-bit system calls. Instances of INT $0x80 can be found inline in - * various programs and libraries. It is also used by the vDSO's - * __kernel_vsyscall fallback for hardware that doesn't support a faster - * entry method. Restarted 32-bit system calls also fall back to INT - * $0x80 regardless of what instruction was originally used to do the - * system call. - * - * This is considered a slow path. It is not used by most libc - * implementations on modern hardware except during process startup. - * - * Arguments: - * eax system call number - * ebx arg1 - * ecx arg2 - * edx arg3 - * esi arg4 - * edi arg5 - * ebp arg6 - */ -SYM_CODE_START(entry_INT80_compat) - UNWIND_HINT_EMPTY - ENDBR - /* - * Interrupts are off on entry. - */ - ASM_CLAC /* Do this early to minimize exposure */ - SWAPGS - - /* - * User tracing code (ptrace or signal handlers) might assume that - * the saved RAX contains a 32-bit number when we're invoking a 32-bit - * syscall. Just in case the high bits are nonzero, zero-extend - * the syscall number. (This could almost certainly be deleted - * with no ill effects.) - */ - movl %eax, %eax - - /* switch to thread stack expects orig_ax and rdi to be pushed */ - pushq %rax /* pt_regs->orig_ax */ - pushq %rdi /* pt_regs->di */ - - /* Need to switch before accessing the thread stack. */ - SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdi - - /* In the Xen PV case we already run on the thread stack. */ - ALTERNATIVE "", "jmp .Lint80_keep_stack", X86_FEATURE_XENPV - - movq %rsp, %rdi - movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - - pushq 6*8(%rdi) /* regs->ss */ - pushq 5*8(%rdi) /* regs->rsp */ - pushq 4*8(%rdi) /* regs->eflags */ - pushq 3*8(%rdi) /* regs->cs */ - pushq 2*8(%rdi) /* regs->ip */ - pushq 1*8(%rdi) /* regs->orig_ax */ - pushq (%rdi) /* pt_regs->di */ -.Lint80_keep_stack: - - pushq %rsi /* pt_regs->si */ - xorl %esi, %esi /* nospec si */ - pushq %rdx /* pt_regs->dx */ - xorl %edx, %edx /* nospec dx */ - pushq %rcx /* pt_regs->cx */ - xorl %ecx, %ecx /* nospec cx */ - pushq $-ENOSYS /* pt_regs->ax */ - pushq %r8 /* pt_regs->r8 */ - xorl %r8d, %r8d /* nospec r8 */ - pushq %r9 /* pt_regs->r9 */ - xorl %r9d, %r9d /* nospec r9 */ - pushq %r10 /* pt_regs->r10*/ - xorl %r10d, %r10d /* nospec r10 */ - pushq %r11 /* pt_regs->r11 */ - xorl %r11d, %r11d /* nospec r11 */ - pushq %rbx /* pt_regs->rbx */ - xorl %ebx, %ebx /* nospec rbx */ - pushq %rbp /* pt_regs->rbp */ - xorl %ebp, %ebp /* nospec rbp */ - pushq %r12 /* pt_regs->r12 */ - xorl %r12d, %r12d /* nospec r12 */ - pushq %r13 /* pt_regs->r13 */ - xorl %r13d, %r13d /* nospec r13 */ - pushq %r14 /* pt_regs->r14 */ - xorl %r14d, %r14d /* nospec r14 */ - pushq %r15 /* pt_regs->r15 */ - xorl %r15d, %r15d /* nospec r15 */ - - UNWIND_HINT_REGS - - cld - - movq %rsp, %rdi - call do_int80_syscall_32 - jmp swapgs_restore_regs_and_return_to_usermode -SYM_CODE_END(entry_INT80_compat) diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentr= y.h index 7924f27f5c8b..fac5db38c895 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -206,6 +206,20 @@ __visible noinstr void func(struct pt_regs *regs, \ \ static noinline void __##func(struct pt_regs *regs, u32 vector) =20 +/** + * DECLARE_IDTENTRY_IA32_EMULATION - Declare functions for int80 + * @vector: Vector number (ignored for C) + * @asm_func: Function name of the entry point + * @cfunc: The C handler called from the ASM entry point (ignored for C) + * + * Declares two functions: + * - The ASM entry point: asm_func + * - The XEN PV trap entry point: xen_##asm_func (maybe unused) + */ +#define DECLARE_IDTENTRY_IA32_EMULATION(vector, asm_func, cfunc) \ + asmlinkage void asm_func(void); \ + asmlinkage void xen_##asm_func(void) + /** * DECLARE_IDTENTRY_SYSVEC - Declare functions for system vector entry poi= nts * @vector: Vector number (ignored for C) @@ -432,6 +446,35 @@ __visible noinstr void func(struct pt_regs *regs, \ #define DECLARE_IDTENTRY_ERRORCODE(vector, func) \ idtentry vector asm_##func func has_error_code=3D1 =20 +/* + * 32-bit legacy system call entry. + * + * 32-bit x86 Linux system calls traditionally used the INT $0x80 + * instruction. INT $0x80 lands here. + * + * This entry point can be used by 32-bit and 64-bit programs to perform + * 32-bit system calls. Instances of INT $0x80 can be found inline in + * various programs and libraries. It is also used by the vDSO's + * __kernel_vsyscall fallback for hardware that doesn't support a faster + * entry method. Restarted 32-bit system calls also fall back to INT + * $0x80 regardless of what instruction was originally used to do the + * system call. + * + * This is considered a slow path. It is not used by most libc + * implementations on modern hardware except during process startup. + * + * Arguments: + * eax system call number + * ebx arg1 + * ecx arg2 + * edx arg3 + * esi arg4 + * edi arg5 + * ebp arg6 + */ +#define DECLARE_IDTENTRY_IA32_EMULATION(vector, asm_func, cfunc) \ + idtentry vector asm_func cfunc has_error_code=3D0 + /* Special case for 32bit IRET 'trap'. Do not emit ASM code */ #define DECLARE_IDTENTRY_SW(vector, func) =20 @@ -638,6 +681,10 @@ DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER, common_interrupt); DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER, spurious_interrupt); #endif =20 +#ifdef CONFIG_IA32_EMULATION +DECLARE_IDTENTRY_IA32_EMULATION(IA32_SYSCALL_VECTOR, entry_INT80_compat, d= o_int80_syscall_32); +#endif + /* System vector entry points */ #ifdef CONFIG_X86_LOCAL_APIC DECLARE_IDTENTRY_SYSVEC(ERROR_APIC_VECTOR, sysvec_error_interrupt); diff --git a/arch/x86/include/asm/proto.h b/arch/x86/include/asm/proto.h index feed36d44d04..c4d331fe65ff 100644 --- a/arch/x86/include/asm/proto.h +++ b/arch/x86/include/asm/proto.h @@ -28,10 +28,6 @@ void entry_SYSENTER_compat(void); void __end_entry_SYSENTER_compat(void); void entry_SYSCALL_compat(void); void entry_SYSCALL_compat_safe_stack(void); -void entry_INT80_compat(void); -#ifdef CONFIG_XEN_PV -void xen_entry_INT80_compat(void); -#endif #endif =20 void x86_configure_nx(void); --=20 2.19.1.6.gb485710b