From nobody Sun Feb 8 19:55:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 805A7C6FD1C for ; Wed, 22 Mar 2023 17:05:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231447AbjCVRFT (ORCPT ); Wed, 22 Mar 2023 13:05:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231373AbjCVREf (ORCPT ); Wed, 22 Mar 2023 13:04:35 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7EEF65135; Wed, 22 Mar 2023 10:03:51 -0700 (PDT) Date: Wed, 22 Mar 2023 17:03:07 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1679504587; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=edp4sA8P2SeTyb0SGU+QuEOamlFlg79W7OTGRixB7Og=; b=KXrbtHPssTbaO9apAZDqMZeeWyMNj+4Z51dlK8txaU6mx5vlRWZuGuDnjGwp0LxclQjxzB Js1+2bp5N+x78UH+KQZVF0sgKKKzvb09dKwuMSNgrzMUJwCBtyjmbiJCRV7NGwS1WQR0zI R6SK8x9lmfg1dwKxQCTg1If1zVXyKvwtQO6d7q7QEeyV1vs3tcctHjYNgR8JDXJHOQbIoL hv/5iwvqOWdK9jB2O0rIE5ZbD/XgvRo4P82d8+avchC4Q2InP8OlP91rLHeKb765TjbC6C cWUYnQMYWVwI5tSMDp/D23LngVF8ZKtlgsnFe3WdBgIp8i0HaxZ+LvhwlFHLeQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1679504587; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=edp4sA8P2SeTyb0SGU+QuEOamlFlg79W7OTGRixB7Og=; b=fjLWQZOeP1usF/QnNO1Efsy9cmDPxaChH9QfF5Low+CmNziVpa2MONCiV565UC2mVoNlw6 tZxGlekVEdQNjkDg== From: "tip-bot2 for Kirill A. Shutemov" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/mm] x86/mm: Reduce untagged_addr() overhead for systems without LAM Cc: "Kirill A. Shutemov" , Dave Hansen , x86@kernel.org, linux-kernel@vger.kernel.org MIME-Version: 1.0 Message-ID: <167950458737.5837.5944537315353690399.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The following commit has been merged into the x86/mm branch of tip: Commit-ID: e0bddc19ba9578bc4e4c60a3f29ccc227277854c Gitweb: https://git.kernel.org/tip/e0bddc19ba9578bc4e4c60a3f29ccc227= 277854c Author: Kirill A. Shutemov AuthorDate: Sun, 12 Mar 2023 14:26:02 +03:00 Committer: Dave Hansen CommitterDate: Thu, 16 Mar 2023 13:08:39 -07:00 x86/mm: Reduce untagged_addr() overhead for systems without LAM Use alternatives to reduce untagged_addr() overhead. Signed-off-by: Kirill A. Shutemov Signed-off-by: Dave Hansen Link: https://lore.kernel.org/all/20230312112612.31869-8-kirill.shutemov%40= linux.intel.com --- arch/x86/include/asm/disabled-features.h | 8 +++- arch/x86/include/asm/uaccess.h | 41 ++++++++++++++++------- 2 files changed, 37 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/as= m/disabled-features.h index 5dfa4fb..fafe9be 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -75,6 +75,12 @@ # define DISABLE_CALL_DEPTH_TRACKING (1 << (X86_FEATURE_CALL_DEPTH & 31)) #endif =20 +#ifdef CONFIG_ADDRESS_MASKING +# define DISABLE_LAM 0 +#else +# define DISABLE_LAM (1 << (X86_FEATURE_LAM & 31)) +#endif + #ifdef CONFIG_INTEL_IOMMU_SVM # define DISABLE_ENQCMD 0 #else @@ -115,7 +121,7 @@ #define DISABLED_MASK10 0 #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_LAM) #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index c79ebdb..457e814 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -30,26 +31,44 @@ static inline bool pagefault_disabled(void); * Magic with the 'sign' allows to untag userspace pointer without any bra= nches * while leaving kernel addresses intact. */ -static inline unsigned long __untagged_addr(unsigned long addr, - unsigned long mask) +static inline unsigned long __untagged_addr(unsigned long addr) { - long sign =3D addr >> 63; + long sign; + + /* + * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation + * in alternative instructions. The relocation gets wrong when gets + * copied to the target place. + */ + asm (ALTERNATIVE("", + "sar $63, %[sign]\n\t" /* user_ptr ? 0 : -1UL */ + "or %%gs:tlbstate_untag_mask, %[sign]\n\t" + "and %[sign], %[addr]\n\t", X86_FEATURE_LAM) + : [addr] "+r" (addr), [sign] "=3Dr" (sign) + : "m" (tlbstate_untag_mask), "[sign]" (addr)); =20 - addr &=3D mask | sign; return addr; } =20 #define untagged_addr(addr) ({ \ - u64 __addr =3D (__force u64)(addr); \ - __addr =3D __untagged_addr(__addr, current_untag_mask()); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr =3D (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr(__addr); \ }) =20 +static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, + unsigned long addr) +{ + long sign =3D addr >> 63; + + mmap_assert_locked(mm); + addr &=3D (mm)->context.untag_mask | sign; + + return addr; +} + #define untagged_addr_remote(mm, addr) ({ \ - u64 __addr =3D (__force u64)(addr); \ - mmap_assert_locked(mm); \ - __addr =3D __untagged_addr(__addr, (mm)->context.untag_mask); \ - (__force __typeof__(addr))__addr; \ + unsigned long __addr =3D (__force unsigned long)(addr); \ + (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) =20 #else