From nobody Thu Feb 12 15:47:29 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E05854784; Mon, 10 Jun 2024 20:48:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052528; cv=none; b=T2D53MYQL+w7aofLoYeU8ie6jmNDKntlPyQ5qH3tLf/rZXeqDGL0FlGaWlyXqPgnUnoAh5sshybp8WwbAb808YXSblZ8ijAM8HRfYTSL/0tWwZ1wwEExTKoV7XsTagK/pZw5NsFCT4qRLmS9LvAQh4HiwfXMtu3py76NfFerBwU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052528; c=relaxed/simple; bh=AxeTD70Xdnmf73ygmURvb2qqMwfuW1qpYQt8rNtFhzM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cdXhq2xdxoTmrGhvt2F34V/u4XF4TobJNAHzsGSrIaoZYHR0yhzvUOYM58Cwyqa0eenASR/QNMOMS1iRdsmGzyu7AbOiyJEdb38ebQPtwipoF4/U6Ix5Dj/FCarEptZepSqwu0qU04/kYuqsYOYYYzEtMNU77DaMWVcJi8DHtZo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=ijnhJX3k; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="ijnhJX3k" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 48743C32789; Mon, 10 Jun 2024 20:48:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1718052527; bh=AxeTD70Xdnmf73ygmURvb2qqMwfuW1qpYQt8rNtFhzM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ijnhJX3k6ykhDoLXMHH3gTvuCLj8D856jE/hjb4+4CVZsd077UMEAHeeBYoxA7MNu 6gVgX6tWfYfvQ4iuKpUERomSR+MkULWmDebnRvSAWDrDIuf0QBvJmtQShosHJJipgt gYo/n7fprCYiVs/jtM6i1XJrxG14F3QwJ56XOj8Y= From: Linus Torvalds To: Peter Anvin , Ingo Molnar , Borislav Petkov , Thomas Gleixner , Rasmus Villemoes , Josh Poimboeuf , Catalin Marinas , Will Deacon Cc: Linux Kernel Mailing List , the arch/x86 maintainers , linux-arm-kernel@lists.infradead.org, linux-arch , Linus Torvalds Subject: [PATCH 1/7] vfs: dcache: move hashlen_hash() from callers into d_hash() Date: Mon, 10 Jun 2024 13:48:15 -0700 Message-ID: <20240610204821.230388-2-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240610204821.230388-1-torvalds@linux-foundation.org> References: <20240610204821.230388-1-torvalds@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Both __d_lookup_rcu() and __d_lookup_rcu_op_compare() have the full 'name_hash' value of the qstr that they want to look up, and mask it off to just the low 32-bit hash before calling down to d_hash(). Other callers just load the 32-bit hash and pass it as the argument. If we move the masking into d_hash() itself, it simplifies the two callers that currently do the masking, and is a no-op for the other cases. It doesn't actually change the generated code since the compiler will inline d_hash() and see that the end result is the same. [ Technically, since the parse tree changes, the code generation may not be 100% the same, and for me on x86-64, this does result in gcc switching the operands around for one 'cmpl' instruction. So not necessarily the exact same code generation, but equivalent ] However, this does encapsulate the 'd_hash()' operation more, and makes the shift operation in particular be a "shift 32 bits right, return full word". Which matches the instruction semantics on both x86-64 and arm64 better, since a 32-bit shift will clear the upper bits. That makes the next step of introducing a "shift by runtime constant" more obvious and generates the shift with no extraneous type masking. Signed-off-by: Linus Torvalds --- fs/dcache.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 407095188f83..8b4382f5c99d 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -100,9 +100,9 @@ static unsigned int d_hash_shift __ro_after_init; =20 static struct hlist_bl_head *dentry_hashtable __ro_after_init; =20 -static inline struct hlist_bl_head *d_hash(unsigned int hash) +static inline struct hlist_bl_head *d_hash(unsigned long hashlen) { - return dentry_hashtable + (hash >> d_hash_shift); + return dentry_hashtable + ((u32)hashlen >> d_hash_shift); } =20 #define IN_LOOKUP_SHIFT 10 @@ -2104,7 +2104,7 @@ static noinline struct dentry *__d_lookup_rcu_op_comp= are( unsigned *seqp) { u64 hashlen =3D name->hash_len; - struct hlist_bl_head *b =3D d_hash(hashlen_hash(hashlen)); + struct hlist_bl_head *b =3D d_hash(hashlen); struct hlist_bl_node *node; struct dentry *dentry; =20 @@ -2171,7 +2171,7 @@ struct dentry *__d_lookup_rcu(const struct dentry *pa= rent, { u64 hashlen =3D name->hash_len; const unsigned char *str =3D name->name; - struct hlist_bl_head *b =3D d_hash(hashlen_hash(hashlen)); + struct hlist_bl_head *b =3D d_hash(hashlen); struct hlist_bl_node *node; struct dentry *dentry; =20 --=20 2.45.1.209.gc6f12300df From nobody Thu Feb 12 15:47:29 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF1A8132117; Mon, 10 Jun 2024 20:48:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052530; cv=none; b=nxBhLC37elo6eICj5KF9dpMQq10IwBwJigx+HHPvuZdqirDX94WdLK8uYJTzT0iZL3phR+4Ho5H/dS2VJexYRwqXsn+ROah6giK0u1qbdvk8/2kb5csxpnUwgxMZHcU0fQn0L/DqkYirtnOkuh6EOiaJI4TmtvkCt0Za42vGut0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052530; c=relaxed/simple; bh=MfMv4iuvVtYp4fD/Tz2vUcjF5O+YDb2bOLk4u2a4UXw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HSEYMow6p0P3hW5SY60u5eogdurcoXqg4GSv2ef+329AERUcM9cLjlNQGRP2X5Tf5zX4aoq7QLHy+f/fY4A2qspOT8m7ad+3EiGpxnWchMhfRv53DPx3lE6HDr0617mwr+fywslgP7t8rANmAnG2ooJotzmycr2e5rwz0EcLwvk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=oNUVm+WZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="oNUVm+WZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 12469C2BBFC; Mon, 10 Jun 2024 20:48:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1718052529; bh=MfMv4iuvVtYp4fD/Tz2vUcjF5O+YDb2bOLk4u2a4UXw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oNUVm+WZChQbUcGuxj0iePgzWN/STCk1UDe6eSft6fNqLQzRWj+cEg6N7Iub4DgQT gFmRbO2VfLFjfkDGuPpTzofcmjldor6JMmMVNRboHNJ+QDhCRD1vXvPcqriY2jlZwk nb0xo8U5Gn6m4lnOBGwcfoDV9Lp536H4cH2d1m6E= From: Linus Torvalds To: Peter Anvin , Ingo Molnar , Borislav Petkov , Thomas Gleixner , Rasmus Villemoes , Josh Poimboeuf , Catalin Marinas , Will Deacon Cc: Linux Kernel Mailing List , the arch/x86 maintainers , linux-arm-kernel@lists.infradead.org, linux-arch , Linus Torvalds Subject: [PATCH 2/7] add default dummy 'runtime constant' infrastructure Date: Mon, 10 Jun 2024 13:48:16 -0700 Message-ID: <20240610204821.230388-3-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240610204821.230388-1-torvalds@linux-foundation.org> References: <20240610204821.230388-1-torvalds@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds the initial dummy support for 'runtime constants' for when an architecture doesn't actually support an implementation of fixing up said runtime constants. This ends up being the fallback to just using the variables as regular __ro_after_init variables, and changes the dcache d_hash() function to use this model. Signed-off-by: Linus Torvalds --- fs/dcache.c | 11 ++++++++++- include/asm-generic/Kbuild | 1 + include/asm-generic/runtime-const.h | 15 +++++++++++++++ include/asm-generic/vmlinux.lds.h | 8 ++++++++ 4 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 include/asm-generic/runtime-const.h diff --git a/fs/dcache.c b/fs/dcache.c index 8b4382f5c99d..5d3a5b315692 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -35,6 +35,8 @@ #include "internal.h" #include "mount.h" =20 +#include + /* * Usage: * dcache->d_inode->i_lock protects: @@ -102,7 +104,8 @@ static struct hlist_bl_head *dentry_hashtable __ro_afte= r_init; =20 static inline struct hlist_bl_head *d_hash(unsigned long hashlen) { - return dentry_hashtable + ((u32)hashlen >> d_hash_shift); + return runtime_const_ptr(dentry_hashtable) + + runtime_const_shift_right_32(hashlen, d_hash_shift); } =20 #define IN_LOOKUP_SHIFT 10 @@ -3129,6 +3132,9 @@ static void __init dcache_init_early(void) 0, 0); d_hash_shift =3D 32 - d_hash_shift; + + runtime_const_init(shift, d_hash_shift); + runtime_const_init(ptr, dentry_hashtable); } =20 static void __init dcache_init(void) @@ -3157,6 +3163,9 @@ static void __init dcache_init(void) 0, 0); d_hash_shift =3D 32 - d_hash_shift; + + runtime_const_init(shift, d_hash_shift); + runtime_const_init(ptr, dentry_hashtable); } =20 /* SLAB cache for __getname() consumers */ diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild index b20fa25a7e8d..052e5c98c105 100644 --- a/include/asm-generic/Kbuild +++ b/include/asm-generic/Kbuild @@ -46,6 +46,7 @@ mandatory-y +=3D pci.h mandatory-y +=3D percpu.h mandatory-y +=3D pgalloc.h mandatory-y +=3D preempt.h +mandatory-y +=3D runtime-const.h mandatory-y +=3D rwonce.h mandatory-y +=3D sections.h mandatory-y +=3D serial.h diff --git a/include/asm-generic/runtime-const.h b/include/asm-generic/runt= ime-const.h new file mode 100644 index 000000000000..670499459514 --- /dev/null +++ b/include/asm-generic/runtime-const.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RUNTIME_CONST_H +#define _ASM_RUNTIME_CONST_H + +/* + * This is the fallback for when the architecture doesn't + * support the runtime const operations. + * + * We just use the actual symbols as-is. + */ +#define runtime_const_ptr(sym) (sym) +#define runtime_const_shift_right_32(val, sym) ((u32)(val)>>(sym)) +#define runtime_const_init(type,sym) do { } while (0) + +#endif diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinu= x.lds.h index 5703526d6ebf..389a78415b9b 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -944,6 +944,14 @@ #define CON_INITCALL \ BOUNDED_SECTION_POST_LABEL(.con_initcall.init, __con_initcall, _start, _e= nd) =20 +#define RUNTIME_NAME(t,x) runtime_##t##_##x + +#define RUNTIME_CONST(t,x) \ + . =3D ALIGN(8); \ + RUNTIME_NAME(t,x) : AT(ADDR(RUNTIME_NAME(t,x)) - LOAD_OFFSET) { \ + *(RUNTIME_NAME(t,x)); \ + } + /* Alignment must be consistent with (kunit_suite *) in include/kunit/test= .h */ #define KUNIT_TABLE() \ . =3D ALIGN(8); \ --=20 2.45.1.209.gc6f12300df From nobody Thu Feb 12 15:47:29 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79A2014D6E4; Mon, 10 Jun 2024 20:48:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052531; cv=none; b=h5RdNCTI32Nhqa34XakL4twgKnQOFtQZnDI4jgUXs3hntuhlBXJaZvgKQQAuGojJa0boH0odxO1xek9/KGrkfkWyYDK9/Ia3kJ9AA8bwbkjMG81xBkTco7p+Mnz04P+hdkuxFMh2FTioysKfRffVzkZ6olIvx1rXl14on/VIsJ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052531; c=relaxed/simple; bh=qEWhUwOtY4dP2oKEOdCg3gFZW6ZZXNIFIaLC2/7lZ38=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FaLzT9OIYcy0PZDWpA+ASm5FZd+CseCksekHT7fux31YZlvngmVKkFqdNdtmPrQXqS4a7eJ+pQUBRBlWmTYMtVuWGhsELXj+f91dUtW5AQaCPcpl4XgnPu1XXXba+zCEq89m/kInVrO4VzMAd05iB7pZ5MF23Nszenkt2G6eIpk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=JiH6A5U0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="JiH6A5U0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0984C2BBFC; Mon, 10 Jun 2024 20:48:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1718052531; bh=qEWhUwOtY4dP2oKEOdCg3gFZW6ZZXNIFIaLC2/7lZ38=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JiH6A5U0arD1NhlbdfpjK8+SD3NECg/OLaKv9RnBd5T4AY15CPLPBn42T6GSAQYe0 huRV6nMOULL9ce93ox3iDF9DgMHNhwPGXHnXzZuoAnF9gMVEKsa6esEKLrVWVk74Q7 1anFZ5tXKEpPNvsXPKCh2bUP4fB3HkiBWds2d1Io= From: Linus Torvalds To: Peter Anvin , Ingo Molnar , Borislav Petkov , Thomas Gleixner , Rasmus Villemoes , Josh Poimboeuf , Catalin Marinas , Will Deacon Cc: Linux Kernel Mailing List , the arch/x86 maintainers , linux-arm-kernel@lists.infradead.org, linux-arch , Linus Torvalds Subject: [PATCH 3/7] x86: add 'runtime constant' support Date: Mon, 10 Jun 2024 13:48:17 -0700 Message-ID: <20240610204821.230388-4-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240610204821.230388-1-torvalds@linux-foundation.org> References: <20240610204821.230388-1-torvalds@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This implements the runtime constant infrastructure for x86, allowing the dcache d_hash() function to be generated using as a constant for hash table address followed by shift by a constant of the hash index. Signed-off-by: Linus Torvalds --- arch/x86/include/asm/runtime-const.h | 61 ++++++++++++++++++++++++++++ arch/x86/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 64 insertions(+) create mode 100644 arch/x86/include/asm/runtime-const.h diff --git a/arch/x86/include/asm/runtime-const.h b/arch/x86/include/asm/ru= ntime-const.h new file mode 100644 index 000000000000..24e3a53ca255 --- /dev/null +++ b/arch/x86/include/asm/runtime-const.h @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RUNTIME_CONST_H +#define _ASM_RUNTIME_CONST_H + +#define runtime_const_ptr(sym) ({ \ + typeof(sym) __ret; \ + asm_inline("mov %1,%0\n1:\n" \ + ".pushsection runtime_ptr_" #sym ",\"a\"\n\t" \ + ".long 1b - %c2 - .\n\t" \ + ".popsection" \ + :"=3Dr" (__ret) \ + :"i" ((unsigned long)0x0123456789abcdefull), \ + "i" (sizeof(long))); \ + __ret; }) + +// The 'typeof' will create at _least_ a 32-bit type, but +// will happily also take a bigger type and the 'shrl' will +// clear the upper bits +#define runtime_const_shift_right_32(val, sym) ({ \ + typeof(0u+(val)) __ret =3D (val); \ + asm_inline("shrl $12,%k0\n1:\n" \ + ".pushsection runtime_shift_" #sym ",\"a\"\n\t" \ + ".long 1b - 1 - .\n\t" \ + ".popsection" \ + :"+r" (__ret)); \ + __ret; }) + +#define runtime_const_init(type, sym) do { \ + extern s32 __start_runtime_##type##_##sym[]; \ + extern s32 __stop_runtime_##type##_##sym[]; \ + runtime_const_fixup(__runtime_fixup_##type, \ + (unsigned long)(sym), \ + __start_runtime_##type##_##sym, \ + __stop_runtime_##type##_##sym); \ +} while (0) + +/* + * The text patching is trivial - you can only do this at init time, + * when the text section hasn't been marked RO, and before the text + * has ever been executed. + */ +static inline void __runtime_fixup_ptr(void *where, unsigned long val) +{ + *(unsigned long *)where =3D val; +} + +static inline void __runtime_fixup_shift(void *where, unsigned long val) +{ + *(unsigned char *)where =3D val; +} + +static inline void runtime_const_fixup(void (*fn)(void *, unsigned long), + unsigned long val, s32 *start, s32 *end) +{ + while (start < end) { + fn(*start + (void *)start, val); + start++; + } +} + +#endif diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 3509afc6a672..6e73403e874f 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -357,6 +357,9 @@ SECTIONS PERCPU_SECTION(INTERNODE_CACHE_BYTES) #endif =20 + RUNTIME_CONST(shift, d_hash_shift) + RUNTIME_CONST(ptr, dentry_hashtable) + . =3D ALIGN(PAGE_SIZE); =20 /* freed after init ends here */ --=20 2.45.1.209.gc6f12300df From nobody Thu Feb 12 15:47:29 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B6B514F9F2; Mon, 10 Jun 2024 20:48:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052533; cv=none; b=DeFpLY5L9onn6Ir9NHroMmidT9nW3oXCXzoB1aSYSAXsWlTfPIRJYN/p/PJZVm55E37fysZT+EjGgE2qQXhXAblDXhpZUVGp+tCvif43w4WL4vUpKhQfWW0ygSFUR1tRscwLA1d7EY3HH9/xoHT/dXlhRb/nVAXFmLPz0UTsHIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052533; c=relaxed/simple; bh=7f3f228SGMKS0WITa2x2BowrmDUER/1fLEQ+DyBwzuw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eoatNG239RQrNuchl8PPJMvFgDzA85vOw3cwNdMGNNwQqGGc9Noe9iWxCrruvqmgs3XJqvWjq3k0v3tFnChhjMgUghfqbRcg1unCZAKQYG+t9mWXF2UQ5PyP4uo4P1QLt0O0nYDgnLnPrbAvGYKHrgDd8wS0rihFFKtGxuaPsDA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=YPwqi78X; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="YPwqi78X" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6F8FAC4AF1C; Mon, 10 Jun 2024 20:48:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1718052532; bh=7f3f228SGMKS0WITa2x2BowrmDUER/1fLEQ+DyBwzuw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YPwqi78XF8g+oefGjk6SWIu9gETbsXAcfc1xuXxesUVODbHxG6F+Ihl/hQS8A6wHG h+h5f5IcE9LJ94AoyVcNT9TRaIM1jZTVmlB+cNNR9Ws/oUpeqTbhAiEgblNqzvHmGB V1scnDa858xxUlqxCoLdiKtjB+7xyFKKBoIeCoB0= From: Linus Torvalds To: Peter Anvin , Ingo Molnar , Borislav Petkov , Thomas Gleixner , Rasmus Villemoes , Josh Poimboeuf , Catalin Marinas , Will Deacon Cc: Linux Kernel Mailing List , the arch/x86 maintainers , linux-arm-kernel@lists.infradead.org, linux-arch , Linus Torvalds Subject: [PATCH 4/7] arm64: add 'runtime constant' support Date: Mon, 10 Jun 2024 13:48:18 -0700 Message-ID: <20240610204821.230388-5-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240610204821.230388-1-torvalds@linux-foundation.org> References: <20240610204821.230388-1-torvalds@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This implements the runtime constant infrastructure for arm64, allowing the dcache d_hash() function to be generated using as a constant for hash table address followed by shift by a constant of the hash index. Signed-off-by: Linus Torvalds --- arch/arm64/include/asm/runtime-const.h | 75 ++++++++++++++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 78 insertions(+) create mode 100644 arch/arm64/include/asm/runtime-const.h diff --git a/arch/arm64/include/asm/runtime-const.h b/arch/arm64/include/as= m/runtime-const.h new file mode 100644 index 000000000000..02462b2cb6f9 --- /dev/null +++ b/arch/arm64/include/asm/runtime-const.h @@ -0,0 +1,75 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RUNTIME_CONST_H +#define _ASM_RUNTIME_CONST_H + +#define runtime_const_ptr(sym) ({ \ + typeof(sym) __ret; \ + asm_inline("1:\t" \ + "movz %0, #0xcdef\n\t" \ + "movk %0, #0x89ab, lsl #16\n\t" \ + "movk %0, #0x4567, lsl #32\n\t" \ + "movk %0, #0x0123, lsl #48\n\t" \ + ".pushsection runtime_ptr_" #sym ",\"a\"\n\t" \ + ".long 1b - .\n\t" \ + ".popsection" \ + :"=3Dr" (__ret)); \ + __ret; }) + +#define runtime_const_shift_right_32(val, sym) ({ \ + unsigned long __ret; \ + asm_inline("1:\t" \ + "lsr %w0,%w1,#12\n\t" \ + ".pushsection runtime_shift_" #sym ",\"a\"\n\t" \ + ".long 1b - .\n\t" \ + ".popsection" \ + :"=3Dr" (__ret) \ + :"r" (0u+(val))); \ + __ret; }) + +#define runtime_const_init(type, sym) do { \ + extern s32 __start_runtime_##type##_##sym[]; \ + extern s32 __stop_runtime_##type##_##sym[]; \ + runtime_const_fixup(__runtime_fixup_##type, \ + (unsigned long)(sym), \ + __start_runtime_##type##_##sym, \ + __stop_runtime_##type##_##sym); \ +} while (0) + +// 16-bit immediate for wide move (movz and movk) in bits 5..20 +static inline void __runtime_fixup_16(unsigned int *p, unsigned int val) +{ + unsigned int insn =3D *p; + insn &=3D 0xffe0001f; + insn |=3D (val & 0xffff) << 5; + *p =3D insn; +} + +static inline void __runtime_fixup_ptr(void *where, unsigned long val) +{ + unsigned int *p =3D lm_alias(where); + __runtime_fixup_16(p, val); + __runtime_fixup_16(p+1, val >> 16); + __runtime_fixup_16(p+2, val >> 32); + __runtime_fixup_16(p+3, val >> 48); +} + +// Immediate value is 5 bits starting at bit #16 +static inline void __runtime_fixup_shift(void *where, unsigned long val) +{ + unsigned int *p =3D lm_alias(where); + unsigned int insn =3D *p; + insn &=3D 0xffc0ffff; + insn |=3D (val & 63) << 16; + *p =3D insn; +} + +static inline void runtime_const_fixup(void (*fn)(void *, unsigned long), + unsigned long val, s32 *start, s32 *end) +{ + while (start < end) { + fn(*start + (void *)start, val); + start++; + } +} + +#endif diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.ld= s.S index 755a22d4f840..55a8e310ea12 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -264,6 +264,9 @@ SECTIONS EXIT_DATA } =20 + RUNTIME_CONST(shift, d_hash_shift) + RUNTIME_CONST(ptr, dentry_hashtable) + PERCPU_SECTION(L1_CACHE_BYTES) HYPERVISOR_PERCPU_SECTION =20 --=20 2.45.1.209.gc6f12300df From nobody Thu Feb 12 15:47:29 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFA051509A7; Mon, 10 Jun 2024 20:48:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052534; cv=none; b=p23o2ypytSC9brFK+JDLnQfAQMcXyBzrJrtH3qZqMIXY6X3iRF+TYqIBbF78xK1DWN3jLrUJaiwSGmSeLxgPDxeqrK47nVGReVspGWTpYQ3L32rwE84Xl8IO7QgewIPj+BRQoK1/UfbC0uUIkQZR1LuRICBIusK/xfn1Lq0PKlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052534; c=relaxed/simple; bh=KAQJIaOfP+WEGVQXzV2TNQP3Of4y7BukbXhKYFCQjeY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DnkIPo6br/NuHJrpbPK3wkCLdVgX/e7qyJwH7CodtCJTPy+URDTpTlC0WzIUQ9aprKMkXuz4yrBInxlWY6gqm0pXFK6gx6cR9X/CrOZHRIQuz7UbBhgd6woHuM/yc5z42TvQpj+ufZCcBNLeOuxU3KcTLUWvx1WJ+lKkhrusqw4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=WcEqUO5L; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="WcEqUO5L" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2DAA0C32789; Mon, 10 Jun 2024 20:48:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1718052534; bh=KAQJIaOfP+WEGVQXzV2TNQP3Of4y7BukbXhKYFCQjeY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WcEqUO5Lpe6OHFnwpQMXyUdoGKQHdTYR2rlVNODqpnaJNG/fEglmr5ZPYmRTFPxyB S/0CvQksrc3pBlx1FM6hLWMsrRO3S64HMU37Zt+XxM3ilNXSME55wL01uTt4M4aSYC 4U9DhTQFdC+7o4TjhgQaMhKS4U0LEYWKIJsZ5BZ8= From: Linus Torvalds To: Peter Anvin , Ingo Molnar , Borislav Petkov , Thomas Gleixner , Rasmus Villemoes , Josh Poimboeuf , Catalin Marinas , Will Deacon Cc: Linux Kernel Mailing List , the arch/x86 maintainers , linux-arm-kernel@lists.infradead.org, linux-arch , Linus Torvalds Subject: [PATCH 5/7] arm64: start using 'asm goto' for get_user() when available Date: Mon, 10 Jun 2024 13:48:19 -0700 Message-ID: <20240610204821.230388-6-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240610204821.230388-1-torvalds@linux-foundation.org> References: <20240610204821.230388-1-torvalds@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This generates noticeably better code with compilers that support it, since we don't need to test the error register etc, the exception just jumps to the error handling directly. Signed-off-by: Linus Torvalds --- arch/arm64/include/asm/uaccess.h | 113 ++++++++++++++++++++++++------- arch/arm64/kernel/mte.c | 12 ++-- 2 files changed, 95 insertions(+), 30 deletions(-) diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uacc= ess.h index 14be5000c5a0..23c2edf517ed 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -184,29 +184,40 @@ static inline void __user *__uaccess_mask_ptr(const v= oid __user *ptr) * The "__xxx_error" versions set the third argument to -EFAULT if an error * occurs, and leave it unchanged on success. */ -#define __get_mem_asm(load, reg, x, addr, err, type) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_mem_asm(load, reg, x, addr, label, type) \ + asm_goto_output( \ + "1: " load " " reg "0, [%1]\n" \ + _ASM_EXTABLE_##type##ACCESS_ERR(1b, %l2, %w0) \ + : "=3Dr" (x) \ + : "r" (addr) : : label) +#else +#define __get_mem_asm(load, reg, x, addr, label, type) do { \ + int __gma_err =3D 0; \ asm volatile( \ "1: " load " " reg "1, [%2]\n" \ "2:\n" \ _ASM_EXTABLE_##type##ACCESS_ERR_ZERO(1b, 2b, %w0, %w1) \ - : "+r" (err), "=3Dr" (x) \ - : "r" (addr)) + : "+r" (__gma_err), "=3Dr" (x) \ + : "r" (addr)); \ + if (__gma_err) goto label; } while (0) +#endif =20 -#define __raw_get_mem(ldr, x, ptr, err, type) \ +#define __raw_get_mem(ldr, x, ptr, label, type) \ do { \ unsigned long __gu_val; \ switch (sizeof(*(ptr))) { \ case 1: \ - __get_mem_asm(ldr "b", "%w", __gu_val, (ptr), (err), type); \ + __get_mem_asm(ldr "b", "%w", __gu_val, (ptr), label, type); \ break; \ case 2: \ - __get_mem_asm(ldr "h", "%w", __gu_val, (ptr), (err), type); \ + __get_mem_asm(ldr "h", "%w", __gu_val, (ptr), label, type); \ break; \ case 4: \ - __get_mem_asm(ldr, "%w", __gu_val, (ptr), (err), type); \ + __get_mem_asm(ldr, "%w", __gu_val, (ptr), label, type); \ break; \ case 8: \ - __get_mem_asm(ldr, "%x", __gu_val, (ptr), (err), type); \ + __get_mem_asm(ldr, "%x", __gu_val, (ptr), label, type); \ break; \ default: \ BUILD_BUG(); \ @@ -219,27 +230,34 @@ do { \ * uaccess_ttbr0_disable(). As `x` and `ptr` could contain blocking functi= ons, * we must evaluate these outside of the critical section. */ -#define __raw_get_user(x, ptr, err) \ +#define __raw_get_user(x, ptr, label) \ do { \ __typeof__(*(ptr)) __user *__rgu_ptr =3D (ptr); \ __typeof__(x) __rgu_val; \ __chk_user_ptr(ptr); \ - \ - uaccess_ttbr0_enable(); \ - __raw_get_mem("ldtr", __rgu_val, __rgu_ptr, err, U); \ - uaccess_ttbr0_disable(); \ - \ - (x) =3D __rgu_val; \ + do { \ + __label__ __rgu_failed; \ + uaccess_ttbr0_enable(); \ + __raw_get_mem("ldtr", __rgu_val, __rgu_ptr, __rgu_failed, U); \ + uaccess_ttbr0_disable(); \ + (x) =3D __rgu_val; \ + break; \ + __rgu_failed: \ + uaccess_ttbr0_disable(); \ + goto label; \ + } while (0); \ } while (0) =20 #define __get_user_error(x, ptr, err) \ do { \ + __label__ __gu_failed; \ __typeof__(*(ptr)) __user *__p =3D (ptr); \ might_fault(); \ if (access_ok(__p, sizeof(*__p))) { \ __p =3D uaccess_mask_ptr(__p); \ - __raw_get_user((x), __p, (err)); \ + __raw_get_user((x), __p, __gu_failed); \ } else { \ + __gu_failed: \ (x) =3D (__force __typeof__(x))0; (err) =3D -EFAULT; \ } \ } while (0) @@ -262,15 +280,18 @@ do { \ do { \ __typeof__(dst) __gkn_dst =3D (dst); \ __typeof__(src) __gkn_src =3D (src); \ - int __gkn_err =3D 0; \ + do { \ + __label__ __gkn_label; \ \ - __mte_enable_tco_async(); \ - __raw_get_mem("ldr", *((type *)(__gkn_dst)), \ - (__force type *)(__gkn_src), __gkn_err, K); \ - __mte_disable_tco_async(); \ - \ - if (unlikely(__gkn_err)) \ + __mte_enable_tco_async(); \ + __raw_get_mem("ldr", *((type *)(__gkn_dst)), \ + (__force type *)(__gkn_src), __gkn_label, K); \ + __mte_disable_tco_async(); \ + break; \ + __gkn_label: \ + __mte_disable_tco_async(); \ goto err_label; \ + } while (0); \ } while (0) =20 #define __put_mem_asm(store, reg, x, addr, err, type) \ @@ -381,6 +402,52 @@ extern unsigned long __must_check __arch_copy_to_user(= void __user *to, const voi __actu_ret; \ }) =20 +static __must_check __always_inline bool user_access_begin(const void __us= er *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr,len))) + return 0; + uaccess_ttbr0_enable(); + return 1; +} +#define user_access_begin(a,b) user_access_begin(a,b) +#define user_access_end() uaccess_ttbr0_disable() + +/* + * The arm64 inline asms should learn abut asm goto, and we should + * teach user_access_begin() about address masking. + */ +#define unsafe_put_user(x, ptr, label) do { \ + int __upu_err =3D 0; \ + __raw_put_mem("sttr", x, uaccess_mask_ptr(ptr), __upu_err, U); \ + if (__upu_err) goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) \ + __raw_get_mem("ldtr", x, uaccess_mask_ptr(ptr), label, U) + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_copy_loop(dst, src, len, type, label) \ + while (len >=3D sizeof(type)) { \ + unsafe_put_user(*(type *)(src),(type __user *)(dst),label); \ + dst +=3D sizeof(type); \ + src +=3D sizeof(type); \ + len -=3D sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst,_src,_len,label) \ +do { \ + char __user *__ucu_dst =3D (_dst); \ + const char *__ucu_src =3D (_src); \ + size_t __ucu_len =3D (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, label); \ +} while (0) + #define INLINE_COPY_TO_USER #define INLINE_COPY_FROM_USER =20 diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index dcdcccd40891..6174671be7c1 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -582,12 +582,9 @@ subsys_initcall(register_mte_tcf_preferred_sysctl); size_t mte_probe_user_range(const char __user *uaddr, size_t size) { const char __user *end =3D uaddr + size; - int err =3D 0; char val; =20 - __raw_get_user(val, uaddr, err); - if (err) - return size; + __raw_get_user(val, uaddr, efault); =20 uaddr =3D PTR_ALIGN(uaddr, MTE_GRANULE_SIZE); while (uaddr < end) { @@ -595,12 +592,13 @@ size_t mte_probe_user_range(const char __user *uaddr,= size_t size) * A read is sufficient for mte, the caller should have probed * for the pte write permission if required. */ - __raw_get_user(val, uaddr, err); - if (err) - return end - uaddr; + __raw_get_user(val, uaddr, efault); uaddr +=3D MTE_GRANULE_SIZE; } (void)val; =20 return 0; + +efault: + return end - uaddr; } --=20 2.45.1.209.gc6f12300df From nobody Thu Feb 12 15:47:29 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 877151514E3; Mon, 10 Jun 2024 20:48:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052536; cv=none; b=C4L0qbYzRvMY4AxmStQnYUjdGmOc0mt/BV4RyAhFYkBwO8bBZBgECTf5xRZkZivFUA2ud6y/vD/oslZASu3DywVo6ipphX2Ed+PUE0tJweJIWb/WjZ23LahwcemQ3qWRJxPCkiGy4iQfpTJkUmNhZvprcLfUkSwf1KVEihB2UCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052536; c=relaxed/simple; bh=HEom2f9nqPDH+BblCEPQfasllgGSfBKQH1eKuClsvcM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n7/zuPPzcRtWhIT0TTc4F10vWihK9YB6ZkSSjaCiu/fiBtoJDuj12GPndkMn2dpiZcwqXoiwocmTEjpT7AJvZfQJqyhEv/qTmKX0vLjTpMuHDqIETNUAjnpTihFtvDqoIdi60uMd+8aW37Y8W54VoaXd8Le0i80Pe7UMG9XY/VA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=tsQpcNeB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="tsQpcNeB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D708EC4AF48; Mon, 10 Jun 2024 20:48:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1718052536; bh=HEom2f9nqPDH+BblCEPQfasllgGSfBKQH1eKuClsvcM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tsQpcNeB57uThZqtmJz2uP7dIDIbOiuBLKtYN1S0h9KofsPY+LiMsarc3KDWA800n vZ87ZsrW9o+P/HKUD3jAGoNbkk6gC8IvyFp9EifHDY3/X+cXSqpq8GA3ooI0lHiMLB lFocJn6d0REtPkDmMazP7SwwKsNdR8Ami0VhulN4= From: Linus Torvalds To: Peter Anvin , Ingo Molnar , Borislav Petkov , Thomas Gleixner , Rasmus Villemoes , Josh Poimboeuf , Catalin Marinas , Will Deacon Cc: Linux Kernel Mailing List , the arch/x86 maintainers , linux-arm-kernel@lists.infradead.org, linux-arch , Linus Torvalds Subject: [PATCH 6/7] arm64: start using 'asm goto' for put_user() when available Date: Mon, 10 Jun 2024 13:48:20 -0700 Message-ID: <20240610204821.230388-7-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240610204821.230388-1-torvalds@linux-foundation.org> References: <20240610204821.230388-1-torvalds@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This generates noticeably better code with compilers that support it, since we don't need to test the error register etc, the exception just jumps to the error handling directly. Signed-off-by: Linus Torvalds --- arch/arm64/include/asm/uaccess.h | 77 +++++++++++++++++++------------- 1 file changed, 46 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uacc= ess.h index 23c2edf517ed..4ab3938290ab 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -294,29 +294,41 @@ do { \ } while (0); \ } while (0) =20 -#define __put_mem_asm(store, reg, x, addr, err, type) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO +#define __put_mem_asm(store, reg, x, addr, label, type) \ + asm goto( \ + "1: " store " " reg "0, [%1]\n" \ + "2:\n" \ + _ASM_EXTABLE_##type##ACCESS_ZERO(1b, %l2) \ + : : "rZ" (x), "r" (addr) : : label) +#else +#define __put_mem_asm(store, reg, x, addr, label, type) do { \ + int __pma_err =3D 0; \ asm volatile( \ "1: " store " " reg "1, [%2]\n" \ "2:\n" \ _ASM_EXTABLE_##type##ACCESS_ERR(1b, 2b, %w0) \ - : "+r" (err) \ - : "rZ" (x), "r" (addr)) + : "+r" (__pma_err) \ + : "rZ" (x), "r" (addr)); \ + if (__pma_err) goto label; \ +} while (0) +#endif =20 -#define __raw_put_mem(str, x, ptr, err, type) \ +#define __raw_put_mem(str, x, ptr, label, type) \ do { \ __typeof__(*(ptr)) __pu_val =3D (x); \ switch (sizeof(*(ptr))) { \ case 1: \ - __put_mem_asm(str "b", "%w", __pu_val, (ptr), (err), type); \ + __put_mem_asm(str "b", "%w", __pu_val, (ptr), label, type); \ break; \ case 2: \ - __put_mem_asm(str "h", "%w", __pu_val, (ptr), (err), type); \ + __put_mem_asm(str "h", "%w", __pu_val, (ptr), label, type); \ break; \ case 4: \ - __put_mem_asm(str, "%w", __pu_val, (ptr), (err), type); \ + __put_mem_asm(str, "%w", __pu_val, (ptr), label, type); \ break; \ case 8: \ - __put_mem_asm(str, "%x", __pu_val, (ptr), (err), type); \ + __put_mem_asm(str, "%x", __pu_val, (ptr), label, type); \ break; \ default: \ BUILD_BUG(); \ @@ -328,25 +340,34 @@ do { \ * uaccess_ttbr0_disable(). As `x` and `ptr` could contain blocking functi= ons, * we must evaluate these outside of the critical section. */ -#define __raw_put_user(x, ptr, err) \ +#define __raw_put_user(x, ptr, label) \ do { \ + __label__ __rpu_failed; \ __typeof__(*(ptr)) __user *__rpu_ptr =3D (ptr); \ __typeof__(*(ptr)) __rpu_val =3D (x); \ __chk_user_ptr(__rpu_ptr); \ \ - uaccess_ttbr0_enable(); \ - __raw_put_mem("sttr", __rpu_val, __rpu_ptr, err, U); \ - uaccess_ttbr0_disable(); \ + do { \ + uaccess_ttbr0_enable(); \ + __raw_put_mem("sttr", __rpu_val, __rpu_ptr, __rpu_failed, U); \ + uaccess_ttbr0_disable(); \ + break; \ + __rpu_failed: \ + uaccess_ttbr0_disable(); \ + goto label; \ + } while (0); \ } while (0) =20 #define __put_user_error(x, ptr, err) \ do { \ + __label__ __pu_failed; \ __typeof__(*(ptr)) __user *__p =3D (ptr); \ might_fault(); \ if (access_ok(__p, sizeof(*__p))) { \ __p =3D uaccess_mask_ptr(__p); \ - __raw_put_user((x), __p, (err)); \ + __raw_put_user((x), __p, __pu_failed); \ } else { \ + __pu_failed: \ (err) =3D -EFAULT; \ } \ } while (0) @@ -369,15 +390,18 @@ do { \ do { \ __typeof__(dst) __pkn_dst =3D (dst); \ __typeof__(src) __pkn_src =3D (src); \ - int __pkn_err =3D 0; \ \ - __mte_enable_tco_async(); \ - __raw_put_mem("str", *((type *)(__pkn_src)), \ - (__force type *)(__pkn_dst), __pkn_err, K); \ - __mte_disable_tco_async(); \ - \ - if (unlikely(__pkn_err)) \ + do { \ + __label__ __pkn_err; \ + __mte_enable_tco_async(); \ + __raw_put_mem("str", *((type *)(__pkn_src)), \ + (__force type *)(__pkn_dst), __pkn_err, K); \ + __mte_disable_tco_async(); \ + break; \ + __pkn_err: \ + __mte_disable_tco_async(); \ goto err_label; \ + } while (0); \ } while(0) =20 extern unsigned long __must_check __arch_copy_from_user(void *to, const vo= id __user *from, unsigned long n); @@ -411,17 +435,8 @@ static __must_check __always_inline bool user_access_b= egin(const void __user *pt } #define user_access_begin(a,b) user_access_begin(a,b) #define user_access_end() uaccess_ttbr0_disable() - -/* - * The arm64 inline asms should learn abut asm goto, and we should - * teach user_access_begin() about address masking. - */ -#define unsafe_put_user(x, ptr, label) do { \ - int __upu_err =3D 0; \ - __raw_put_mem("sttr", x, uaccess_mask_ptr(ptr), __upu_err, U); \ - if (__upu_err) goto label; \ -} while (0) - +#define unsafe_put_user(x, ptr, label) \ + __raw_put_mem("sttr", x, uaccess_mask_ptr(ptr), label, U) #define unsafe_get_user(x, ptr, label) \ __raw_get_mem("ldtr", x, uaccess_mask_ptr(ptr), label, U) =20 --=20 2.45.1.209.gc6f12300df From nobody Thu Feb 12 15:47:29 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6DCE15217E; Mon, 10 Jun 2024 20:48:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052538; cv=none; b=U3/hrJA7DIZC9cMEE7g8m1Xa0vGs8yfAe6BcaXEg/HPKikS4gVWPwyR1hXPtHlzaxNQZclkYRj/UMKwofF47Y5smVVxXRcDU07fDMAEXLEVE+koewfV5vBd50ATjjqHp2PvtfRnBABhNnLLBeTbaDsIFMWgd8xbot1yiBCJfZGM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718052538; c=relaxed/simple; bh=ZLKP+8SlFBbrksEeWgQE5JXZVIKp4sUqZEnPm1wWp08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DmQmGWsbsFaV0WwYhbBAvHQMrK4eKeGFueAFv+n0znOjbrvD0cihqYEcZBv1WGR4TKFn4TaEiINUrlU9vqY8nR5/3JZ7Ern4v8gK6M98ai0VqlFfrOGV+P2PXWrKxX7RHuyixIzmQKnH4eietw0jSkYy6GVlAkVPk4Egsz5a1sE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=oW6YdVQZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="oW6YdVQZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8A8B8C2BBFC; Mon, 10 Jun 2024 20:48:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1718052537; bh=ZLKP+8SlFBbrksEeWgQE5JXZVIKp4sUqZEnPm1wWp08=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oW6YdVQZvb7pT3RZFzWwOH0DwmEzdPSEz6ug0yrQT5f7eFcIFPll+TIRF/JYMqYAN IPuGjDiAiCBVYcvtkSVVlzyLqnVfjT59DCr4d/zHaCDWs3pNa9EDzgXhxWifioijEi SB0wd1qaiCK27TsS/Cb+VMdqMj9GjYqEaSE5JufY= From: Linus Torvalds To: Peter Anvin , Ingo Molnar , Borislav Petkov , Thomas Gleixner , Rasmus Villemoes , Josh Poimboeuf , Catalin Marinas , Will Deacon Cc: Linux Kernel Mailing List , the arch/x86 maintainers , linux-arm-kernel@lists.infradead.org, linux-arch , Linus Torvalds Subject: [PATCH 7/7] arm64: access_ok() optimization Date: Mon, 10 Jun 2024 13:48:21 -0700 Message-ID: <20240610204821.230388-8-torvalds@linux-foundation.org> X-Mailer: git-send-email 2.45.1.209.gc6f12300df In-Reply-To: <20240610204821.230388-1-torvalds@linux-foundation.org> References: <20240610204821.230388-1-torvalds@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The TBI setup on arm64 is very strange: HW is set up to always do TBI, but the kernel enforcement for system calls is purely a software contract, and user space is supposed to mask off the top bits before the system call. Except all the actual brk/mmap/etc() system calls then mask it in kernel space anyway, and accept any TBI address. This basically unifies things and makes access_ok() also ignore it. This is an ABI change, but the current situation is very odd, and this change avoids the current mess and makes the kernel more permissive, and as such is unlikely to break anything. The way forward - for some possible future situation when people want to use more bits - is probably to introduce a new "I actually want the full 64-bit address space" prctl. But we should make sure that the software and hardware rules actually match at that point. Signed-off-by: Linus Torvalds --- arch/arm64/include/asm/uaccess.h | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uacc= ess.h index 4ab3938290ab..a435eff4ee93 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -30,23 +30,20 @@ static inline int __access_ok(const void __user *ptr, u= nsigned long size); =20 /* * Test whether a block of memory is a valid user space address. - * Returns 1 if the range is valid, 0 otherwise. * - * This is equivalent to the following test: - * (u65)addr + (u65)size <=3D (u65)TASK_SIZE_MAX + * We only care that the address cannot reach the kernel mapping, and + * that an invalid address will fault. */ -static inline int access_ok(const void __user *addr, unsigned long size) +static inline int access_ok(const void __user *p, unsigned long size) { - /* - * Asynchronous I/O running in a kernel thread does not have the - * TIF_TAGGED_ADDR flag of the process owning the mm, so always untag - * the user address before checking. - */ - if (IS_ENABLED(CONFIG_ARM64_TAGGED_ADDR_ABI) && - (current->flags & PF_KTHREAD || test_thread_flag(TIF_TAGGED_ADDR))) - addr =3D untagged_addr(addr); + unsigned long addr =3D (unsigned long)p; =20 - return likely(__access_ok(addr, size)); + /* Only bit 55 of the address matters */ + addr |=3D addr+size; + addr =3D (addr >> 55) & 1; + size >>=3D 55; + + return !(addr | size); } #define access_ok access_ok =20 --=20 2.45.1.209.gc6f12300df