From nobody Mon Feb 9 13:36:17 2026 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [62.89.141.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BC3E248F73; Sat, 10 Jan 2026 04:01:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.89.141.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768017664; cv=none; b=I2JL5ItujKQl+5ZmqtoUYq83aSA2jZVlIma2RbtgQYHh45d+laTiQ2HmqalM/pFKguj2obggJ1kwEa8VuD69DfpduLJ38h4xVpmVY1rG+Dafk+M1tGVbNsDXnVIyBJIkPMGE8F0Qwj7ZSgr4k3l0lOnuYoVbXGVb/Z+FU6kmo4E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768017664; c=relaxed/simple; bh=HOsK/G56W4aJSsighlbNaxoXSE0SWAUvdOsMg4W6JqM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eTHzSEQMf6bCZ+P+0ttJc6tbxYM8TmMmvsZAsjVEe+gmjjF5g9qHcndJm5lcrlPkKBOY5ME96eyReXN8HfPGfWseQnrXfpABZ13b39Wkb3GAw/FYcRTbL2zjgS2xY7+lywg+y/v9JOXhCvQe81MpL8aqy7NKh4yqTQ2L42O8kUU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zeniv.linux.org.uk; spf=none smtp.mailfrom=ftp.linux.org.uk; dkim=pass (2048-bit key) header.d=linux.org.uk header.i=@linux.org.uk header.b=HSRxtOhp; arc=none smtp.client-ip=62.89.141.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=zeniv.linux.org.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ftp.linux.org.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linux.org.uk header.i=@linux.org.uk header.b="HSRxtOhp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=Alv7P4uLpxdxMD2oNaydKzkEIOwEPgrlWZHJemiwZ9Q=; b=HSRxtOhpMmaEouMQr/0/lYOM/I qgQUuo0j8DPxV4mM+vI3CLx7moN8FzGu/b9+Lm039eQa0I4Zcm2bKlGuejHfqZP+UghkZEL+g8B41 tKO7jjHY32xsn88QfbgW2QCe//hF3o/ckmKNn0QYjqG/DBvegOWChUgqDNMKsRXIIzpJPSNv2LU6I xQ0wFbMvXF2ABC5l4RB3QfxPGcuF0p6kxppzBCMeDe/J22kflfNl5x7j2Blv19cnr/vZjPqz3CZc0 cGdexsOCdNtgPIp+4aD5j4Kt0/wkXDfUq5wMgcGAKW3bP5DtuP11QFxUIqrZ/+H3k3IwdheUoJzSR xgCg4lYQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.99 #2 (Red Hat Linux)) id 1veQB8-000000085Z6-0p2j; Sat, 10 Jan 2026 04:02:18 +0000 From: Al Viro To: linux-mm@kvack.org Cc: Vlastimil Babka , Harry Yoo , linux-fsdevel@vger.kernel.org, Linus Torvalds , Christian Brauner , Jan Kara , Mateusz Guzik , linux-kernel@vger.kernel.org Subject: [RFC PATCH 01/15] static kmem_cache instances for core caches Date: Sat, 10 Jan 2026 04:02:03 +0000 Message-ID: <20260110040217.1927971-2-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260110040217.1927971-1-viro@zeniv.linux.org.uk> References: <20260110040217.1927971-1-viro@zeniv.linux.org.uk> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Al Viro Content-Type: text/plain; charset="utf-8" kmem_cache_create() and friends create new instances of struct kmem_cache and return pointers to those. Quite a few things in core kernel are allocated from such caches; each allocation involves dereferencing an assign-once pointer and for sufficiently hot ones that dereferencing does show in profiles. There had been patches floating around switching some of those to runtime_const infrastructure. Unfortunately, it's arch-specific and most of the architectures lack it. There's an alternative approach applicable at least to the caches that are never destroyed, which covers a lot of them. No matter what, runtime_const for pointers is not going to be faster than plain &, so if we had struct kmem_cache instances with static storage duration, we would be at least no worse off than we are with runtime_const variants. There are obstacles to doing that, but they turn out to be easy to deal with. 1) as it is, struct kmem_cache is opaque for anything outside of a few files in mm/*; that avoids serious headache with header dependencies, etc., and it's not something we want to lose. Solution: struct kmem_cache_opaque, with the size and alignment identical to struct kmem_cache. Calculation of size and alignment can be done via the same mechanism we use for asm-offsets.h and rq-offsets.h, with build-time check for mismatches. With that done, we get an opaque type defined in linux/slab-static.h that can be used for declaring those caches. In linux/slab.h we add a forward declaration of kmem_cache_opaque + helper (to_kmem_cache()) converting a pointer to kmem_cache_opaque into pointer to kmem_cache. 2) real constructor of kmem_cache needs to be taught to deal with preallocated instances. That turns out to be easy - we already pass an obscene amount of optional arguments via struct kmem_cache_args, so we can stash the pointer to preallocated instance in there. Changes in mm/slab_common.c are very minor - we should treat preallocated caches as unmergable, use the instance passed to us instead of allocating a new one and we should not free them. That's it. Note that slab-static.h is needed only in places that create such instances; all users need only slab.h (and they can be modular, unlike runtime_const-based approach). That covers the instances that never get destroyed. Quite a few fall into that category, but there's a major exception - anything in modules must be destroyed before the module gets removed. For example, filesystems that have their inodes allocated from a private kmem_cache can't make use of that technics for their inode allocations, etc. It's not that hard to deal with, but for now let's just ban including slab-static.h from modules. Signed-off-by: Al Viro --- Kbuild | 13 +++++++- include/linux/slab-static.h | 65 +++++++++++++++++++++++++++++++++++++ include/linux/slab.h | 7 ++++ mm/kmem_cache_size.c | 20 ++++++++++++ mm/slab_common.c | 30 ++++++++--------- mm/slub.c | 7 ++++ 6 files changed, 126 insertions(+), 16 deletions(-) create mode 100644 include/linux/slab-static.h create mode 100644 mm/kmem_cache_size.c diff --git a/Kbuild b/Kbuild index 13324b4bbe23..eb985a6614eb 100644 --- a/Kbuild +++ b/Kbuild @@ -45,13 +45,24 @@ kernel/sched/rq-offsets.s: $(offsets-file) $(rq-offsets-file): kernel/sched/rq-offsets.s FORCE $(call filechk,offsets,__RQ_OFFSETS_H__) =20 +# generate kmem_cache_size.h + +kmem_cache_size-file :=3D include/generated/kmem_cache_size.h + +targets +=3D mm/kmem_cache_size.s + +mm/kmem_cache_size.s: $(rq-offsets-file) + +$(kmem_cache_size-file): mm/kmem_cache_size.s FORCE + $(call filechk,offsets,__KMEM_CACHE_SIZE_H__) + # Check for missing system calls =20 quiet_cmd_syscalls =3D CALL $< cmd_syscalls =3D $(CONFIG_SHELL) $< $(CC) $(c_flags) $(missing_sysca= lls_flags) =20 PHONY +=3D missing-syscalls -missing-syscalls: scripts/checksyscalls.sh $(rq-offsets-file) +missing-syscalls: scripts/checksyscalls.sh $(kmem_cache_size-file) $(call cmd,syscalls) =20 # Check the manual modification of atomic headers diff --git a/include/linux/slab-static.h b/include/linux/slab-static.h new file mode 100644 index 000000000000..47b2220b4988 --- /dev/null +++ b/include/linux/slab-static.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_SLAB_STATIC_H +#define _LINUX_SLAB_STATIC_H + +#ifdef MODULE +#error "can't use that in modules" +#endif + +#include + +/* same size and alignment as struct kmem_cache: */ +struct kmem_cache_opaque { + unsigned char opaque[KMEM_CACHE_SIZE]; +} __aligned(KMEM_CACHE_ALIGN); + +#define __KMEM_CACHE_SETUP(cache, name, size, flags, ...) \ + __kmem_cache_create_args((name), (size), \ + &(struct kmem_cache_args) { \ + .preallocated =3D (cache), \ + __VA_ARGS__}, (flags)) + +static inline int +kmem_cache_setup_usercopy(struct kmem_cache *s, + const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)) +{ + struct kmem_cache *res; + res =3D __KMEM_CACHE_SETUP(s, name, size, flags, + .align =3D align, + .ctor =3D ctor, + .useroffset =3D useroffset, + .usersize =3D usersize); + if (IS_ERR(res)) + return PTR_ERR(res); + return 0; +} + +static inline int +kmem_cache_setup(struct kmem_cache *s, + const char *name, unsigned int size, + unsigned int align, slab_flags_t flags, + void (*ctor)(void *)) +{ + struct kmem_cache *res; + res =3D __KMEM_CACHE_SETUP(s, name, size, flags, + .align =3D align, + .ctor =3D ctor); + if (IS_ERR(res)) + return PTR_ERR(res); + return 0; +} + +#define KMEM_CACHE_SETUP(s, __struct, __flags) \ + __KMEM_CACHE_SETUP((s), #__struct, sizeof(struct __struct), (__flags), \ + .align =3D __alignof__(struct __struct)) + +#define KMEM_CACHE_SETUP_USERCOPY(s, __struct, __flags, __field) \ + __KMEM_CACHE_SETUP((s), #__struct, sizeof(struct __struct), (__flags), \ + .align =3D __alignof__(struct __struct), \ + .useroffset =3D offsetof(struct __struct, __field), \ + .usersize =3D sizeof_field(struct __struct, __field)) + +#endif diff --git a/include/linux/slab.h b/include/linux/slab.h index 2482992248dc..f16c784148b4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -261,11 +261,17 @@ enum _slab_flag_bits { =20 struct list_lru; struct mem_cgroup; +struct kmem_cache_opaque; /* * struct kmem_cache related prototypes */ bool slab_is_available(void); =20 +static inline struct kmem_cache *to_kmem_cache(struct kmem_cache_opaque *p) +{ + return (struct kmem_cache *)p; +} + /** * struct kmem_cache_args - Less common arguments for kmem_cache_create() * @@ -366,6 +372,7 @@ struct kmem_cache_args { * %0 means no sheaves will be created. */ unsigned int sheaf_capacity; + struct kmem_cache *preallocated; }; =20 struct kmem_cache *__kmem_cache_create_args(const char *name, diff --git a/mm/kmem_cache_size.c b/mm/kmem_cache_size.c new file mode 100644 index 000000000000..1ddbfa41a507 --- /dev/null +++ b/mm/kmem_cache_size.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Generate definitions needed by the preprocessor. + * This code generates raw asm output which is post-processed + * to extract and format the required data. + */ + +#define COMPILE_OFFSETS +#include +#include "slab.h" + +int main(void) +{ + /* The constants to put into include/generated/kmem_cache_size.h */ + DEFINE(KMEM_CACHE_SIZE, sizeof(struct kmem_cache)); + DEFINE(KMEM_CACHE_ALIGN, __alignof(struct kmem_cache)); + /* End of constants */ + + return 0; +} diff --git a/mm/slab_common.c b/mm/slab_common.c index eed7ea556cb1..81a413b44afb 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -224,33 +224,30 @@ static struct kmem_cache *create_cache(const char *na= me, struct kmem_cache_args *args, slab_flags_t flags) { - struct kmem_cache *s; + struct kmem_cache *s =3D args->preallocated; int err; =20 /* If a custom freelist pointer is requested make sure it's sane. */ - err =3D -EINVAL; if (args->use_freeptr_offset && (args->freeptr_offset >=3D object_size || !(flags & SLAB_TYPESAFE_BY_RCU) || !IS_ALIGNED(args->freeptr_offset, __alignof__(freeptr_t)))) - goto out; + return ERR_PTR(-EINVAL); =20 - err =3D -ENOMEM; - s =3D kmem_cache_zalloc(kmem_cache, GFP_KERNEL); - if (!s) - goto out; + if (!s) { + s =3D kmem_cache_zalloc(kmem_cache, GFP_KERNEL); + if (!s) + return ERR_PTR(-ENOMEM); + } err =3D do_kmem_cache_create(s, name, object_size, args, flags); - if (err) - goto out_free_cache; - + if (unlikely(err)) { + if (!args->preallocated) + kmem_cache_free(kmem_cache, s); + return ERR_PTR(err); + } s->refcount =3D 1; list_add(&s->list, &slab_caches); return s; - -out_free_cache: - kmem_cache_free(kmem_cache, s); -out: - return ERR_PTR(err); } =20 /** @@ -324,6 +321,9 @@ struct kmem_cache *__kmem_cache_create_args(const char = *name, object_size - args->usersize < args->useroffset)) args->usersize =3D args->useroffset =3D 0; =20 + if (args->preallocated) + flags |=3D SLAB_NO_MERGE; + if (!args->usersize && !args->sheaf_capacity) s =3D __kmem_cache_alias(name, object_size, args->align, flags, args->ctor); diff --git a/mm/slub.c b/mm/slub.c index 861592ac5425..41fe79b3f055 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -47,6 +47,7 @@ #include #include #include +#include #include =20 #include "internal.h" @@ -8491,6 +8492,12 @@ void __init kmem_cache_init(void) boot_kmem_cache_node; int node; =20 + /* verify that kmem_cache_opaque is correct */ + BUILD_BUG_ON(sizeof(struct kmem_cache) !=3D + sizeof(struct kmem_cache_opaque)); + BUILD_BUG_ON(__alignof(struct kmem_cache) !=3D + __alignof(struct kmem_cache_opaque)); + if (debug_guardpage_minorder()) slub_max_order =3D 0; =20 --=20 2.47.3