From nobody Thu Sep 24 20:30:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCAF844C65C; Mon, 21 Sep 2026 07:58:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977504; cv=none; b=I9C+GQPOWrVdXXZs2C1mLKxozYGPPJtneHJ8W4pcOt+SYlxzkvB0UPYZfi60n6KEgqkvRNHJbghup+uzTIFzjhEN0uSdsAXGaao+MugATFeWdvc8mA4yLu6cVcTJdTgsNlJWN2LJ57dHFCEzBB8ovFZ2Wx4cY8/V27DXuupNvVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977504; c=relaxed/simple; bh=yoNf68AxG5w4U/BVM+cYoBMu5dGzeo9HzcEeFRFwN4A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Uffrb0DQbpmDiU6h1ofz1dOIGtBE8qqvhvi9Zv9ToksSAQR00mB0CazrPYL2scFOM2EuUuqscJSnV5HlMKBhogPKsA62dvAS7chpzJ3Yj7sh/mzGS0TAx8HJVt2Kh/bxO1kZCwArtyllUX9RxMO49e8em8EOckccEzY5/J7zmn8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ouHcHhEn; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ouHcHhEn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 76CCD1F00899; Mon, 21 Sep 2026 07:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789977500; bh=pXEMn8PYoauzMFPqYoyRVQk4fKHRKQ7Pm/+2eUzLBwM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ouHcHhEn3pF1QVHo3Sbazx3hPddkjUOD89SP/L9S0cHcaBKN2+A2Rc4sof5g++TcQ xtgz9khnxu5eYU8LtTPSz1IosPFoOjs+erMVMkRaMkgks9irTWcn9stk6CP7gumuCk DJMrd6hgPkrecnH+TQc4md2wYTvbjgzAjf/msUVgslIx30KHvAaMgb0nL1T3yxYO+9 rhYDqZVAMMXUB6bvKscdjGIkzXR4ua6fInw4hqN+PihF+6ynX/64IwcuwerJAFXS8W 3B8xV+zqDVo9H+bPTMTAp1EoUJe21NyRn8mrY5coc0C6iWF7Bnn/8P941Pr9nTLZpF isN880f4ZFbIQ== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Harry Yoo , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, Pedro Falcato , Kuniyuki Iwashima , linux-hardening@vger.kernel.org, Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Simon Horman , Jason Xing , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Jiayuan Chen , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 1/7] mm/slab: Mark the kmem_buckets_create() context as a Context: section Date: Mon, 21 Sep 2026 00:58:12 -0700 Message-Id: <20260921075820.1718334-1-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260921075811.too.775-kees@kernel.org> References: <20260921075811.too.775-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2066; i=kees@kernel.org; h=from:subject; bh=yoNf68AxG5w4U/BVM+cYoBMu5dGzeo9HzcEeFRFwN4A=; b=owGbwMvMwCVmps19z/KJym7G02pJDFkbHs+w2O1WFJVu0y0uMin1v8cLsWNM4fPnBuyPdni3Q 2Jp/+v1HaUsDGJcDLJiiixBdu5xLh5v28Pd5yrCzGFlAhnCwMUpABNJ0GX4H3HjgaPAG2lOySjz D/KnFzll2SyvEQpcP3l65wyFVUsSJBgZ7mxoXb/3/x2WcLHVPIdkHWcafppsOI3vgHNYcW38MU5 /FgA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kmem_buckets_create() had the same sentence about calling context as kmem_cache_create(), but lacked the "Context:" prefix, so kernel-doc rendered in the body instead of as a "context" section. Give it the missing prefix. Additionally fix the "a interrupt" typo __kmem_cache_create_args() had. Fixes: b32801d1255be ("mm/slab: Introduce kmem_buckets_create() and family") Assisted-by: LLM Signed-off-by: Kees Cook Acked-by: Pedro Falcato =20 Reviewed-by: Harry Yoo (Meta) --- Cc: Vlastimil Babka Cc: Harry Yoo Cc: Andrew Morton Cc: Hao Li Cc: Christoph Lameter Cc: David Rientjes Cc: Roman Gushchin Cc: Cc: Pedro Falcato Cc: Kuniyuki Iwashima Cc: --- mm/slab_common.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index b19ba1b31484..270408ce5a9d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -311,7 +311,7 @@ __kmem_cache_alias(const char *name, unsigned int size,= slab_flags_t flags, * &SLAB_TYPESAFE_BY_RCU - Slab page (not individual objects) freeing dela= yed * by a grace period - see the full description before using. * - * Context: Cannot be called within a interrupt, but can be interrupted. + * Context: Cannot be called within an interrupt, but can be interrupted. * * Return: a pointer to the cache on success, NULL on failure. */ @@ -422,7 +422,7 @@ static struct kmem_cache *kmem_buckets_cache __ro_after= _init; * to/from userspace. * @ctor: A constructor for the objects, run when new allocations are made. * - * Cannot be called within an interrupt, but can be interrupted. + * Context: Cannot be called within an interrupt, but can be interrupted. * * Return: a pointer to the cache on success, NULL on failure. When * CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and --=20 2.34.1 From nobody Thu Sep 24 20:30:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB27644C65D; Mon, 21 Sep 2026 07:58:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977504; cv=none; b=tkInJyyjcTEtbLPbDNH7nYXDnMv+SobiMb/yrOLeFNU9Cn2+Fd8m3Gbn6t4dEtejOKlqykVUSqUneQoS1N4ssSJzcbDppfG8aryjAHwxslchqWsz9z0sPZJpkSCEU9Umg17lWmXIlRvkKaA19IcKVWyRjREbd0/f8sgdFFA0w2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977504; c=relaxed/simple; bh=U4cVv3SUTzhH/lrc/2fcsnidFcJazCsEDw7sFYGmqdg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VzIKIPjLBLWTiD98ZTMAvrrpENDWzZMzT5oXUNCYpT5EUtSUhheRfxGfrJFc7gK6UHpb+LQT7xsK/YRnF1l5Qg9zDY4PoRnZ6NRGnZC0YmnOsMWcerB/SmOdn0W5pD2QwozhJfRP0ZJGow/FvG3jUqHPXhfJI2H1ePX9PQEWX2g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FdL0Rct0; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FdL0Rct0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71C4D1F00893; Mon, 21 Sep 2026 07:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789977500; bh=LrV5HlUofGQHQ/kNdAFIhT1ZS3PTpyWCIefSybl8wfs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=FdL0Rct0o8D1RMzCLiAIR3g8WKsP5grYoTTdBh0dDgSKhqumLJ/MVuJYKSBgsSsbW gCgPDkx1tfK8qP+AWxKq9B+0hy69AL55t1FnlywiLjI10GviMm1hUZ1CPRoDpylmwY Nd7BHaHp6WU0lv3/93R+xhNIDwJo07svjVpMcL0e4qRmDpkaRBbXLGEHiHwiX83wT4 qs87VMBFZjX9Th97lNW1bMRg6ahQV4kXTYfxHMEU61iupQpMvjzw8izbcklS6qa0t9 UJ2M0R+to0SeDtu36W5k4VJYtSbwZmlZjaZEa/w9GyGjueJPlfUcN+Nb7z4xYPZXNr tGXfalSsm70wg== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Harry Yoo , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, Pedro Falcato , Kuniyuki Iwashima , linux-hardening@vger.kernel.org, Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Simon Horman , Jason Xing , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Jiayuan Chen , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 2/7] mm/slab: Give bucket caches the alignment of the caches they mirror Date: Mon, 21 Sep 2026 00:58:13 -0700 Message-Id: <20260921075820.1718334-2-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260921075811.too.775-kees@kernel.org> References: <20260921075811.too.775-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2387; i=kees@kernel.org; h=from:subject; bh=U4cVv3SUTzhH/lrc/2fcsnidFcJazCsEDw7sFYGmqdg=; b=owGbwMvMwCVmps19z/KJym7G02pJDFkbHs/4sDlPq9X8gb7/bpsnB7XWCMtlbF5fMs+qiOlbz eQtvnwsHaUsDGJcDLJiiixBdu5xLh5v28Pd5yrCzGFlAhnCwMUpABNR+Mbw33d+keP64y2Zywy3 Z9Va3Y0Ud5otrZGsueeRMpODbe3tYkaG5r37+zvfr/z2PVUhSt9uNvN1w5vbrnNfUql6WeVyIP0 oKwA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A bucket set is created with kmem_cache_create_usercopy(..., align =3D 0), so calculate_alignment() falls back to arch_slab_minalign(), typically 8 bytes. The general kmalloc caches it stands in for are created through create_boot_cache(), which starts from ARCH_KMALLOC_MINALIGN and raises it to the largest power-of-two divisor of the size: if (flags & SLAB_KMALLOC) align =3D max(align, 1U << (ffs(size) - 1)); This is only a problem when slab metadata is enabled with CONFIG_KASAN=3Dy, CONFIG_SLUB_DEBUG_ON=3Dy, or "slab_debug=3D...", because metadata changes the stride size off a power of two, for example: size 128: bucket align=3D8 size=3D224 | kmalloc align=3D128 size=3D384 size 512: bucket align=3D8 size=3D608 | kmalloc align=3D512 size=3D1536 size 2048: bucket align=3D8 size=3D2144 | kmalloc align=3D2048 size=3D6144 So bucket allocations will fail the IS_ALIGNED(p, ARCH_DMA_MINALIGN) check, potentially creating problems for non-coherent DMA situation. Take the alignment from the cache being mirrored, which is where the size and the name suffix already come from. Nothing changes where the alignment was already implied by the size. Fixes: b32801d1255be ("mm/slab: Introduce kmem_buckets_create() and family") Assisted-by: LLM Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Harry Yoo Cc: Andrew Morton Cc: Hao Li Cc: Christoph Lameter Cc: David Rientjes Cc: Roman Gushchin Cc: Cc: Pedro Falcato Cc: Kuniyuki Iwashima Cc: --- mm/slab_common.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 270408ce5a9d..cc58f192f349 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -487,7 +487,8 @@ kmem_buckets *kmem_buckets_create(const char *name, sla= b_flags_t flags, if (WARN_ON(!cache_name)) goto fail; (*b)[aligned_idx] =3D kmem_cache_create_usercopy(cache_name, size, - 0, flags, cache_useroffset, + kmalloc_caches[KMALLOC_NORMAL][idx]->align, + flags, cache_useroffset, cache_usersize, ctor); kfree(cache_name); if (WARN_ON(!(*b)[aligned_idx])) --=20 2.34.1 From nobody Thu Sep 24 20:30:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC8CF3B42CC; Mon, 21 Sep 2026 07:58:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977505; cv=none; b=eO5pXowWc3xAD7tRYXYBvv1MxJGebWjhe6qxnoTbngWIwYXr3t98xrOY9xjCzTYyWs614W1I3tDwaviTuufISZW4VbERmjRRuj0IWP61RQwEIrXG1knLe83x6DnLFtS1QkyBP8mRtvYAhOfyurq9j0964iBcbO/Q0G0nfWWII2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977505; c=relaxed/simple; bh=+WGfdR3Y4f4MJOabUzkDEAYVtCQqjvgmchKMVu5bAso=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mG5vXLvzuKlTDEduePUHGXHBL6Q7FeQ0qGwVNJ1G8ixts7z9NyH4wn0dWnOCDTLc5KHG7JILPGbelnOOrQ1KbKuC6i3hZ74ALVwH0ASdWWrehWUpgLo0CUbPJhDGSYdD28hTOjzPVfalUQzArSPM2NnznzO2GkXi82m7o6+FhS8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=A/Tbm2wJ; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="A/Tbm2wJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 745871F00898; Mon, 21 Sep 2026 07:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789977500; bh=L/18n8TfyaFMDLnAYEp51nwGZPC1Y8Qvb7X09zPa7Vc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=A/Tbm2wJxme0+N0pPuGOo8l68ppGyPNpDhIC88m3xFlT8W2gs2H0UtOcaFX9iXwkS COcGAExn661bK6PWCy47/Y9KvNRK9Capy8ppPxO3wWFDlJV5njIu70kOH9B8waNcL6 d6b/mpFAr35uy3KaVJDRyPlxeT1kPwDY4TDyg3KiNWt9J1VHjqAzIcp80PjutjKnqS K3D9lhTABsKYOqMjzwN3SKVtRrBzxlWF96B+TLvD3c4i7oouEgH78A03DroLWGeF2Z JuJwF8pMShZX1rKiCS5CtTF1YuKEr/PwoCl8dl2GystyIzixncFflxEEHpkeOvvb/l 2GcjE3lK1EDJw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Harry Yoo , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, Pedro Falcato , Kuniyuki Iwashima , linux-hardening@vger.kernel.org, Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Simon Horman , Jason Xing , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Jiayuan Chen , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 3/7] mm/slab: Add kmem_buckets_destroy() Date: Mon, 21 Sep 2026 00:58:14 -0700 Message-Id: <20260921075820.1718334-3-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260921075811.too.775-kees@kernel.org> References: <20260921075811.too.775-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4729; i=kees@kernel.org; h=from:subject; bh=+WGfdR3Y4f4MJOabUzkDEAYVtCQqjvgmchKMVu5bAso=; b=owGbwMvMwCVmps19z/KJym7G02pJDFkbHs9UUVx8Nfu38g2JFUlnXu/wXSKoUZe4uPxUm0ODS qZpdcy7jlIWBjEuBlkxRZYgO/c4F4+37eHucxVh5rAygQxh4OIUgIkkVjD8z+N5yGySI+I2a/8V v7vmn5ezRey+emMBg+2JhF2itXI8uQz/a1aHH9OSFKp4G/eEN0gx01qHScXVznvTV4XDy6Kd8vg 5AA== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kmem_buckets_create() intentionally had no "destroy" counterpart. Every caller has lived in core kernel code and creates its set once at boot, so nothing has needed to take one down. However, KUnit tests may be built module, so we need it now to support the coming tests. Some caches have size aliases, so the same pointer is stored at more then one index, so we have to save it, clear all matching instances, and then free the saved cache pointer. (This is what the bitmap was tracking before in the "allocation failed" error path.) When CONFIG_SLAB_BUCKETS=3Dn the whole body compiles away, matching the ZERO_SIZE_PTR that kmem_buckets_create() hands back in that configuration. Link: https://lore.kernel.org/all/20240809073309.2134488-1-kees@kernel.org/ Assisted-by: LLM Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Harry Yoo Cc: Andrew Morton Cc: Hao Li Cc: Christoph Lameter Cc: David Rientjes Cc: Roman Gushchin Cc: Cc: Pedro Falcato Cc: Kuniyuki Iwashima Cc: --- include/linux/slab.h | 1 + mm/slab_common.c | 49 +++++++++++++++++++++++++++++++++++++------- 2 files changed, 43 insertions(+), 7 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index cda126def67a..18a2351f9084 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -893,6 +893,7 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *)); +void kmem_buckets_destroy(kmem_buckets *bucket); =20 /* * Bulk allocation and freeing operations. These are accelerated in an diff --git a/mm/slab_common.c b/mm/slab_common.c index cc58f192f349..eb29cfb2f0a9 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -434,12 +434,9 @@ kmem_buckets *kmem_buckets_create(const char *name, sl= ab_flags_t flags, unsigned int usersize, void (*ctor)(void *)) { - unsigned long mask =3D 0; unsigned int idx; kmem_buckets *b; =20 - BUILD_BUG_ON(ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]) > BITS_PER_LONG); - /* * When the separate buckets API is not built in, just return * a non-NULL value for the kmem_buckets pointer, which will be @@ -493,7 +490,6 @@ kmem_buckets *kmem_buckets_create(const char *name, sla= b_flags_t flags, kfree(cache_name); if (WARN_ON(!(*b)[aligned_idx])) goto fail; - set_bit(aligned_idx, &mask); } if (idx !=3D aligned_idx) (*b)[idx] =3D (*b)[aligned_idx]; @@ -502,14 +498,53 @@ kmem_buckets *kmem_buckets_create(const char *name, s= lab_flags_t flags, return b; =20 fail: - for_each_set_bit(idx, &mask, ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL])) - kmem_cache_destroy((*b)[idx]); - kmem_cache_free(kmem_buckets_cache, b); + kmem_buckets_destroy(b); =20 return NULL; } EXPORT_SYMBOL(kmem_buckets_create); =20 +/** + * kmem_buckets_destroy - Destroy a set of caches made by kmem_buckets_cre= ate() + * @bucket: The set to destroy, which may be NULL. + * + * Destroys each cache in @bucket and then frees @bucket itself. As for + * kmem_cache_destroy(), every object allocated from @bucket must have been + * freed beforehand, and @bucket must not be used afterwards. + * + * Context: Process context. May sleep, as kmem_cache_destroy() takes the + * slab mutex and can wait on RCU callbacks for each cache. + */ +void kmem_buckets_destroy(kmem_buckets *bucket) +{ + unsigned int idx, i; + + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS) || ZERO_OR_NULL_PTR(bucket)) + return; + + for (idx =3D 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + struct kmem_cache *cache =3D (*bucket)[idx]; + + if (!cache) + continue; + + /* + * Sizes below arch_slab_minalign() share one cache, which + * kmem_buckets_create() then stores at each of their indices. + * Drop every reference to it before destroying it, so that no + * later pass reads a pointer to a cache that is already gone. + */ + for (i =3D idx; i < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); i++) + if ((*bucket)[i] =3D=3D cache) + (*bucket)[i] =3D NULL; + + kmem_cache_destroy(cache); + } + + kmem_cache_free(kmem_buckets_cache, bucket); +} +EXPORT_SYMBOL(kmem_buckets_destroy); + /* * For a given kmem_cache, kmem_cache_destroy() should only be called * once or there will be a use-after-free problem. The actual deletion --=20 2.34.1 From nobody Thu Sep 24 20:30:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB94544CF29; Mon, 21 Sep 2026 07:58:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977506; cv=none; b=h6d6g6wL+RlzCMiIjA0NjOI5qKpSo+2EEJuWs00641TjV9FdtOKlWirX7O/Ga5Fwen0EP7FJFdAAPddPqNjW8kZMDrmxbqvxVX+c38A/4KjTrNHMz5DOeW/6WwDF4frFc6GBxBJpFDOEdgrTJz+dV2UDiXP5+j8vU/JB4vWLs7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977506; c=relaxed/simple; bh=SnTKIJvg4KJffuAyuRjPcaAaTJbJzSFi0J72zvJPYgY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QNLPXDZaMwNtIcu5BluK10qhdzLLpLNcxeOWWqmKIpM7TbiNYZ0FJN6ItCp1GgEploQZPcA5uNGuVFeUNnYOwt5yAFb5xgvouzjTADEv3ECtcT0cSemkZOHveamkOIl90RiKZkMdi9nwfB5krj0LDYYzjl4EHyhz8wRkW6RPDeU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=h8nZF8ha; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="h8nZF8ha" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84BEA1F0089A; Mon, 21 Sep 2026 07:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789977500; bh=SPdbaNQoVg5IDwmkrXrppVKDvRdhDZUAbimxg5b38Z8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=h8nZF8ham89fKiaZ+iAiVedjQM1Rb6NmE/n3hNFTPyGvwzrlgOgR4gp2zr52ZhVgx CAcDjy/Rb6z1cCgwPIzyCgR5etTaKm/xrBr6VFMeT/a5tzEs9RqIzmQdXgYIrwjv8p sIx1QVr/BfNISn/RDP2AaaM+5aCA2CAXKG8YnuttVRejN8re+QkDChBCZH4PiGfQbf NFK4zXmS6QSzjWwsTh8CsC9wWV5y04OB0fCANqhb2ydIf4FQO/27P06BJsiIqwBoie neZ22vlS57CN8yc/v7LVq8bwBwhXJgH8kv6QTGA9re2KEQDokPvp45KA0hx1qfcdiB OHeE4qj67Kt1Q== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Harry Yoo , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, Pedro Falcato , Kuniyuki Iwashima , linux-hardening@vger.kernel.org, Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Simon Horman , Jason Xing , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Jiayuan Chen , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 4/7] mm/slab: Add tests for the existing kmem_buckets behaviour Date: Mon, 21 Sep 2026 00:58:15 -0700 Message-Id: <20260921075820.1718334-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260921075811.too.775-kees@kernel.org> References: <20260921075811.too.775-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9846; i=kees@kernel.org; h=from:subject; bh=SnTKIJvg4KJffuAyuRjPcaAaTJbJzSFi0J72zvJPYgY=; b=owGbwMvMwCVmps19z/KJym7G02pJDFkbHs/cvWajh9Ls5vQbukJz9C566fP/D/796jybyh27I MlrIqExHaUsDGJcDLJiiixBdu5xLh5v28Pd5yrCzGFlAhnCwMUpABOZosbwv5pHism0f3n/p6AT ft0ZeecD3GNF338s417iK7vUc3tlIsM/fX6ehPenDlef5WHeduuP7b9QsSush3zkayvjfkZe5nV iAgA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kmem_buckets has had no test coverage since it was added. Add tests, including stuff unique to the bucket design: - A bucket allocation comes from a cache of the set's own, and that cache carries SLAB_NO_MERGE. - Each size class is served by the set, including 96 and 192, which are not powers of two and are filled in from an aligned index by kmem_buckets_create(). Sizes above KMALLOC_MAX_CACHE_SIZE go to the page allocator instead, bucket set or not. - With CONFIG_SLAB_BUCKETS=3Dn, kmem_buckets_create() still returns a non-NULL (zero size alloc pointer), so that callers only have to check for failure, and allocations through it come from the general caches. - Destroying a set takes its caches down rather than only freeing the set, which is what a module creating one on each load depends on. A set freed without its caches would leave the names taken, and the next load would warn about every one of them. The tests skip rather than compile out, which is useful for testing the CONFIG_SLAB_BUCKETS=3Dn behaviors. Built and tests passing (with expected skips) on ARCH=3Dx86_64 defconfig with GCC 16.2.0, with CONFIG_SLAB_BUCKETS as y and n. Assisted-by: LLM Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Harry Yoo Cc: Andrew Morton Cc: Hao Li Cc: Christoph Lameter Cc: David Rientjes Cc: Roman Gushchin Cc: Cc: Pedro Falcato Cc: Kuniyuki Iwashima Cc: --- lib/tests/slub_kunit.c | 214 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+) diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c index e3b63f0338d5..d6467dd5cf9b 100644 --- a/lib/tests/slub_kunit.c +++ b/lib/tests/slub_kunit.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include #include #include @@ -474,6 +475,214 @@ static int test_init(struct kunit *test) return 0; } =20 +/* Destroy buckets on test exit so a failed KUNIT_ASSERT_*() doesn't leak.= */ +KUNIT_DEFINE_ACTION_WRAPPER(destroy_buckets, kmem_buckets_destroy, kmem_bu= ckets *); + +#define KUNIT_ASSERT_BUCKETS_CREATED(test, b) \ + do { \ + KUNIT_ASSERT_NOT_NULL(test, b); \ + KUNIT_ASSERT_EQ(test, 0, \ + kunit_add_action_or_reset(test, \ + destroy_buckets, b)); \ + } while (0) + +/* + * The cache an allocation came from, or NULL if it came from no cache at + * all, e.g. a size too big for any of them is served by the page allocato= r. + */ +static struct kmem_cache *cache_of(void *p) +{ + struct slab *slab =3D virt_to_slab(p); + + return slab ? slab->slab_cache : NULL; +} + +/* + * A bucket set exists to keep its allocations out of the caches everything + * else uses, so check the two things that make that true: they come from a + * cache of the set's own, and that cache is never merged into another. + */ +static void test_kmem_buckets_isolation(struct kunit *test) +{ + struct kmem_cache *bucket_cache, *general_cache; + kmem_buckets *b; + void *p, *q; + + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + kunit_skip(test, "needs CONFIG_SLAB_BUCKETS"); + + b =3D kmem_buckets_create("isolated_buckets", 0, 0, INT_MAX, NULL); + KUNIT_ASSERT_BUCKETS_CREATED(test, b); + + /* + * Free each allocation before asserting on the next one: the cache + * outlives its objects, so nothing below needs them, and an assertion + * that leaves one behind would make the deferred teardown report a + * cache that is still in use. + */ + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + bucket_cache =3D cache_of(p); + kfree(p); + KUNIT_ASSERT_NOT_NULL(test, bucket_cache); + + KUNIT_EXPECT_TRUE_MSG(test, strstarts(bucket_cache->name, "isolated_bucke= ts-"), + "expected a bucket cache, got %s", bucket_cache->name); + + /* + * Cache merging is on by default, and a bucket cache merged into a + * same-sized general one would quietly undo the whole separation. + */ + KUNIT_EXPECT_TRUE(test, bucket_cache->flags & SLAB_NO_MERGE); + + q =3D kmalloc(128, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, q); + general_cache =3D cache_of(q); + kfree(q); + KUNIT_ASSERT_NOT_NULL(test, general_cache); + + KUNIT_EXPECT_PTR_NE(test, bucket_cache, general_cache); +} + +/* + * Every size class gets its own cache in the set, including the ones that + * are not powers of two and are filled in from an aligned index. Sizes pa= st + * the largest cache are served by the page allocator, bucket set or not. + */ +static void test_kmem_buckets_sizes(struct kunit *test) +{ + static const size_t sizes[] =3D { 8, 96, 192, 1024, 4096 }; + struct kmem_cache *c; + kmem_buckets *b; + void *p; + int i; + + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + kunit_skip(test, "needs CONFIG_SLAB_BUCKETS"); + + b =3D kmem_buckets_create("sized_buckets", 0, 0, INT_MAX, NULL); + KUNIT_ASSERT_BUCKETS_CREATED(test, b); + + for (i =3D 0; i < ARRAY_SIZE(sizes); i++) { + p =3D kmem_buckets_alloc(b, sizes[i], GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + c =3D cache_of(p); + kfree(p); + KUNIT_ASSERT_NOT_NULL(test, c); + + KUNIT_EXPECT_TRUE_MSG(test, strstarts(c->name, "sized_buckets-"), + "size %zu: expected a bucket cache, got %s", + sizes[i], c->name); + KUNIT_EXPECT_GE(test, c->object_size, sizes[i]); + } + + /* Too big for any cache: a folio from the page allocator, not a slab. */ + p =3D kmem_buckets_alloc(b, KMALLOC_MAX_CACHE_SIZE + 1, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + c =3D cache_of(p); + kfree(p); + + KUNIT_EXPECT_NULL(test, c); +} + +/* + * A bucket cache stands in for a kmalloc cache, so it has to be aligned l= ike + * one. The DMA layer decides whether a buffer needs bouncing from its siz= e, + * on the grounds that a kmalloc cache of that size is already aligned for + * the device, so a weaker alignment here is not something a caller can see + * coming. Without slab debugging the size implies the alignment and this + * holds either way; with it, only the cache's own alignment does. + */ +static void test_kmem_buckets_alignment(struct kunit *test) +{ + static const size_t sizes[] =3D { 128, 512, 2048 }; + struct kmem_cache *bucket_cache, *general_cache; + kmem_buckets *b; + void *p; + int i; + + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + kunit_skip(test, "needs CONFIG_SLAB_BUCKETS"); + + b =3D kmem_buckets_create("aligned_buckets", 0, 0, INT_MAX, NULL); + KUNIT_ASSERT_BUCKETS_CREATED(test, b); + + for (i =3D 0; i < ARRAY_SIZE(sizes); i++) { + p =3D kmem_buckets_alloc(b, sizes[i], GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + bucket_cache =3D cache_of(p); + KUNIT_EXPECT_TRUE_MSG(test, + IS_ALIGNED((unsigned long)p, ARCH_DMA_MINALIGN), + "size %zu: object %p is not %d byte aligned", + sizes[i], p, (int)ARCH_DMA_MINALIGN); + kfree(p); + + p =3D kmalloc(sizes[i], GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + general_cache =3D cache_of(p); + kfree(p); + + KUNIT_ASSERT_NOT_NULL(test, bucket_cache); + KUNIT_ASSERT_NOT_NULL(test, general_cache); + KUNIT_EXPECT_EQ_MSG(test, bucket_cache->align, general_cache->align, + "size %zu: bucket cache aligned to %u, %s to %u", + sizes[i], bucket_cache->align, + general_cache->name, general_cache->align); + } +} + +/* + * With the feature compiled out, kmem_buckets_create() still returns + * something non-NULL so that callers only have to check for failure, and + * allocations through it work (i.e. come from the general caches). + */ +static void test_kmem_buckets_disabled(struct kunit *test) +{ + kmem_buckets *b; + struct kmem_cache *c; + void *p; + + if (IS_ENABLED(CONFIG_SLAB_BUCKETS)) + kunit_skip(test, "only meaningful without CONFIG_SLAB_BUCKETS"); + + b =3D kmem_buckets_create("disabled_buckets", 0, 0, INT_MAX, NULL); + KUNIT_ASSERT_BUCKETS_CREATED(test, b); + + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + c =3D cache_of(p); + kfree(p); + KUNIT_ASSERT_NOT_NULL(test, c); + + KUNIT_EXPECT_TRUE_MSG(test, !strstarts(c->name, "disabled_buckets-"), + "expected a general cache, got %s", c->name); +} + +/* Destroying a set has to take its caches down, not just free the set. */ +static void test_kmem_buckets_destroy(struct kunit *test) +{ + kmem_buckets *b; + void *p; + + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + kunit_skip(test, "needs CONFIG_SLAB_BUCKETS"); + + b =3D kmem_buckets_create("destroyed_buckets", 0, 0, INT_MAX, NULL); + KUNIT_ASSERT_NOT_NULL(test, b); + + /* + * Deliberately leaked, as test_leak_destroy() leaks its own: the + * teardown below has to find it. kmem_cache_destroy() unlists the + * cache either way, so the name is still released. + */ + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL); + KUNIT_EXPECT_NOT_NULL(test, p); + + kmem_buckets_destroy(b); + + KUNIT_EXPECT_EQ(test, 2, slab_errors); +} + static struct kunit_case test_cases[] =3D { KUNIT_CASE(test_clobber_zone), =20 @@ -495,6 +704,11 @@ static struct kunit_case test_cases[] =3D { #if defined(CONFIG_KPROBES) && defined(CONFIG_SMP) KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_kprobe), #endif + KUNIT_CASE(test_kmem_buckets_isolation), + KUNIT_CASE(test_kmem_buckets_sizes), + KUNIT_CASE(test_kmem_buckets_alignment), + KUNIT_CASE(test_kmem_buckets_disabled), + KUNIT_CASE(test_kmem_buckets_destroy), {} }; =20 --=20 2.34.1 From nobody Thu Sep 24 20:30:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43F7144E64F; Mon, 21 Sep 2026 07:58:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977506; cv=none; b=EeB5xtFcW/2zNN2vSEn46Z+iXK+QKe4zhBxvR/m6eLKViJoaZjnDZ4qno0AaqvNb91lcBnchOX6xSAn2M7+0BCGyEMIjqkkvt6pxYSp4Irigq2UA63e0i6DUkgI2PmseuVS8d6hyH8pkrjF5G3r36AcO56DHUT8ysFS981GJxFo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977506; c=relaxed/simple; bh=JttRvJ5Bt7Z1wv5k3OeZaQLY1XifIpQ6GOQSBGFqk9c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mfBlKtZtSA9Nm+Yrr5lwzNorxg+6MTXO4VRCjqPKncxPm3LB8RR2P+Nc92bL98p2LB2wEUrj7Mci513sN7C93D9LFizAIeJsi7TOMCePMx1n0B0cBL1lRT3mPxqpRGLgmSU0Zb/YFVPDHEFXohmmAXziwBcFpRIruc7I/niOLIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=F7gaLOOy; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="F7gaLOOy" Received: by smtp.kernel.org (Postfix) with ESMTPSA id ADF991F0089F; Mon, 21 Sep 2026 07:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789977500; bh=yijECeyPSWNw4ynQbXuj4ormaGJLP5eNf5p0NnJNYTA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=F7gaLOOyxLvwBiW7gSUta9X63c8Dgm/E8HGRTY2VHCQMCjFT/ne11cizs+X+lvWkG wX1KjpOQEm+9px1QueJc++e57qYKlyBU9gfNJ/OnikvetwQJbBfgovwi/TYYvlSyYT s+Q39M9FmFVZJKK2GA7JYy+8Zz/vsJI5AjF6joWESIutRegzES931DIQ1J47nZE6Zm oDYYtzbpGB3Gyi4+f7vp2e4FbaWzWCyfS1mNPJWuAXvHKJ6HyX1fo/SqyJzTgCxk9E mOQe/tKjsAHoRk7mMrUxL3fOXIhZGF/4he05kJ1a86y6olq9/s3BVPdjhFxOUuClPD m6RNtXsEiz+VQ== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Harry Yoo , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, Pedro Falcato , Kuniyuki Iwashima , linux-hardening@vger.kernel.org, Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Simon Horman , Jason Xing , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Jiayuan Chen , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 5/7] mm/slab: Provide kmalloc type fallback for bucket allocations Date: Mon, 21 Sep 2026 00:58:16 -0700 Message-Id: <20260921075820.1718334-5-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260921075811.too.775-kees@kernel.org> References: <20260921075811.too.775-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6106; i=kees@kernel.org; h=from:subject; bh=JttRvJ5Bt7Z1wv5k3OeZaQLY1XifIpQ6GOQSBGFqk9c=; b=owGbwMvMwCVmps19z/KJym7G02pJDFkbHs906rW+v3eOw4TVFQofjDkNs+Xbee77upmkZh9NU T2y7pJYRykLgxgXg6yYIkuQnXuci8fb9nD3uYowc1iZQIYwcHEKwETCOBj+x/c9ThZ6eXPZWZMJ qx8J/r41+aG0x+EkdlaPxb5G3Bo3pRkZ9py0ZZ9T75xktja5xzL9Tojb0ektIf16vUxb6gu3Kv3 gAQA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kmem_buckets_create() clones kmalloc_caches[KMALLOC_NORMAL]. kmalloc_slab() figures out the kmalloc type the caller asks for, but then ignored it whenever a bucket set was in use, returning a normal cache regardless. This would be a problem if a caller asked for GFP_DMA, __GFP_ACCOUNT, etc. None of the current users do this, so there is problem, but it makes adding new users fragile. For example, skb data[1] needs to handle GFP_DMA (rarely) and __GFP_ACCOUNT (often). Send those allocations to the general caches instead so nothing breaks and regular allocations remain isolated with the bucket. The kmem_bucket_type enum contains only a single item here, but will be expanded in the next patch. Built and tests pass with ARCH=3Dx86_64 defconfig with GCC 16.2.0, with CONFIG_SLAB_BUCKETS as y and n. Assisted-by: LLM Link: https://lore.kernel.org/all/04debe19-bbe8-4b5f-9668-753d1f97832d@redh= at.com/ [1] Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Harry Yoo Cc: Andrew Morton Cc: Hao Li Cc: Christoph Lameter Cc: David Rientjes Cc: Roman Gushchin Cc: Cc: Pedro Falcato Cc: Kuniyuki Iwashima Cc: --- include/linux/slab.h | 13 +++++++++++ mm/slab.h | 23 ++++++++++++++++-- lib/tests/slub_kunit.c | 53 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 87 insertions(+), 2 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 18a2351f9084..ab9ab3d34847 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -742,6 +742,19 @@ typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT= _HIGH + 1]; =20 extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; =20 +/* + * The kmalloc types a bucket set can hold a copy of. This is deliberately= not + * enum kmalloc_cache_type: the KMALLOC_PARTITION copies are all "normal" = to a + * bucket set, which already separates what they were there to separate, so + * indexing by those would mean up to KMALLOC_PARTITION_CACHES_NR unusable + * rows per set. Allocations of any type not listed here are served by the + * general caches. + */ +enum kmem_bucket_type { + KMEM_BUCKET_NORMAL =3D 0, + NR_KMEM_BUCKET_TYPES +}; + /* * Define gfp bits that should not be set for KMALLOC_NORMAL. */ diff --git a/mm/slab.h b/mm/slab.h index 8fd6835e4235..7f1bfee83b92 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -421,6 +421,26 @@ static inline unsigned int size_index_elem(unsigned in= t bytes) return (bytes - 1) / 8; } =20 +/* + * Which set of buckets to use for the given kmalloc_cache_type. If not + * handled by the kmem_buckets, fall back to general caches. + */ +static inline kmem_buckets * +kmalloc_choose_bucket(kmem_buckets *bucket, enum kmalloc_cache_type type) +{ + enum kmem_bucket_type btype; + + if (!bucket) + return &kmalloc_caches[type]; + + if (type <=3D KMALLOC_PARTITION_END) + btype =3D KMEM_BUCKET_NORMAL; + else + return &kmalloc_caches[type]; /* No set holds a row for it. */ + + return &bucket[btype]; +} + /* * Find the kmem_cache structure that serves a given size of * allocation @@ -438,8 +458,7 @@ kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags,= kmalloc_token_t token, if (alloc_flags & SLAB_ALLOC_NO_OBJ_EXT) type =3D KMALLOC_NO_OBJ_EXT; =20 - if (!b) - b =3D &kmalloc_caches[type]; + b =3D kmalloc_choose_bucket(b, type); if (size <=3D 192) index =3D kmalloc_size_index[size_index_elem(size)]; else diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c index d6467dd5cf9b..823607e06248 100644 --- a/lib/tests/slub_kunit.c +++ b/lib/tests/slub_kunit.c @@ -683,6 +683,58 @@ static void test_kmem_buckets_destroy(struct kunit *te= st) KUNIT_EXPECT_EQ(test, 2, slab_errors); } =20 +/* + * A bucket set holds only the kmalloc types it was created with, so an + * allocation that asks for a different one has to come from the general + * caches. Check that it does, rather than being served a normal cache that + * does not satisfy what the flags asked for. + */ +static void test_kmem_buckets_type_fallback(struct kunit *test) +{ + struct kmem_cache *c; + kmem_buckets *b; + void *p; + + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + kunit_skip(test, "needs CONFIG_SLAB_BUCKETS"); + + b =3D kmem_buckets_create("test_buckets", 0, 0, INT_MAX, NULL); + KUNIT_ASSERT_BUCKETS_CREATED(test, b); + + /* A plain allocation stays isolated in the bucket set. */ + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + c =3D cache_of(p); + kfree(p); + KUNIT_ASSERT_NOT_NULL(test, c); + + KUNIT_EXPECT_TRUE_MSG(test, strstarts(c->name, "test_buckets-"), + "expected a bucket cache, got %s", c->name); + + /* One that needs ZONE_DMA cannot, so it falls back. */ + if (IS_ENABLED(CONFIG_ZONE_DMA)) { + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL | GFP_DMA); + KUNIT_ASSERT_NOT_NULL(test, p); + c =3D cache_of(p); + kfree(p); + KUNIT_ASSERT_NOT_NULL(test, c); + + KUNIT_EXPECT_TRUE_MSG(test, strstarts(c->name, "dma-kmalloc-"), + "expected a DMA cache, got %s", c->name); + } + + /* Nor can one that has to be accounted. */ + if (IS_ENABLED(CONFIG_MEMCG) && !mem_cgroup_kmem_disabled()) { + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL | __GFP_ACCOUNT); + KUNIT_ASSERT_NOT_NULL(test, p); + c =3D virt_to_slab(p)->slab_cache; + kfree(p); + + KUNIT_EXPECT_TRUE_MSG(test, strstarts(c->name, "kmalloc-cg-"), + "expected an accounted cache, got %s", c->name); + } +} + static struct kunit_case test_cases[] =3D { KUNIT_CASE(test_clobber_zone), =20 @@ -709,6 +761,7 @@ static struct kunit_case test_cases[] =3D { KUNIT_CASE(test_kmem_buckets_alignment), KUNIT_CASE(test_kmem_buckets_disabled), KUNIT_CASE(test_kmem_buckets_destroy), + KUNIT_CASE(test_kmem_buckets_type_fallback), {} }; =20 --=20 2.34.1 From nobody Thu Sep 24 20:30:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43BA844E64A; Mon, 21 Sep 2026 07:58:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977507; cv=none; b=XMnIXJ5PD1YCFM+C0cULcQGIWT0rOUbOch4BA48nIsNIeOHIbJPVIx5CGR/tuEL0b8cKspwrTvvFgf6IBKaTcj4brrlfycqvyFD5DfWLsofxOW8w4dkvekh1AqexGwn0wYXDVkdTV955uVxmezBugWTzQUZdyOrxzpdu6zAvKI8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977507; c=relaxed/simple; bh=TFRiW7uKuGLJ5RAboQjREyhJzsC0Jl/peT6jDjy0Hc8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TMkK8wIcEwX0ktb2qQFi0xy8vUTsvqBnP3Vb/ftNGgQa3at5ZHZ6BWCHHj2Jx15HiLH56rc8J8sZcilGNfc6WixbG18M3LMcdlw1aCdsjMyv3lWQRQdhEch2cRbGWjzi2PQd8aOX+EKvlMitK7OP7g5aJzTRnXEXgn5pnGbNmaQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DnDCIUYj; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DnDCIUYj" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B0A571F008A0; Mon, 21 Sep 2026 07:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789977500; bh=5fdg4lhf7l6/dVSDg/4vChlOULkSsDgKH3aMkmNkZpA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=DnDCIUYj9hIZIXpHnIxt/PK9wHSQiUd5i9g6ixo4ueszCOAj3NYiDtyBq32Urq9BO hUF8WallP+4OY9VsnqSPkg/q4maHxSAUsX4yszDcroGJJGloWqOjd5UckiLLVY/qRF 72YwFTtVSptyazK+lj5BpJv4QiR/Jux+jKw2iVc1PfiBdosAYqRaIkpXwO5C9qOJl1 3QATLFG1dmsdGJm/PnGtlJGvDjXp6h5SH3l3dMUZGaFEFaRgz5lfRIbRnO8HXGJbme /y7UvrhOoDz8eYn6oqOVIfhMSN1efoqlR9THHYJV8otFKeAK6rElh7ACMgm9Qknqyt CZQx7pn5AzUaA== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Harry Yoo , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, Pedro Falcato , Kuniyuki Iwashima , linux-hardening@vger.kernel.org, Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Simon Horman , Jason Xing , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Jiayuan Chen , Willem de Bruijn , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v4 6/7] mm/slab: Let a bucket set handle __GFP_ACCOUNT Date: Mon, 21 Sep 2026 00:58:17 -0700 Message-Id: <20260921075820.1718334-6-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260921075811.too.775-kees@kernel.org> References: <20260921075811.too.775-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=17623; i=kees@kernel.org; h=from:subject; bh=TFRiW7uKuGLJ5RAboQjREyhJzsC0Jl/peT6jDjy0Hc8=; b=owGbwMvMwCVmps19z/KJym7G02pJDFkbHs+S+xNt8WTXlO75v8Qm+FctfPnxAduiDvMAc8t04 QM7u3MmdZSyMIhxMciKKbIE2bnHuXi8bQ93n6sIM4eVCWQIAxenAEzkz2SGf5bsj/dEyj5S+Hxj OsumzOK4ivbMs3p1BWk3+ZWnfF61fQIjw7TIPu0+jZY/R8VZhZNZIrmM32mEfjs4p0Ny+TRDdda PfAA= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A bucket set holds one row of caches, cloned from KMALLOC_NORMAL, and an allocation of any other kmalloc type falls back to the general caches. Extend this to handle __GFP_ACCOUNT, so that a single bucket user can isolate either GFP_KERNEL or GFP_KERNEL_ACCOUNT allocations, as is needed for skb data, where AF_UNIX uses: sk->sk_allocation =3D GFP_KERNEL_ACCOUNT; The coverage is selected at bucket creation time: b =3D kmem_buckets_create_types(name, flags, 0, INT_MAX, NULL, BIT(KMEM_BUCKET_NORMAL) | BIT(KMEM_BUCKET_CGROUP)); The prior kmem_buckets_create() function keeps its name and defaults to only KMEM_BUCKET_NORMAL, leaving existing users as-is. Only the accounted type is offered. Nothing wants a reclaimable or no-obj-ext row, and of the twelve places passing GFP_DMA to an skb allocator, all rare hardware: b44, b43legacy, prestera and s390 ctcm. The choice is made at creation rather than every set getting every type because the rows, when populated, are not free. Each holds 13 caches, and a cache is a 1208 byte struct plus an unconditional per-cpu allocation, a node struct, and an entry in /proc/slabinfo and under /sys/kernel/slab. KMEM_BUCKET_CGROUP collapses to KMEM_BUCKET_NORMAL without CONFIG_MEMCG, exactly as KMALLOC_CGROUP does, so NR_KMEM_BUCKET_TYPES is 1 there and a bucket set is the same single row it is today. Where the type is asked for but the system is not creating caches of it (under "cgroup.memory=3Dnokmem") the row is aliased to the normal one, as new_kmalloc_cache() does for the general caches, so those allocations stay isolated rather than falling back to the general caches. Built and tests pass (and skip as expected) on ARCH=3Dx86_64 defconfig with GCC 16.2.0 in all combinations of CONFIG_SLAB_BUCKETS=3Dy/n and CONFIG_MEMCG=3Dy/n/y+"cgroup.memory=3Dnokmem". Assisted-by: LLM Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Harry Yoo Cc: Andrew Morton Cc: Hao Li Cc: Christoph Lameter Cc: David Rientjes Cc: Roman Gushchin Cc: Cc: Pedro Falcato Cc: Kuniyuki Iwashima Cc: --- include/linux/slab.h | 45 ++++++++++++- mm/slab.h | 23 ++++++- lib/tests/slub_kunit.c | 65 +++++++++++++++--- mm/slab_common.c | 145 ++++++++++++++++++++++++++++++++--------- 4 files changed, 235 insertions(+), 43 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index ab9ab3d34847..bbc27e4e5e1c 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -752,6 +752,11 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; */ enum kmem_bucket_type { KMEM_BUCKET_NORMAL =3D 0, +#ifdef CONFIG_MEMCG + KMEM_BUCKET_CGROUP, +#else + KMEM_BUCKET_CGROUP =3D KMEM_BUCKET_NORMAL, +#endif NR_KMEM_BUCKET_TYPES }; =20 @@ -903,9 +908,43 @@ void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s= , struct list_lru *lru, bool kmem_cache_charge(void *objp, gfp_t gfpflags); void kmem_cache_free(struct kmem_cache *s, void *objp); =20 -kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, - unsigned int useroffset, unsigned int usersize, - void (*ctor)(void *)); +kmem_buckets *kmem_buckets_create_types(const char *name, slab_flags_t fla= gs, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *), + unsigned int type_mask); + +/** + * kmem_buckets_create - Create a set of caches that handle dynamic sized + * allocations via kmem_buckets_alloc() + * @name: A prefix string which is used in /proc/slabinfo to identify this + * cache. The individual caches with have their sizes as the suffix. + * @flags: SLAB flags (see kmem_cache_create() for details). + * @useroffset: Starting offset within an allocation that may be copied + * to/from userspace. + * @usersize: How many bytes, starting at @useroffset, may be copied + * to/from userspace. + * @ctor: A constructor for the objects, run when new allocations are made. + * + * Covers KMEM_BUCKET_NORMAL only. Allocations needing another kmalloc type + * are served by the general caches, keeping the type they asked for and + * losing only the isolation. Use kmem_buckets_create_types() to cover mor= e. + * + * Context: Cannot be called within an interrupt, but can be interrupted. + * + * Return: a pointer to the cache on success, NULL on failure. When + * CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and + * subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). + * (i.e. callers only need to check for NULL on failure.) + */ +static inline kmem_buckets *kmem_buckets_create(const char *name, slab_fla= gs_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + return kmem_buckets_create_types(name, flags, useroffset, usersize, ctor, + BIT(KMEM_BUCKET_NORMAL)); +} + void kmem_buckets_destroy(kmem_buckets *bucket); =20 /* diff --git a/mm/slab.h b/mm/slab.h index 7f1bfee83b92..2af44e09edda 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -435,10 +435,31 @@ kmalloc_choose_bucket(kmem_buckets *bucket, enum kmal= loc_cache_type type) =20 if (type <=3D KMALLOC_PARTITION_END) btype =3D KMEM_BUCKET_NORMAL; + else if (IS_ENABLED(CONFIG_MEMCG) && type =3D=3D KMALLOC_CGROUP) + btype =3D KMEM_BUCKET_CGROUP; else return &kmalloc_caches[type]; /* No set holds a row for it. */ =20 - return &bucket[btype]; + /* + * Either this row was created, and holds a cache everywhere the + * general caches hold one, or it was never created and holds nothing. + * Test with the KMALLOC_SHIFT_LOW which exists in every configuration. + */ + if (likely(bucket[btype][KMALLOC_SHIFT_LOW])) + return &bucket[btype]; + + /* + * A row this set _could_ have held, but was not created with: the type + * mask passed to kmem_buckets_create_types() did not cover what its + * callers actually tried to allocate. Report the mismatch but still + * fall back to the general caches. + * + * At present, only __GFP_ACCOUNT can be missing. + */ + WARN_ONCE(1, + "kmem_buckets: __GFP_ACCOUNT needs BIT(KMEM_BUCKET_CGROUP) in create m= ask\n"); + + return &kmalloc_caches[type]; } =20 /* diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c index 823607e06248..58f800582170 100644 --- a/lib/tests/slub_kunit.c +++ b/lib/tests/slub_kunit.c @@ -723,15 +723,63 @@ static void test_kmem_buckets_type_fallback(struct ku= nit *test) "expected a DMA cache, got %s", c->name); } =20 - /* Nor can one that has to be accounted. */ - if (IS_ENABLED(CONFIG_MEMCG) && !mem_cgroup_kmem_disabled()) { - p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL | __GFP_ACCOUNT); - KUNIT_ASSERT_NOT_NULL(test, p); - c =3D virt_to_slab(p)->slab_cache; - kfree(p); + /* + * An accounted allocation would fall back too, but a bucket set can + * hold that type, so reaching the fallback means the create mask was + * wrong and kmalloc_slab() warns. Not exercised here for that reason; + * test_kmem_buckets_type_covered() checks the type that is asked for. + */ +} + +/* + * A bucket set created for a kmalloc type keeps those allocations isolated + * too, rather than sending them to the general caches. Where nothing crea= tes + * accounted caches at all, the row aliases the normal one, so this also + * covers tearing down a set whose rows share their caches. + */ +static void test_kmem_buckets_type_covered(struct kunit *test) +{ + struct kmem_cache *c, *normal_cache; + kmem_buckets *b; + void *p; + + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + kunit_skip(test, "needs CONFIG_SLAB_BUCKETS"); + + b =3D kmem_buckets_create_types("covered_buckets", 0, 0, INT_MAX, NULL, + BIT(KMEM_BUCKET_NORMAL) | + BIT(KMEM_BUCKET_CGROUP)); + KUNIT_ASSERT_BUCKETS_CREATED(test, b); + + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, p); + normal_cache =3D cache_of(p); + kfree(p); + KUNIT_ASSERT_NOT_NULL(test, normal_cache); =20 - KUNIT_EXPECT_TRUE_MSG(test, strstarts(c->name, "kmalloc-cg-"), - "expected an accounted cache, got %s", c->name); + KUNIT_EXPECT_TRUE_MSG(test, strstarts(normal_cache->name, "covered_bucket= s-128"), + "expected the normal bucket cache, got %s", + normal_cache->name); + + /* Accounted, and still in the bucket set rather than kmalloc-cg-*. */ + p =3D kmem_buckets_alloc(b, 128, GFP_KERNEL | __GFP_ACCOUNT); + KUNIT_ASSERT_NOT_NULL(test, p); + c =3D cache_of(p); + kfree(p); + KUNIT_ASSERT_NOT_NULL(test, c); + + if (IS_ENABLED(CONFIG_MEMCG) && !mem_cgroup_kmem_disabled()) { + KUNIT_EXPECT_TRUE_MSG(test, strstarts(c->name, "covered_buckets-cg-"), + "expected the accounted bucket cache, got %s", + c->name); + KUNIT_EXPECT_TRUE(test, c->flags & SLAB_ACCOUNT); + } else { + /* + * Nothing is creating accounted caches, so the row aliases + * the normal one and the allocation lands there -- isolated + * still, just not separately accounted. + */ + KUNIT_EXPECT_PTR_EQ(test, c, normal_cache); } } =20 @@ -762,6 +810,7 @@ static struct kunit_case test_cases[] =3D { KUNIT_CASE(test_kmem_buckets_disabled), KUNIT_CASE(test_kmem_buckets_destroy), KUNIT_CASE(test_kmem_buckets_type_fallback), + KUNIT_CASE(test_kmem_buckets_type_covered), {} }; =20 diff --git a/mm/slab_common.c b/mm/slab_common.c index eb29cfb2f0a9..3861d8b3d849 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -410,9 +410,15 @@ EXPORT_SYMBOL(__kmem_cache_create_args); =20 static struct kmem_cache *kmem_buckets_cache __ro_after_init; =20 +static int kmem_buckets_create_row(kmem_buckets *b, + enum kmalloc_cache_type type, + const char *name, + slab_flags_t flags, unsigned int useroffset, + unsigned int usersize, void (*ctor)(void *)); + /** - * kmem_buckets_create - Create a set of caches that handle dynamic sized - * allocations via kmem_buckets_alloc() + * kmem_buckets_create_types - Create a set of caches that handle dynamic = sized + * allocations via kmem_buckets_alloc() * @name: A prefix string which is used in /proc/slabinfo to identify this * cache. The individual caches with have their sizes as the suffix. * @flags: SLAB flags (see kmem_cache_create() for details). @@ -421,6 +427,11 @@ static struct kmem_cache *kmem_buckets_cache __ro_afte= r_init; * @usersize: How many bytes, starting at @useroffset, may be copied * to/from userspace. * @ctor: A constructor for the objects, run when new allocations are made. + * @type_mask: Which kmalloc types to hold caches for, as a mask of + * BIT(KMEM_BUCKET_*). KMEM_BUCKET_NORMAL is always included. + * Allocations of a type that is not covered are served by the + * general caches instead, so a caller need not know in advance + * which types its own callers will ask for. * * Context: Cannot be called within an interrupt, but can be interrupted. * @@ -429,12 +440,13 @@ static struct kmem_cache *kmem_buckets_cache __ro_aft= er_init; * subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). * (i.e. callers only need to check for NULL on failure.) */ -kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, - unsigned int useroffset, - unsigned int usersize, - void (*ctor)(void *)) +kmem_buckets *kmem_buckets_create_types(const char *name, slab_flags_t fla= gs, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *), + unsigned int type_mask) { - unsigned int idx; + enum kmem_bucket_type btype; kmem_buckets *b; =20 /* @@ -453,20 +465,87 @@ kmem_buckets *kmem_buckets_create(const char *name, s= lab_flags_t flags, return NULL; =20 flags |=3D SLAB_NO_MERGE; + type_mask |=3D BIT(KMEM_BUCKET_NORMAL); + + for (btype =3D 0; btype < NR_KMEM_BUCKET_TYPES; btype++) { + enum kmalloc_cache_type src =3D KMALLOC_NORMAL; + slab_flags_t type_flags =3D 0; + + if (!(type_mask & BIT(btype))) + continue; + + /* + * Under CONFIG_MEMCG=3Dn the two types are the same value, so + * the IS_ENABLED() is what keeps the normal row out of here. + */ + if (IS_ENABLED(CONFIG_MEMCG) && btype =3D=3D KMEM_BUCKET_CGROUP) { + /* + * Aliasing below reads the normal row, so this loop + * must have built it already. That holds only while + * the normal type sorts first. + */ + BUILD_BUG_ON(KMEM_BUCKET_CGROUP <=3D KMEM_BUCKET_NORMAL); + + /* + * Nothing anywhere is creating accounted caches, as + * with "cgroup.memory=3Dnokmem". Point this row's + * entries at the normal row's caches, the way + * new_kmalloc_cache() aliases kmalloc_caches[] for + * the same reason. Leaving the row empty instead + * would send every accounted allocation out of the + * set and into the general caches. + */ + if (mem_cgroup_kmem_disabled()) { + memcpy(b[btype], b[KMEM_BUCKET_NORMAL], + sizeof(b[btype])); + continue; + } + + type_flags =3D SLAB_ACCOUNT; + src =3D KMALLOC_CGROUP; + } + + if (kmem_buckets_create_row(&b[btype], src, name, + flags | type_flags, useroffset, + usersize, ctor)) + goto fail; + } + + return b; + +fail: + kmem_buckets_destroy(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create_types); + +/* + * Build one row of @b by mirroring the general caches of @type: a cache p= er + * kmalloc size, each named "@name-" followed by that cache's own suffix, = so + * a row of KMALLOC_CGROUP ("kmalloc-cg-96") gets "@name-cg-96". + */ +static int kmem_buckets_create_row(kmem_buckets *b, + enum kmalloc_cache_type type, + const char *name, + slab_flags_t flags, unsigned int useroffset, + unsigned int usersize, void (*ctor)(void *)) +{ + unsigned int idx; =20 for (idx =3D 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { char *short_size, *cache_name; unsigned int cache_useroffset, cache_usersize; unsigned int size, aligned_idx; =20 - if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + if (!kmalloc_caches[type][idx]) continue; =20 - size =3D kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + size =3D kmalloc_caches[type][idx]->object_size; if (!size) continue; =20 - short_size =3D strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + short_size =3D strchr(kmalloc_caches[type][idx]->name, '-'); if (WARN_ON(!short_size)) goto fail; =20 @@ -484,7 +563,7 @@ kmem_buckets *kmem_buckets_create(const char *name, sla= b_flags_t flags, if (WARN_ON(!cache_name)) goto fail; (*b)[aligned_idx] =3D kmem_cache_create_usercopy(cache_name, size, - kmalloc_caches[KMALLOC_NORMAL][idx]->align, + kmalloc_caches[type][idx]->align, flags, cache_useroffset, cache_usersize, ctor); kfree(cache_name); @@ -495,14 +574,11 @@ kmem_buckets *kmem_buckets_create(const char *name, s= lab_flags_t flags, (*b)[idx] =3D (*b)[aligned_idx]; } =20 - return b; + return 0; =20 fail: - kmem_buckets_destroy(b); - - return NULL; + return -ENOMEM; } -EXPORT_SYMBOL(kmem_buckets_create); =20 /** * kmem_buckets_destroy - Destroy a set of caches made by kmem_buckets_cre= ate() @@ -517,28 +593,35 @@ EXPORT_SYMBOL(kmem_buckets_create); */ void kmem_buckets_destroy(kmem_buckets *bucket) { + enum kmem_bucket_type btype, t; unsigned int idx, i; =20 if (!IS_ENABLED(CONFIG_SLAB_BUCKETS) || ZERO_OR_NULL_PTR(bucket)) return; =20 - for (idx =3D 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { - struct kmem_cache *cache =3D (*bucket)[idx]; + for (btype =3D 0; btype < NR_KMEM_BUCKET_TYPES; btype++) { + for (idx =3D 0; idx < ARRAY_SIZE(bucket[btype]); idx++) { + struct kmem_cache *cache =3D bucket[btype][idx]; =20 - if (!cache) - continue; + if (!cache) + continue; =20 - /* - * Sizes below arch_slab_minalign() share one cache, which - * kmem_buckets_create() then stores at each of their indices. - * Drop every reference to it before destroying it, so that no - * later pass reads a pointer to a cache that is already gone. - */ - for (i =3D idx; i < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); i++) - if ((*bucket)[i] =3D=3D cache) - (*bucket)[i] =3D NULL; + /* + * A cache is reachable from more than one entry: sizes + * below arch_slab_minalign() share one, and a row that + * kmem_buckets_create_types() aliased onto the normal + * one under "cgroup.memory=3Dnokmem" holds all of them a + * second time. Drop every reference before destroying + * it, so that no later pass reads a pointer to a cache + * that is already gone. + */ + for (t =3D 0; t < NR_KMEM_BUCKET_TYPES; t++) + for (i =3D 0; i < ARRAY_SIZE(bucket[t]); i++) + if (bucket[t][i] =3D=3D cache) + bucket[t][i] =3D NULL; =20 - kmem_cache_destroy(cache); + kmem_cache_destroy(cache); + } } =20 kmem_cache_free(kmem_buckets_cache, bucket); @@ -1088,7 +1171,7 @@ void __init create_kmalloc_caches(void) =20 if (IS_ENABLED(CONFIG_SLAB_BUCKETS)) kmem_buckets_cache =3D kmem_cache_create("kmalloc_buckets", - sizeof(kmem_buckets), + sizeof(kmem_buckets) * NR_KMEM_BUCKET_TYPES, 0, SLAB_NO_MERGE, NULL); } =20 --=20 2.34.1 From nobody Thu Sep 24 20:30:33 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D510A44F542; Mon, 21 Sep 2026 07:58:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977506; cv=none; b=j/S0Km0WBuvpZjDjbBFQ1eazMwN9IYbTyx1DnZope4jS46266vy0SQ5v0ihlj5tONtGRzFw9ZcauRJaOMl/ZzVG1rvK0NBW2K3PywZUuK96F8pYz3Y4hVgtZyuq1gZkblO1EQqMEqCU4vhJV+TwiBwD40l4Wd5AftL+5FDYO/wQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789977506; c=relaxed/simple; bh=5/W6d8/fGN0H8flCw1+s2MH+b1ysv235gOyZUqiA6PY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OkiOn/ybPIVpZWUN1gJ0f2y+TXRctw4+PK4Frb2BPDp2GDg47Ch6UpFy0bpplUkXdempykUd0NMMTuU2UUEqyCHukYr/0XDfl9ItHTmp4HP4izfNBAM3EUzS5rPAXHF8B5XYj8A9cxuFP68cWt+8EH33RKAE3W5YgDg41Oe0C6U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=l5l6xMLr; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="l5l6xMLr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B30A71F0092B; Mon, 21 Sep 2026 07:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789977500; bh=NYCJwCjTojZoMSg1K1fjM/5lK8uQavHZIHSOSFY3WTI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=l5l6xMLr+mdsyD4v9Exth0ro3zP53iCdhEmLvIqv/LW76z/hhaaHJOsgHfnUWXIZ+ GrnzjyQXahhr94V3EyvWJwwhX4xXN55NzWW4lpD/CtK0uRf6PfPC8XxSzxp0FfCpnt xyz9yUy+4l5YMv/tWafJttCsw3tx+I6jyPbWuVjkH1/6SRaQvoLbO715gjAw1Ze8Wx bH6sLPxEJTj6YROx4jOtMDMQy9vbJTxIY2/KuP9pFMtgx4qmMFjUmfodEasuIIfQMM 6mWXurW0lopmPAjxJnZxWvYuLY8bAis2OPiOKTuxS/9u3qCpxQleshG2Gnp44ctCqm Zuw0hGkOFp+0g== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Pedro Falcato , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Willem de Bruijn , Jason Xing , netdev@vger.kernel.org, Kuniyuki Iwashima , linux-hardening@vger.kernel.org, Harry Yoo , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Jiayuan Chen , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 7/7] net: skb: isolate skb data area allocations into a separate bucket Date: Mon, 21 Sep 2026 00:58:18 -0700 Message-Id: <20260921075820.1718334-7-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260921075811.too.775-kees@kernel.org> References: <20260921075811.too.775-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3388; i=kees@kernel.org; h=from:subject; bh=YbmMITYuL1x/iTXXiiC6TtrZwPTf+P8ee7ihm7zXbV8=; b=owGbwMvMwCVmps19z/KJym7G02pJDFkbHs96Od9twi4pFTetfSFhyZfmVrMo1B9ll3bbeJ/t7 pcFHYp2HaUsDGJcDLJiiixBdu5xLh5v28Pd5yrCzGFlAhnCwMUpABPZZc7wv3rT2r5XJVMCz6wK vqIT9T/GlW9Rd9/7p/57/rs+yFdZ/I/hf3DgwrXRchzeZREev01/vRJKDn01aQmLgKWqcFv/nrn H+QE= X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Pedro Falcato SKB data area allocations (as done from alloc_skb()) use kmalloc(). These allocations can be variably sized and their contents can be more or less controlled from userspace, which makes them useful for attackers that want to overwrite a use-after-free'd object from the same kmalloc slab (which often just requires the sizes to roughly match into the same kmalloc bucket). [0] is an easy example of an exploit that uses netlink skb allocation to target another similarly-sized accidentally freed object. While other mitigations like CONFIG_RANDOM_KMALLOC_CACHES exist, these are probabilistic. Use the existing kmem buckets API to further isolate these allocations in a guaranteed fashion, when CONFIG_SLAB_BUCKETS=3Dy. Ask for the accounted kmalloc type as well as the normal one. AF_UNIX sets sk_allocation to GFP_KERNEL_ACCOUNT, so without it every AF_UNIX skb data area would fall back to the general caches, and those are the ones most worth isolating. GFP_DMA is left to fall back, being passed to an skb allocator only by rare devices. Link: https://github.com/google/security-research/blob/master/pocs/linux/ke= rnelctf/CVE-2023-4207_lts_cos_mitigation_2/docs/exploit.md [0] Reviewed-by: Kees Cook Signed-off-by: Pedro Falcato --- Cc: "David S. Miller" Cc: Eric Dumazet Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Simon Horman Cc: Willem de Bruijn Cc: Jason Xing Cc: Cc: Pedro Falcato Cc: Kuniyuki Iwashima Cc: --- net/core/skbuff.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 966af3beed94..865eed3c57d1 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -586,6 +586,8 @@ struct sk_buff *napi_build_skb(void *data, unsigned int= frag_size) } EXPORT_SYMBOL(napi_build_skb); =20 +static kmem_buckets *skb_data_buckets __ro_after_init; + static void *kmalloc_pfmemalloc(size_t obj_size, gfp_t flags, int node) { if (!gfp_pfmemalloc_allowed(flags)) @@ -593,7 +595,8 @@ static void *kmalloc_pfmemalloc(size_t obj_size, gfp_t = flags, int node) if (!obj_size) return kmem_cache_alloc_node(net_hotdata.skb_small_head_cache, flags, node); - return kmalloc_node_track_caller(obj_size, flags, node); + return kmem_buckets_alloc_node_track_caller(skb_data_buckets, obj_size, + flags, node); } =20 /* @@ -634,7 +637,7 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t = flags, int node, * Try a regular allocation, when that fails and we're not entitled * to the reserves, fail. */ - obj =3D kmalloc_node_track_caller(obj_size, + obj =3D kmem_buckets_alloc_node_track_caller(skb_data_buckets, obj_size, flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node); if (likely(obj)) @@ -5235,6 +5238,10 @@ void __init skb_init(void) 0, SKB_SMALL_HEAD_HEADROOM, NULL); + skb_data_buckets =3D kmem_buckets_create_types("skb_data", SLAB_PANIC, 0, + INT_MAX, NULL, + BIT(KMEM_BUCKET_NORMAL) | + BIT(KMEM_BUCKET_CGROUP)); skb_extensions_init(); } =20 --=20 2.34.1