From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A28C43B6E5; Wed, 15 Jul 2026 10:10:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110252; cv=none; b=S5vTsF+qEYGi+8Yx+8264PDPeDK7gDbIRv5U0bmDAIm4PicHfFj5SVC00/yHMHPtKYpT24H09ezHLAzn7bo2w+BOjYaT1GAB9RiOQ0gbFCXo0Tao2AoYfi+cCtXQE6KGLeu+q2eDEQpMa+uUNfIF9/kANtGqCs4QvNqH9D9YVT8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110252; c=relaxed/simple; bh=qrvBi037NaDO5Ac9ofIMepjLFwbLj4Xq6ZZHsuekDEQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Ikn17uTBFDdNNsmJl0TkaWeYzZSVXvgdnRYGT6eyqIT9obdRb1VXEYtaSGosnMDlKdGLlEM1SrUwOGfbUiHyvv5FeVS+dCxDBC71C9tk+E1U8HZIQatdcb49vs2r/udxRpTrqzzvCN7Drh9KesztOGfaI9/V7DRCHE3745eYN2o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RrmWcN1z; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RrmWcN1z" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0AE2F1F000E9; Wed, 15 Jul 2026 10:10:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110250; bh=j/MNUihFnJ5m2LB0LV8GlcwqRsbdT05/VOgE/B2r1H8=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=RrmWcN1zFUQL1nATXeEdG5TTLgpmKdOGfLgt4DJn3a2jKiMPGqNcTxonnc7uopQua SX5xTyAFXfths7sBj+EG8Bw3uI05RRsRknVthAW5YiEzYKpajVnvd5POJcPO8HjcKk gOYmeN1kdT8L3irY4goVANf7kkv8Pvnypzm0sLrbQiiW3javGplxXqCnnAsvMS1gSZ NAs92/Kw46cHjkeCe6/8a1yqIA4Rj51qCtlTX0aH+uxvTeuVUm9EP0VD2glJQKEkDg 5a0cfcP25m0gj45nBITGhc3Sn+R+QYrRr0dYmt08vFRBldJS1dKr/YqJCwSFa73ySf kIuo/G8jpo8fg== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:41 +0200 Subject: [PATCH RFC 01/12] mm/slab: skip kfence objects in allocation profiling Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-1-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 struct kfence_metadata only has obj_exts with CONFIG_MEMCG. If it's enabled, it does also work for allocation profiling, but there's little value recording tags for KFENCE objects. Furthermore it would complicate the upcoming changes, so just skip them in the slab hooks. Signed-off-by: Vlastimil Babka (SUSE) --- mm/slub.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 0337e60db5ac..a4be70d080fb 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2352,6 +2352,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s,= void *object, gfp_t flags, if (alloc_flags & SLAB_ALLOC_NO_RECURSE) return; =20 + if (is_kfence_address(object)) + return; + slab =3D virt_to_slab(object); obj_exts =3D prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, obje= ct); /* @@ -2399,6 +2402,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, = struct slab *slab, void **p for (i =3D 0; i < objects; i++) { unsigned int off =3D obj_to_index(s, slab, p[i]); =20 + if (is_kfence_address(p[i])) + continue; + alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size); } put_slab_obj_exts(obj_exts); --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0CAB43B6DE; Wed, 15 Jul 2026 10:10:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110255; cv=none; b=HA/W2gTZxtbC2n6nwHsdbpsVqrsUYSgAWs+sxoOGRUd2D3eQYB6lUwjAH2Jg563fmGRsm3MVx7o0aq7dihvgH+WlvdvGL2Ef+CBE9ZCzrsTKrmD0Ty/P2/C1kgMLCt0c3roI7Yr5EC1fSGRk5sdsvvZYT78BLaTIRKSaNLbW5Iw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110255; c=relaxed/simple; bh=uGyMHKNxgN1D5nK6Ig3/QLKOBnKctCHDvEjNMMoaiWg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ewZMdhPjUDkZuhJFA0YnD6VF68o/EywdMuAqlvDUUHF+/lou1hhTWCFq+UZGrR28zBC1H0SbCJc6zD7bGumAQMA10oLMwclwKyQIUMi8vRredO1J9Q/dVjFcm1u24232w/k2wzbd1+a6HUVyXNWI81UehXiUxaE8WETfuqxhy7U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=U+TsQ7mv; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="U+TsQ7mv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2BFAA1F00A3A; Wed, 15 Jul 2026 10:10:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110253; bh=mYcfDPC5eqMJoGRcXKwqOJ+dQiUJ7iaZsQUAOpeN25Q=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=U+TsQ7mvqJoL/byybbcUpGpAEnF1qYGMAQ0WC1Z8s8XBzlN8LmrJ/6WMh4UCAubEI 9lWnQh06X6f8DE/UvYiaTigf60+Gqt46cMUvwllTdqfPQI3x6XMES5KC4vdhIa1AHr qORtW3JtdPXh5/8jflldbh9HYALipwCMztuInIeJpopRq5krzh2jH9qY3mkuLVvXm2 r7QEE3v8ot6kjYBboyyCX6hiwOLXXGUZveX8c15ckpT4oBmrc7KASfe+PzXQ5Z7lIE p0EPcRWnwmvJDQ/ksrNtfNu+Vyy3/l8OB0TD1kWrQFObKadMeAzlxlWBy7ACMy06up 4tGu2/eQpzAvg== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:42 +0200 Subject: [PATCH RFC 02/12] mm/slab: remove objs_per_slab() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-2-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 The function has an unused kmem_cache argument and almost nothing uses it anyway; doing slab->objects is simpler. Remove it with the last two users. KUNIT_EXPECT_EQ() needs a cast to avoid "error: =E2=80=98typeof=E2= =80=99 applied to a bit-field" but we don't need to keep a wrapper just for that. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasayan --- mm/kfence/kfence_test.c | 2 +- mm/slab.h | 6 ------ mm/slub.c | 3 +-- 3 files changed, 2 insertions(+), 9 deletions(-) diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c index de2d0f7d62b1..9867c03ef0ae 100644 --- a/mm/kfence/kfence_test.c +++ b/mm/kfence/kfence_test.c @@ -295,7 +295,7 @@ static void *test_alloc(struct kunit *test, size_t size= , gfp_t gfp, enum allocat * memcg accounting works correctly. */ KUNIT_EXPECT_EQ(test, obj_to_index(s, slab, alloc), 0U); - KUNIT_EXPECT_EQ(test, objs_per_slab(s, slab), 1); + KUNIT_EXPECT_EQ(test, ((unsigned int)slab->objects), 1); =20 if (policy =3D=3D ALLOCATE_ANY) return alloc; diff --git a/mm/slab.h b/mm/slab.h index f5e336b6b6b0..01535e1e2d3c 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -330,12 +330,6 @@ static inline unsigned int obj_to_index(const struct k= mem_cache *cache, return __obj_to_index(cache, slab_address(slab), obj); } =20 -static inline int objs_per_slab(const struct kmem_cache *cache, - const struct slab *slab) -{ - return slab->objects; -} - /* * State of the slab allocator. * diff --git a/mm/slub.c b/mm/slub.c index a4be70d080fb..9e25f2dce7a6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2127,7 +2127,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, gfp_t gfp, unsigned int alloc_flags) { const bool allow_spin =3D alloc_flags_allow_spinning(alloc_flags); - unsigned int objects =3D objs_per_slab(s, slab); bool new_slab =3D alloc_flags & SLAB_ALLOC_NEW_SLAB; unsigned long new_exts; unsigned long old_exts; @@ -2183,7 +2182,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, #endif retry: old_exts =3D READ_ONCE(slab->obj_exts); - handle_failed_objexts_alloc(old_exts, vec, objects); + handle_failed_objexts_alloc(old_exts, vec, slab->objects); =20 if (new_slab) { /* --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19A8743D4F6; Wed, 15 Jul 2026 10:10:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110258; cv=none; b=YAFeDc2fKoa6u5HtX+FGNdS/S6VSPBSXhoWxdeurur/DNF7sCqI4uxX3lUhUVHiHdcpOuRH4FLLfsKgt84LHQORt2D3VFGTKAKUILkl9GWJ3APTWopM/9MaZx33QRtZXXKV0nPRgBySIQeaelxFyw4mC1WifJNsuh6rLKwAO9Iw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110258; c=relaxed/simple; bh=dZE2O9PBRnVTwF5p+AiswEnJG1rpDiooBHHek8pj880=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=JVzB4YMqqCuJpPwOblzQdv6R+8giLriAoUSvYH4vkAXD5qHXh2hwD44YXurzc9wcPKNKry9DkaPcg9zP17uKv1wcwcmlhuuEb/xTZmcdtcd0GZ17aMyw3O9BaL7Zmv9sD4uDdFa2qO6M9PAvanMExqOGkXfHMPZFumrpQ0JWee4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BO3ci8u0; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BO3ci8u0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4D0A01F00A3D; Wed, 15 Jul 2026 10:10:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110257; bh=Zd2tlZhAK+JyDltuF6YOH06qJxcu4D6T9r1FM+zMNoA=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=BO3ci8u0HQ6pL5ko8UnpUJPQ3h5oU4nToGn2Fj2huorQN/JOi/UVrQ2sDXVWVdvR7 kmNChn99zLOSFeZVJuRIptt6ohuJ5v5bu39zz+6yKPXLl6F2+edIbUiim1W/NvsQQU 3tFBlNCjHD4p9rKwU5TkukWDMf6BVokXw0xgxLm5Est9bfLs/yRcgZ+2IgaMAJacmT W/qlxjGt2CW7LFQx3GOtNAzIEHe9iVkS6AFfnIZr/w9ILm0MVTTsb3vj+hEGpM8g12 r8KmSJlxVUwZ4uXtH406TA86kUvtNguuARd0i6T26vW9x5XAJYkJrqbOg7WeS9wxxK lv/T+Crz0otHg== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:43 +0200 Subject: [PATCH RFC 03/12] mm: move struct slabobj_ext to mm/slab.h Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-3-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Users of include/linux/memcontrol.h don't need to see this internal structure. Further changes to the struct will reduce recompiling. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 13 ------------- mm/slab.h | 13 +++++++++++++ 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e1f46a0016fc..93869cc35c25 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1440,19 +1440,6 @@ static inline void mem_cgroup_flush_workqueue(void) = { } static inline int mem_cgroup_init(void) { return 0; } #endif /* CONFIG_MEMCG */ =20 -/* - * Extended information for slab objects stored as an array in page->memcg= _data - * if MEMCG_DATA_OBJEXTS is set. - */ -struct slabobj_ext { -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; -#endif -#ifdef CONFIG_MEM_ALLOC_PROFILING - union codetag_ref ref; -#endif -} __aligned(8); - static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) { struct mem_cgroup *memcg; diff --git a/mm/slab.h b/mm/slab.h index 01535e1e2d3c..7bd361447c54 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -549,6 +549,19 @@ static inline bool need_kmalloc_no_objext(void) return false; } =20 +/* + * Extended information for slab objects stored as an array in page->memcg= _data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { +#ifdef CONFIG_MEMCG + struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif +} __aligned(8); + #ifdef CONFIG_SLAB_OBJ_EXT =20 /* --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3D0A43B4B9; Wed, 15 Jul 2026 10:11:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110264; cv=none; b=oGjbRZ2r8Y29Ic93o+XNjw8K6lrCoerDeCc8Jh7gHCRColdnO0b7ZoAWwdM+U2LM+2Ax+3H1dyVHk/wvrwJBih2QZwdLfciNvVpymkoUAg1w3sURrEW322rx1RTSQLQ4L2qUQb/hzjCiQDRIo0jhanmdRcZdyTn8eZPzjO/oc54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110264; c=relaxed/simple; bh=9HhElATUP6juDMMZgc2GJzL6nhuRwWRGWslXQkd7VPs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=q/R5vSodamdWr0wnlpqrxz8glgVGiHopZUQzK4iP9PqMMLtWVMf64m8uEmDGcaIIAJAwEJxS/0hQtWvaTuq9ei97QkZ4FQJ2UcJ0mHVFH+WqXkg5MDQM505vlhT+40okYIzfDmm8LXV7nZKTyhAEkDFSpbIoINHqneYHLGcyutM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XLtRB46a; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XLtRB46a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E1201F000E9; Wed, 15 Jul 2026 10:10:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110260; bh=u6a6gIA6sTVgPNtUhqSfGZDUgWxEBBIZRB5pqLDi1cM=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=XLtRB46atn6QfmN5dAVo6auba4UvEwxMoK50FW7Eo4LIs9OaZCGxHNyui1Mo74TOJ Sgb2RsD62sH/87Y6MZCzKFGZVQblwVwa4K81SVZ4YKOEnbDPpOuAg0JzEb21oMj9Pk qDkFf8Txb5v2ihP3o4EmjRAUFwp6Nd/9/E7Hdk+iPaYYAvY/pMkQwSiA4oS2126PRy HAehdK2jap0CL8ZnUc3sYKKVkDxQPI2tKQBN6+kF10Z12of9emT8KkYpbV+sM/P3RF 7LCOLpV9LPs6x5qHMlnoak0fKDDamqrYl97RoNiOPxGPCfg7lu0jGV3d3dqGG9vcTt yLj7Ozc8pMWxg== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:44 +0200 Subject: [PATCH RFC 04/12] mm/slab: make slab_obj_ext() determine object index Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-4-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 All callers perform the same obj_to_index() calculation to pass the index. Simplify by passing object pointer instead and determining the index by slab_obj_ext(). Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- mm/memcontrol.c | 12 +++--------- mm/slab.h | 14 ++++++++------ mm/slub.c | 22 +++++++--------------- 3 files changed, 18 insertions(+), 30 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6dc4888a90f3..4e427286a88a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2865,15 +2865,13 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct = slab *slab, void *p) */ unsigned long obj_exts; struct slabobj_ext *obj_ext; - unsigned int off; =20 obj_exts =3D slab_obj_exts(slab); if (!obj_exts) return NULL; =20 get_slab_obj_exts(obj_exts); - off =3D obj_to_index(slab->slab_cache, slab, p); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(slab->slab_cache, slab, obj_exts, p); if (obj_ext->objcg) { struct obj_cgroup *objcg =3D obj_ext->objcg; =20 @@ -3541,7 +3539,6 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, size_t obj_size =3D obj_full_size(s); struct obj_cgroup *objcg; struct slab *slab; - unsigned long off; size_t i; =20 /* @@ -3616,8 +3613,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, =20 obj_exts =3D slab_obj_exts(slab); get_slab_obj_exts(obj_exts); - off =3D obj_to_index(s, slab, p[i]); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); obj_cgroup_get(objcg); obj_ext->objcg =3D objcg; put_slab_obj_exts(obj_exts); @@ -3635,10 +3631,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, st= ruct slab *slab, struct obj_cgroup *objcg; struct slabobj_ext *obj_ext; struct obj_stock_pcp *stock; - unsigned int off; =20 - off =3D obj_to_index(s, slab, p[i]); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); objcg =3D obj_ext->objcg; if (!objcg) continue; diff --git a/mm/slab.h b/mm/slab.h index 7bd361447c54..36d067d6e7c0 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -646,14 +646,16 @@ static inline unsigned int slab_get_stride(struct sla= b *slab) * Returns a pointer to the object extension associated with the object. * Must be called within a section covered by get/put_slab_obj_exts(). */ -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab, - unsigned long obj_exts, - unsigned int index) +static inline struct slabobj_ext * +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_ex= ts, + const void *obj) { struct slabobj_ext *obj_ext; + unsigned int index; =20 VM_WARN_ON_ONCE(obj_exts !=3D slab_obj_exts(slab)); =20 + index =3D obj_to_index(s, slab, obj); obj_ext =3D (struct slabobj_ext *)(obj_exts + slab_get_stride(slab) * index); return kasan_reset_tag(obj_ext); @@ -669,9 +671,9 @@ static inline unsigned long slab_obj_exts(struct slab *= slab) return 0; } =20 -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab, - unsigned long obj_exts, - unsigned int index) +static inline struct slabobj_ext * +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_ex= ts, + const void *obj) { return NULL; } diff --git a/mm/slub.c b/mm/slub.c index 9e25f2dce7a6..5e3f53bcd0d3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2070,11 +2070,10 @@ static inline void mark_obj_codetag_empty(const voi= d *obj) obj_slab =3D virt_to_slab(obj); slab_exts =3D slab_obj_exts(obj_slab); if (slab_exts) { + struct slabobj_ext *ext; + get_slab_obj_exts(slab_exts); - unsigned int offs =3D obj_to_index(obj_slab->slab_cache, - obj_slab, obj); - struct slabobj_ext *ext =3D slab_obj_ext(obj_slab, - slab_exts, offs); + ext =3D slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj); =20 if (unlikely(is_codetag_empty(&ext->ref))) { put_slab_obj_exts(slab_exts); @@ -2362,10 +2361,8 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s= , void *object, gfp_t flags, * check should be added before alloc_tag_add(). */ if (obj_exts) { - unsigned int obj_idx =3D obj_to_index(s, slab, object); - get_slab_obj_exts(obj_exts); - obj_ext =3D slab_obj_ext(slab, obj_exts, obj_idx); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, object); alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size); put_slab_obj_exts(obj_exts); } else { @@ -2386,7 +2383,6 @@ static noinline void __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, vo= id **p, int objects) { - int i; unsigned long obj_exts; =20 /* slab->obj_exts might not be NULL if it was created for MEMCG accountin= g. */ @@ -2398,13 +2394,11 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s= , struct slab *slab, void **p return; =20 get_slab_obj_exts(obj_exts); - for (i =3D 0; i < objects; i++) { - unsigned int off =3D obj_to_index(s, slab, p[i]); - + for (int i =3D 0; i < objects; i++) { if (is_kfence_address(p[i])) continue; =20 - alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size); + alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size); } put_slab_obj_exts(obj_exts); } @@ -2489,7 +2483,6 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) struct kmem_cache *s; struct page *page; struct slab *slab; - unsigned long off; =20 page =3D virt_to_page(p); if (PageLargeKmalloc(page)) { @@ -2529,8 +2522,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) obj_exts =3D slab_obj_exts(slab); if (obj_exts) { get_slab_obj_exts(obj_exts); - off =3D obj_to_index(s, slab, p); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, p); if (unlikely(obj_ext->objcg)) { put_slab_obj_exts(obj_exts); return true; --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AD2143B6CA; Wed, 15 Jul 2026 10:11:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110265; cv=none; b=hyemUFs4rTOd/S6hG5Ny85btyCsq427gb+N10hIlH11AklW+P2Ge3AvpJQWmDhUfLraqwEOIzRR2VmL7sqn1mtfewZBrKyXyJQN0TlKv+n06BI95U7oCdA8OX2zsmC5KpC3dJAhEgP07o/cXY3yJ//eGeY7EJkauz9xzzU9S/xY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110265; c=relaxed/simple; bh=WUP+u5LtCO9Lnm0bpz/QH3WvOGZg0zXtJV5J+ezU95M=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=fSe3Xb8ybB82ImKMApaOCqmbKfQJu8KDgXEqdCG8oXYH51kh4hj3kfjS2D6pX6D5iCvAhb90OBeCr3X0uiDG0Qa/4/R0frOdhFdTTKEGYyHG5iqChCViInFCT2t9ot+EAT4smm4cJMM/pBjMhEwxssOqSQOtcCfDTwYIgaaRBJ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CIAt3+pe; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CIAt3+pe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8EF521F00A3D; Wed, 15 Jul 2026 10:11:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110263; bh=GxN00IWro8eCPaP3MfYs22vlSvWNn2fiqNBuPgHAw1Y=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=CIAt3+peHEAjVLCFNdEtjzKI6Bn831EpZQol9Ve4nYeSqpWpoEXH+e1PZiE69sypP 1nJ13de+fohnPBl/BNpnTUPnAvLsaLRYx67o3i3crWhZ+5Yste9ZYlSMVqxJx3jp0t M5ulcTaGX3MFh4DZ4V9bU3v1znNXCpBkEmUcKozrFkKdKb7dSBF/pnOr2fqmADvnYS p1QlnbmjV61UxE8ygT37MjupfQDhVPwJQ1euv8w2itANBlrgDvzIyxNi/L66pVVI0A BoHx+F42FMcDhpYFTkq6U/aCa1NeoRS6Zh0wjgd8TE5xnatzwnEFUmG4r8e9jbvlFu 2ijvQhifFuvNw== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:45 +0200 Subject: [PATCH RFC 05/12] mm/slab: abstract slabobj_ext.objcg access Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-5-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 In preparation for changes to the structure, abstract access to the objcg field with a slab_obj_ext_objcgp() function. Rename the field to _objcg to make an unexpected direct access a compile error. No functional change intended. Signed-off-by: Vlastimil Babka (SUSE) --- mm/kfence/core.c | 2 +- mm/memcontrol.c | 23 +++++++++++++++-------- mm/slab.h | 9 ++++++++- mm/slub.c | 2 +- 4 files changed, 25 insertions(+), 11 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 6577bd76954e..717e8baf7e5d 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -1249,7 +1249,7 @@ void __kfence_free(void *addr) struct kfence_metadata *meta =3D addr_to_metadata((unsigned long)addr); =20 #ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->obj_exts.objcg); + KFENCE_WARN_ON(*slab_obj_ext_objcgp(&meta->obj_exts)); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4e427286a88a..6303a2b1a9d0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2865,6 +2865,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct sl= ab *slab, void *p) */ unsigned long obj_exts; struct slabobj_ext *obj_ext; + struct obj_cgroup *objcg; =20 obj_exts =3D slab_obj_exts(slab); if (!obj_exts) @@ -2872,9 +2873,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct sl= ab *slab, void *p) =20 get_slab_obj_exts(obj_exts); obj_ext =3D slab_obj_ext(slab->slab_cache, slab, obj_exts, p); - if (obj_ext->objcg) { - struct obj_cgroup *objcg =3D obj_ext->objcg; - + objcg =3D *slab_obj_ext_objcgp(obj_ext); + if (objcg) { put_slab_obj_exts(obj_exts); return obj_cgroup_memcg(objcg); } @@ -3577,6 +3577,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, unsigned long obj_exts; struct slabobj_ext *obj_ext; struct obj_stock_pcp *stock; + struct obj_cgroup **objcgp; =20 slab =3D virt_to_slab(p[i]); =20 @@ -3612,10 +3613,15 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache= *s, struct list_lru *lru, unlock_stock(stock); =20 obj_exts =3D slab_obj_exts(slab); + get_slab_obj_exts(obj_exts); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); + objcgp =3D slab_obj_ext_objcgp(obj_ext); + obj_cgroup_get(objcg); - obj_ext->objcg =3D objcg; + *objcgp =3D objcg; + put_slab_obj_exts(obj_exts); } =20 @@ -3628,16 +3634,17 @@ void __memcg_slab_free_hook(struct kmem_cache *s, s= truct slab *slab, size_t obj_size =3D obj_full_size(s); =20 for (int i =3D 0; i < objects; i++) { - struct obj_cgroup *objcg; + struct obj_cgroup **objcgp, *objcg; struct slabobj_ext *obj_ext; struct obj_stock_pcp *stock; =20 obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); - objcg =3D obj_ext->objcg; - if (!objcg) + objcgp =3D slab_obj_ext_objcgp(obj_ext); + if (!*objcgp) continue; =20 - obj_ext->objcg =3D NULL; + objcg =3D *objcgp; + *objcgp =3D NULL; =20 stock =3D trylock_stock(); __refill_obj_stock(objcg, stock, obj_size, true); diff --git a/mm/slab.h b/mm/slab.h index 36d067d6e7c0..789bd292075f 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void) */ struct slabobj_ext { #ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; + struct obj_cgroup *_objcg; #endif #ifdef CONFIG_MEM_ALLOC_PROFILING union codetag_ref ref; @@ -661,6 +661,13 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, = unsigned long obj_exts, return kasan_reset_tag(obj_ext); } =20 +#ifdef CONFIG_MEMCG +static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *= obj_ext) +{ + return &obj_ext->_objcg; +} +#endif + int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags); =20 diff --git a/mm/slub.c b/mm/slub.c index 5e3f53bcd0d3..48e10198a3ce 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2523,7 +2523,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) if (obj_exts) { get_slab_obj_exts(obj_exts); obj_ext =3D slab_obj_ext(s, slab, obj_exts, p); - if (unlikely(obj_ext->objcg)) { + if (unlikely(*slab_obj_ext_objcgp(obj_ext))) { put_slab_obj_exts(obj_exts); return true; } --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC2EC43DA5D; Wed, 15 Jul 2026 10:11:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110268; cv=none; b=SmVXp8g0ACZNgCr3owzI9HnNCXu64+FpOehJqxvZjDB9hK0iC2ryrktN/kij3EV/jfufd4r7w9YCBsB9QiaogvN7U9031mBVw7deiCpjYWJXGSja7MajCSEYFBqk8NT1mz1jlYIDYL3DXSVDoI+XmNFlR+jptx/roxD4T5cwcP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110268; c=relaxed/simple; bh=GfyLpES3wI8YbZIuBZalmdfcOlG7HdO2QJaXY7Zp+TI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=iRVljvoMgHgJdHgRUODIkFeyxcIRB/LMfX8CQ5WqItpDxfAkDyxW9t/qqPgHb1s6slSIXIc4khMDovtcmsKqp6OyApdQTAGlFjdqcr/kIBbXYTjY2Rrl26S2lLH8rMaruQ/Cj6N5hNt6S4fln6Bc9d+dMw6JKQhFj8QHcAMKYJo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=B708yIID; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="B708yIID" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AFF701F00A3A; Wed, 15 Jul 2026 10:11:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110266; bh=HzIaBZel8a8JeHhvZs2k8s/0l5364bN/PN4wTwimX2g=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=B708yIIDl8WT85J9RFbuGnwpgMU6yVOaj/crA1PR95SaKa6nytSokx5tzo+FKV7D6 XpT3SBumfJ9I1/tg/XmpFMZn+Z1V4IFMHCGLtcNePxD0z9V3jpeTpWbO0k5UTTDmqm v97YwPDi+4USWOZlpvWv+5MbygBhjQ9RGRB8CYzPVLt8mo026uLWxwvMu6pW8PG7rr zu7b9AGPwUGbSL3B4lmBAIaE7J/o2tn/0GgwIaufhE8fxpD8I0bqrvRTAlbp088T/u HeoI8Q7Lj2IQtxqJBrzTY63AFWMgJL3wiDwxxHa+BIRzP/68gy/zdnZGWj0MTSbR+w xBI2D/ccN0R5g== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:46 +0200 Subject: [PATCH RFC 06/12] mm/slab: abstract slabobj_ext.ref access Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-6-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 In preparation for changes to the structure, abstract access to the ref field with a slab_obj_ext_codetag_ref() function. Rename the field to _ctref to make an unexpected direct access a compile error. No functional change intended. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- mm/slab.h | 10 +++++++++- mm/slub.c | 42 ++++++++++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 15 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 789bd292075f..e3f8e42070f1 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -558,7 +558,7 @@ struct slabobj_ext { struct obj_cgroup *_objcg; #endif #ifdef CONFIG_MEM_ALLOC_PROFILING - union codetag_ref ref; + union codetag_ref _ctref; #endif } __aligned(8); =20 @@ -668,6 +668,14 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(= struct slabobj_ext *obj_ex } #endif =20 +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline union codetag_ref * +slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext) +{ + return &obj_ext->_ctref; +} +#endif + int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags); =20 diff --git a/mm/slub.c b/mm/slub.c index 48e10198a3ce..2bfcabc4c51a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2071,18 +2071,20 @@ static inline void mark_obj_codetag_empty(const voi= d *obj) slab_exts =3D slab_obj_exts(obj_slab); if (slab_exts) { struct slabobj_ext *ext; + union codetag_ref *ref; =20 get_slab_obj_exts(slab_exts); ext =3D slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj); + ref =3D slab_obj_ext_codetag_ref(obj_slab, ext); =20 - if (unlikely(is_codetag_empty(&ext->ref))) { + if (unlikely(is_codetag_empty(ref))) { put_slab_obj_exts(slab_exts); return; } =20 /* codetag should be NULL here */ - WARN_ON(ext->ref.ct); - set_codetag_empty(&ext->ref); + WARN_ON(ref->ct); + set_codetag_empty(ref); put_slab_obj_exts(slab_exts); } } @@ -2092,19 +2094,22 @@ static inline bool mark_failed_objexts_alloc(struct= slab *slab) return cmpxchg(&slab->obj_exts, 0, OBJEXTS_ALLOC_FAIL) =3D=3D 0; } =20 -static inline void handle_failed_objexts_alloc(unsigned long obj_exts, - struct slabobj_ext *vec, unsigned int objects) +static inline void handle_failed_objexts_alloc(struct slab *slab, + unsigned long obj_exts, struct slabobj_ext *vec) { /* * If vector previously failed to allocate then we have live * objects with no tag reference. Mark all references in this * vector as empty to avoid warnings later on. */ - if (obj_exts =3D=3D OBJEXTS_ALLOC_FAIL) { - unsigned int i; + if (obj_exts !=3D OBJEXTS_ALLOC_FAIL) + return; + + for (unsigned int i =3D 0; i < slab->objects; i++) { + union codetag_ref *ref =3D slab_obj_ext_codetag_ref(slab, vec); =20 - for (i =3D 0; i < objects; i++) - set_codetag_empty(&vec[i].ref); + set_codetag_empty(ref); + vec++; } } =20 @@ -2112,8 +2117,8 @@ static inline void handle_failed_objexts_alloc(unsign= ed long obj_exts, =20 static inline void mark_obj_codetag_empty(const void *obj) {} static inline bool mark_failed_objexts_alloc(struct slab *slab) { return f= alse; } -static inline void handle_failed_objexts_alloc(unsigned long obj_exts, - struct slabobj_ext *vec, unsigned int objects) {} +static inline void handle_failed_objexts_alloc(struct slab *slab, + unsigned long obj_exts, struct slabobj_ext *vec) {} =20 #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ =20 @@ -2181,7 +2186,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, #endif retry: old_exts =3D READ_ONCE(slab->obj_exts); - handle_failed_objexts_alloc(old_exts, vec, slab->objects); + handle_failed_objexts_alloc(slab, old_exts, vec); =20 if (new_slab) { /* @@ -2361,9 +2366,15 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s= , void *object, gfp_t flags, * check should be added before alloc_tag_add(). */ if (obj_exts) { + union codetag_ref *ref; + get_slab_obj_exts(obj_exts); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, object); - alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size); + ref =3D slab_obj_ext_codetag_ref(slab, obj_ext); + + alloc_tag_add(ref, current->alloc_tag, s->size); + put_slab_obj_exts(obj_exts); } else { alloc_tag_set_inaccurate(current->alloc_tag); @@ -2395,10 +2406,13 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s= , struct slab *slab, void **p =20 get_slab_obj_exts(obj_exts); for (int i =3D 0; i < objects; i++) { + struct slabobj_ext *ext; + if (is_kfence_address(p[i])) continue; =20 - alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size); + ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); + alloc_tag_sub(slab_obj_ext_codetag_ref(slab, ext), s->size); } put_slab_obj_exts(obj_exts); } --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5AA8433E7A; Wed, 15 Jul 2026 10:11:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110271; cv=none; b=En+ikWj3B+VUeYScwtqCeDy6sKzBYMeoQeJWDU2q2tF3hTsvbVbVJ8+h3RLkzM6ERvzhgBocC/VgWuoQUJUNqbmHn+55QM85TwRdcPadgDWhGpnD8GcRDyjwgY7qN6IfQbjpR/Sm2Z0A7ugrlwbURwAJ90OtM08pRw09czDRiag= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110271; c=relaxed/simple; bh=exr6yeChGKovDu3PsxGt445/8UY+SaFhSEkS2xjD3mA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=CyxsSBFo71vbLl4YeHvbKCywFMErIKPE1KcmWRVl45zv8FZGHwGqW7tDnBIKWqFgM8GH8EksdcruI84Um+v3sHR3YHRiL0/t04WO3YYl3F64e3zbkAj3QSSaRydAVEhK8BlD2K73bbcWC9vfhA5uZ+I4/avX3xKKS0l3ZZbxdLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=N7zy9xEx; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="N7zy9xEx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D0D501F000E9; Wed, 15 Jul 2026 10:11:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110269; bh=fqtJ5hqMpkrlyY02tT9s783SqiD4ZiPky1xiCi8taEI=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=N7zy9xExGIdDNEynI/MArerPHqN4oLJg1dowzWYxkuWiOTXgemebHAxnlJvKMq2zD 5e0XozLwkb57FUanjtri7zlxMmtL1cY6c+ax9KMc48AK/FaqLcU4Sg2oCP2dBu7EW1 ww5+owBx3CnkDhJvZ7mOBB7UZQmXQwvIJ1RF0XWh/Acum2GzpC5itqKLp93al83d3U 90v5NyecbSqaqQWTHXBzjXdHHx44iuV2p/Ehx/Mk58xFVH34Fvxw9TZFxfkCLIct1Z HqAFRxa93QkcSvKc0D1GNJCnpKmiIILOIO4KIpvCM6KsTu8ss6LNr88hcXpqKosB/2 WOaQo4s93L7pQ== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:47 +0200 Subject: [PATCH RFC 07/12] mm/slab: replace slab.stride with obj_exts_in_object Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-7-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 The stride field is used to convert object index to an slabobj_ext so both compact arrays (kmalloc() or in-slab-leftover) and spread in-object-padding obj_ext layouts are supported. In practice thus the stride is always sizeof(slabobj_ext) or s->size. This simplifies the calculations, but with the upcoming slabobj_ext handling changes, it will be easier to stop storing the stride and instead just have a flag whether obj_ext is in the object padding. obj_exts_in_object() can then rely on this flag and slab_obj_ext() can use that to determine the stride. No functional change intended. Performance impact TBD, hopefully in the noise. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- mm/slab.h | 43 ++++++++++++++++++++++++------------------- mm/slub.c | 42 ++++++++++++++++-------------------------- 2 files changed, 40 insertions(+), 45 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index e3f8e42070f1..3ad9777ad600 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -81,10 +81,10 @@ struct freelist_counters { #ifdef CONFIG_64BIT /* * Some optimizations use free bits in 'counters' field - * to save memory. In case ->stride field is not available, + * to save memory. If these free bits are not available, * such optimizations are disabled. */ - unsigned int stride; + unsigned obj_exts_in_object:1; #endif }; }; @@ -617,22 +617,20 @@ static inline void put_slab_obj_exts(unsigned long ob= j_exts) } =20 #ifdef CONFIG_64BIT -static inline void slab_set_stride(struct slab *slab, unsigned int stride) +static inline bool obj_exts_in_object(struct slab *slab) { - slab->stride =3D stride; -} -static inline unsigned int slab_get_stride(struct slab *slab) -{ - return slab->stride; + /* + * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to + * check the per-slab bit. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but + * allocations within_slab_leftover are preferred. And those may be + * possible or not depending on the particular slab's size. + */ + return slab->obj_exts_in_object; } #else -static inline void slab_set_stride(struct slab *slab, unsigned int stride) +static inline bool obj_exts_in_object(struct slab *slab) { - VM_WARN_ON_ONCE(stride !=3D sizeof(struct slabobj_ext)); -} -static inline unsigned int slab_get_stride(struct slab *slab) -{ - return sizeof(struct slabobj_ext); + return false; } #endif =20 @@ -656,8 +654,14 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, = unsigned long obj_exts, VM_WARN_ON_ONCE(obj_exts !=3D slab_obj_exts(slab)); =20 index =3D obj_to_index(s, slab, obj); - obj_ext =3D (struct slabobj_ext *)(obj_exts + - slab_get_stride(slab) * index); + + if (!obj_exts_in_object(slab)) { + obj_ext =3D ((struct slabobj_ext *)obj_exts) + index; + } else { + unsigned int stride =3D s->size; + obj_ext =3D (struct slabobj_ext *)(obj_exts + index * stride); + } + return kasan_reset_tag(obj_ext); } =20 @@ -693,9 +697,10 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, = unsigned long obj_exts, return NULL; } =20 -static inline void slab_set_stride(struct slab *slab, unsigned int stride)= { } -static inline unsigned int slab_get_stride(struct slab *slab) { return 0; } - +static inline bool obj_exts_in_object(struct slab *slab) +{ + return false; +} =20 #endif /* CONFIG_SLAB_OBJ_EXT */ =20 diff --git a/mm/slub.c b/mm/slub.c index 2bfcabc4c51a..98a14e5842a2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -870,18 +870,6 @@ static inline bool obj_exts_in_slab(struct kmem_cache = *s, struct slab *slab) #endif =20 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT) -static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab) -{ - /* - * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to - * check the stride. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but - * allocations within_slab_leftover are preferred. And those may be - * possible or not depending on the particular slab's size. - */ - return obj_exts_in_slab(s, slab) && - (slab_get_stride(slab) =3D=3D s->size); -} - static unsigned int obj_exts_offset_in_object(struct kmem_cache *s) { unsigned int offset =3D get_info_end(s); @@ -896,16 +884,20 @@ static unsigned int obj_exts_offset_in_object(struct = kmem_cache *s) =20 return offset; } -#else -static inline bool obj_exts_in_object(struct kmem_cache *s, struct slab *s= lab) + +static inline void slab_set_obj_exts_in_object(struct slab *slab) { - return false; + slab->obj_exts_in_object =3D 1; } - +#else static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s) { return 0; } + +static inline void slab_set_obj_exts_in_object(struct slab *slab) +{ +} #endif =20 #ifdef CONFIG_SLUB_DEBUG @@ -1206,7 +1198,7 @@ static void print_trailer(struct kmem_cache *s, struc= t slab *slab, u8 *p) =20 off +=3D kasan_metadata_size(s, false); =20 - if (obj_exts_in_object(s, slab)) + if (obj_exts_in_object(slab)) off +=3D sizeof(struct slabobj_ext); =20 if (off !=3D size_from_object(s)) @@ -1411,7 +1403,7 @@ static int check_pad_bytes(struct kmem_cache *s, stru= ct slab *slab, u8 *p) =20 off +=3D kasan_metadata_size(s, false); =20 - if (obj_exts_in_object(s, slab)) + if (obj_exts_in_object(slab)) off +=3D sizeof(struct slabobj_ext); =20 if (size_from_object(s) =3D=3D off) @@ -1439,7 +1431,7 @@ slab_pad_check(struct kmem_cache *s, struct slab *sla= b) length =3D slab_size(slab); end =3D start + length; =20 - if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(s, slab)) { + if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(slab)) { remainder =3D length; remainder -=3D obj_exts_offset_in_slab(s, slab); remainder -=3D obj_exts_size_in_slab(slab); @@ -2253,9 +2245,6 @@ static void alloc_slab_obj_exts_early(struct kmem_cac= he *s, struct slab *slab) void *addr; unsigned long obj_exts; =20 - /* Initialize stride early to avoid memory ordering issues */ - slab_set_stride(slab, sizeof(struct slabobj_ext)); - if (!need_slab_obj_exts(s)) return; =20 @@ -2289,7 +2278,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cac= he *s, struct slab *slab) obj_exts |=3D MEMCG_DATA_OBJEXTS; #endif slab->obj_exts =3D obj_exts; - slab_set_stride(slab, s->size); + slab_set_obj_exts_in_object(slab); } } =20 @@ -3402,9 +3391,10 @@ static struct slab *allocate_slab(struct kmem_cache = *s, gfp_t flags, stat(s, ORDER_FALLBACK); } =20 + /* Initializes frozen, inuse, and any extra 64bit-only flags */ + slab->counters =3D 0; + slab->objects =3D oo_objects(oo); - slab->inuse =3D 0; - slab->frozen =3D 0; =20 slab->slab_cache =3D s; =20 @@ -6537,7 +6527,7 @@ static inline size_t slab_ksize(struct slab *slab) */ if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER)) return s->inuse; - else if (obj_exts_in_object(s, slab)) + else if (obj_exts_in_object(slab)) return s->inuse; /* * Else we can use all the padding etc for the allocation --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0932B43E9CE; Wed, 15 Jul 2026 10:11:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110274; cv=none; b=Yn8lmMFa8E7t9IJLjh31dqbEg/hnms0+wDD/9EaWxD+ttvtjaN/f1rf6aHP0PfAu8qZPjNe0Y6LHcT5gE2Z9fxZ4HNUZikF0QpoPkjq3+dxYFUZH8mRJH/uzUIax/GcXl0kPX23dIJ4714FSqkpa27FBycim98iWSf9o9xDzkEw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110274; c=relaxed/simple; bh=COHePtv1R+YeIgiHqB4kOJjySGYRX9YbU+4yM3FzOng=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YusR3r1pN6JagOKPNbSJJw2kajgl0v2McCREwWnNyn3rM8Aatful/sq0skXBcjwBEBAKmZCiB5KYoH23CxtKzHL+GJ6Lrypt6gl2o2HaPKXq9pBwLouDP0SBev68kVxSPrEWRjBhRLHz/YAMZAldpOZujBOLUTbY03/Qg9RnFhE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aRAfAY+T; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aRAfAY+T" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F21381F00A3A; Wed, 15 Jul 2026 10:11:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110272; bh=rRhnRYBtmzeslijuXLx/hXq2VN7nb6FRSzXhyrvEmc8=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=aRAfAY+TPSpe/gjNK4eC6wi/35JisVhMsWhvXs35eNe8Gqlj4oJ9yhcxYClB08IHV 6UqS2mzSMjpA//zvA7KNTLP7vsQnxWPlhUP6l+Kn/mgNSwxHvBWuTT2bY+667GDS8W JUcGI4Ssf0YPs690sVfCljZG5j6XG/6KJh7zJwZe5kklyOojM2j5IAZoqpdQtS2H6V aSLXFZ0AXEa2Ots1eWDKBf0FJ6j0nhuFR2EKjBqv5pdKEki7NXNX8gkHy73AVCRz0d FtinLySDKRz009/8P1VbzxIgSyYFETJeuQVZ46a8mIBnNIn/1vP1qRNJjkP/R8CKIs +ZIVTn81cgfdQ== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:48 +0200 Subject: [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-8-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Currently, struct slabobj_ext can hold both objcg pointer and codetag_ref (when both are compile-enabled) and there is an array of as many slabobj_ext instances as there are objects in a slab. This makes the layout fixed so even if codetag_ref is unused (because memory allocation profiling is disabled), the space for them is allocated and wasted. Similarly, some caches (currently kmalloc_normal) do not ever need objcg pointers, leading to wasted memory with memory allocation profiling enabled. To make this more flexible, change the layout so that struct slabobj_ext becomes an union of objcg pointer and codetag_ref (to ensure uniform size; in practice both are the same size anyway). The slabobj_ext array then can have twice as many elements as before. For cache locality purposes, the effective memory layout is unchanged, so objcg and codetag ref for a given object are still adjacent. static_obj_ext_size() returns the effective size of (0-2) struct slabobj_ext's, depending on the config. slab_obj_ext_size() is currently static as well, but takes a slab pointer so it can be made dynamic later. Replace all sizeof(slabobj_ext) usage with these. No functional change intended, the layout is still effectively static. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- mm/slab.h | 41 +++++++++++++++++++++++++++++++++-------- mm/slub.c | 17 +++++++++-------- 2 files changed, 42 insertions(+), 16 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 3ad9777ad600..359ab8caf61e 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void) * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { + union { #ifdef CONFIG_MEMCG - struct obj_cgroup *_objcg; + struct obj_cgroup *_objcg; #endif #ifdef CONFIG_MEM_ALLOC_PROFILING - union codetag_ref _ctref; + union codetag_ref _ctref; #endif + }; } __aligned(8); =20 +static inline size_t static_obj_ext_size(void) +{ + size_t sz =3D 0; + + if (IS_ENABLED(CONFIG_MEMCG)) + sz +=3D 1; + + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + sz +=3D 1; + + return sizeof(struct slabobj_ext) * sz; +} + +static inline size_t slab_obj_ext_size(struct slab *slab) +{ + return static_obj_ext_size(); +} + #ifdef CONFIG_SLAB_OBJ_EXT =20 /* @@ -650,17 +670,18 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab,= unsigned long obj_exts, { struct slabobj_ext *obj_ext; unsigned int index; + unsigned int stride; =20 VM_WARN_ON_ONCE(obj_exts !=3D slab_obj_exts(slab)); =20 index =3D obj_to_index(s, slab, obj); =20 - if (!obj_exts_in_object(slab)) { - obj_ext =3D ((struct slabobj_ext *)obj_exts) + index; - } else { - unsigned int stride =3D s->size; - obj_ext =3D (struct slabobj_ext *)(obj_exts + index * stride); - } + if (!obj_exts_in_object(slab)) + stride =3D slab_obj_ext_size(slab); + else + stride =3D s->size; + + obj_ext =3D (struct slabobj_ext *)(obj_exts + index * stride); =20 return kasan_reset_tag(obj_ext); } @@ -668,6 +689,7 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, u= nsigned long obj_exts, #ifdef CONFIG_MEMCG static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext *= obj_ext) { + /* if objcg exists, it's first, so we don't need to do anything */ return &obj_ext->_objcg; } #endif @@ -676,6 +698,9 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(s= truct slabobj_ext *obj_ex static inline union codetag_ref * slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext) { + if (IS_ENABLED(CONFIG_MEMCG)) + obj_ext +=3D 1; + return &obj_ext->_ctref; } #endif diff --git a/mm/slub.c b/mm/slub.c index 98a14e5842a2..dd15af8abd62 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -803,7 +803,7 @@ static inline bool need_slab_obj_exts(struct kmem_cache= *s) =20 static inline unsigned int obj_exts_size_in_slab(struct slab *slab) { - return sizeof(struct slabobj_ext) * slab->objects; + return slab_obj_ext_size(slab) * slab->objects; } =20 static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s, @@ -1199,7 +1199,7 @@ static void print_trailer(struct kmem_cache *s, struc= t slab *slab, u8 *p) off +=3D kasan_metadata_size(s, false); =20 if (obj_exts_in_object(slab)) - off +=3D sizeof(struct slabobj_ext); + off +=3D slab_obj_ext_size(slab); =20 if (off !=3D size_from_object(s)) /* Beginning of the filler is the free pointer */ @@ -1404,7 +1404,7 @@ static int check_pad_bytes(struct kmem_cache *s, stru= ct slab *slab, u8 *p) off +=3D kasan_metadata_size(s, false); =20 if (obj_exts_in_object(slab)) - off +=3D sizeof(struct slabobj_ext); + off +=3D slab_obj_ext_size(slab); =20 if (size_from_object(s) =3D=3D off) return 1; @@ -2089,6 +2089,8 @@ static inline bool mark_failed_objexts_alloc(struct s= lab *slab) static inline void handle_failed_objexts_alloc(struct slab *slab, unsigned long obj_exts, struct slabobj_ext *vec) { + unsigned int stride =3D slab_obj_ext_size(slab) / sizeof(*vec); + /* * If vector previously failed to allocate then we have live * objects with no tag reference. Mark all references in this @@ -2101,7 +2103,7 @@ static inline void handle_failed_objexts_alloc(struct= slab *slab, union codetag_ref *ref =3D slab_obj_ext_codetag_ref(slab, vec); =20 set_codetag_empty(ref); - vec++; + vec +=3D stride; } } =20 @@ -2127,7 +2129,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, unsigned long new_exts; unsigned long old_exts; struct slabobj_ext *vec; - size_t sz =3D sizeof(struct slabobj_ext) * slab->objects; + size_t sz =3D slab_obj_ext_size(slab) * slab->objects; =20 gfp &=3D ~OBJCGS_CLEAR_MASK; /* @@ -2270,8 +2272,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cac= he *s, struct slab *slab) =20 get_slab_obj_exts(obj_exts); for_each_object(addr, s, slab_address(slab), slab->objects) - memset(kasan_reset_tag(addr) + offset, 0, - sizeof(struct slabobj_ext)); + memset(kasan_reset_tag(addr) + offset, 0, slab_obj_ext_size(slab)); put_slab_obj_exts(obj_exts); =20 #ifdef CONFIG_MEMCG @@ -7930,7 +7931,7 @@ static int calculate_sizes(struct kmem_cache_args *ar= gs, struct kmem_cache *s) aligned_size =3D ALIGN(size, s->align); #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT) if (slab_args_unmergeable(args, s->flags) && - (aligned_size - size >=3D sizeof(struct slabobj_ext))) + (aligned_size - size >=3D static_obj_ext_size())) s->flags |=3D SLAB_OBJ_EXT_IN_OBJ; #endif size =3D aligned_size; --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31EB243B6DE; Wed, 15 Jul 2026 10:11:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110277; cv=none; b=UX0kcrW7143bFA0XXE4KX2toySpFHzqZaN+NWVO/WIBSp+qv9oSK7ddPdu7hwZxa5iGCv5sIKJl+sFEQYd8bHsguPR51e5pKsudBgrdtLHl86wcpHLmzFxBBWXTaYrxmVZ5BaNW9U+mIqEE4aqKR2j8dmHsU9jlHuXFSXvscCec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110277; c=relaxed/simple; bh=SHdGz4rDHOERE3YmVNFGmxnZB4g49fyajAUjwjCyN0c=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=no2QftbJgjPdF2vocXf/66d8R3pd1mI9D6gURyHAeoRtageRD5BNSMDhTZDNkUlXAZjI/xkEAD6JwtAp88stkjQOq2ZqYt1CtlBumEi828yvw3tmcHIbZJjuuPx0VD42xZ3QA2vmL3F2qytsl850qE3wa7laEk8hBrBCuqt96oM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PUdIwloJ; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PUdIwloJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1F1EF1F000E9; Wed, 15 Jul 2026 10:11:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110275; bh=qejNc6/2sAIyKponA71R3jYamk33F9xwBlnyqE8jUxY=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=PUdIwloJ/5Zc31NRZkkCTFSmel7KjvBoFgWtriTOnSrrRfLys/6sgIRFf3ShnyBZ4 HCmzT2Xt0+qG6gMgV3UNz0W9rPZdL5AyBUZyrtIuN6RdQf6xyAjTExn0ObXscDqFMv xfwOViyvNmPxmsIkiF2sn7CEJnphg1GNpZTfleCJ3h7X5nJmOQYHwdwAusQisq+wib d+6Fb86TM14tkAP4NvFSOVhQl/4Z8XpwNOz4HqpKNuFXNwMu8KeLXuMK/YcYnjV4uX ZP0jZVD5/75bTxcVdUjFzBrTpRftxKWLFDf/EjBmsNfz6Tty5R1gDYc7ijVysCdxF+ Sl052/U2RSG+A== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:49 +0200 Subject: [PATCH RFC 09/12] mm/slab: introduce slab_obj_ext_has_codetag() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-9-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 mem_alloc_profiling_enabled() allows evaluating (with a static key) if memory profiling is currently enabled. mem_profiling_support is a variable where false means it's not possible to enable it anymore, because the system was booted with "never" or it was later shut down. This is possible to query by mem_alloc_profiling_permanently_disabled(). To make slabobj_ext array size handling dynamic, we need a snapshot of mem_alloc_profiling_permanently_disabled() early in boot, so that's not affected by a later shutdown. We also need it to be static key based for performance. Neither mem_alloc_profiling_enabled() nor mem_alloc_profiling_permanently_disabled() satisfy this. Therefore introduce slab_obj_ext_has_codetag() with an underlying static key for that use case. Its state is made to reflect the result of mem_alloc_profiling_permanently_disabled() during kmem_cache_init(), which does happen after setup_early_mem_profiling(). Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- mm/slab.h | 16 ++++++++++++++++ mm/slub.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/mm/slab.h b/mm/slab.h index 359ab8caf61e..dcca86799fc9 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -564,6 +564,22 @@ struct slabobj_ext { }; } __aligned(8); =20 +#ifdef CONFIG_MEM_ALLOC_PROFILING +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + slab_obj_ext_has_codetag_key); + +static inline bool slab_obj_ext_has_codetag(void) +{ + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + &slab_obj_ext_has_codetag_key); +} +#else +static inline bool slab_obj_ext_has_codetag(void) +{ + return false; +} +#endif + static inline size_t static_obj_ext_size(void) { size_t sz =3D 0; diff --git a/mm/slub.c b/mm/slub.c index dd15af8abd62..4200e7105b30 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -213,6 +213,11 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); static DEFINE_STATIC_KEY_FALSE(strict_numa); #endif =20 +#ifdef CONFIG_MEM_ALLOC_PROFILING +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + slab_obj_ext_has_codetag_key); +#endif + /* Structure holding extra parameters for slab allocations */ struct slab_alloc_context { unsigned long caller_addr; @@ -2415,6 +2420,26 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, s= truct slab *slab, void **p, __alloc_tagging_slab_free_hook(s, slab, p, objects); } =20 +/* + * Make sure the static key used by slab_obj_ext_has_codetag() reflects the + * value of !mem_alloc_profiling_permanently_disabled() + * + * Any later mem alloc profiling shutdown won't be reflected in the static= key + * because obj_exts with codetags might already exist. + */ +static void __init slab_obj_ext_has_codetag_init(void) +{ + bool key_enabled =3D IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEF= AULT); + bool need_codetag =3D !mem_alloc_profiling_permanently_disabled(); + + if (key_enabled !=3D need_codetag) { + if (need_codetag) + static_branch_enable(&slab_obj_ext_has_codetag_key); + else + static_branch_disable(&slab_obj_ext_has_codetag_key); + } +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ =20 static inline void @@ -2429,6 +2454,10 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, s= truct slab *slab, void **p, { } =20 +static inline void slab_obj_ext_has_codetag_init(void) +{ +} + #endif /* CONFIG_MEM_ALLOC_PROFILING */ =20 =20 @@ -8541,6 +8570,8 @@ void __init kmem_cache_init(void) boot_kmem_cache_node; int node; =20 + slab_obj_ext_has_codetag_init(); + if (debug_guardpage_minorder()) slub_max_order =3D 0; =20 --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BD8943DA56; Wed, 15 Jul 2026 10:11:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110280; cv=none; b=m3d5eqWJQ0b8pcd4lsRuoRnUymt2k+yNBYmJRVU4WFe6dK7+D9VFHoVn75KngIEMIAAnxE4FM3ENOTPUy9a8hGn56Yw1p1YMOppYV636pp2rY7NAraViWbqOM6v0VYbfY4AVGGfdYVWkBYKOftuCEOdIfozSINIHOrql+6AXdCM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110280; c=relaxed/simple; bh=Mxf3aQzzm1ZlIgi0DL+4j9W5p2xB6gAnyg59IfYqc3Y=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=reXHuoslmRmc6yIBE1pmHcV0WxEHJzxw3PWy6FlkSXye0B1BYUH4QAO1PC5FqzINRcj3kJ0U7D2hsotI6G5/yA+8fPjCl2k4mJEJKAA8jQKawoslByVxTAjc5V3x3yGW9g9IZH1DUVMdGK/Opqkl9ZwfA1vVqxYi/BHUvdyYyCo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jdrxhmen; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jdrxhmen" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40E2D1F00A3A; Wed, 15 Jul 2026 10:11:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110278; bh=B2kW+nJ77Mj5M53RnXhGqydD3hqQWm7pjmVwGgIrn1U=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=jdrxhmenWOsGlFcFe34zNIcdL7yScchP+BGEXwVOT4VENtyhZdFST8n9qyp6/L4Yp SWFYyCccxOp3weHr3MOPD+5gneEjf5ugtTQL+LgQOryTgr1mAnLlydiRjoqmwR4TYu +Bpzf045bXGHBWADEZiVvUV2JsqNsXcQlm0/YrEgD6O34cGoIO6UrbfWpQRH0giKWs CF3htpkQHU4rW8ss6v/x9sreGGx95uCcmruWcL01MlduUv/6CNnJRhuHkrKnSIVCa4 462mibkP9u12I9xjfmgn+Zf4E0JAtqA23X+lHwr3La55OrIyWgykAuBm0rq7nq2Hwl 5jFoqHW+fSK5w== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:50 +0200 Subject: [PATCH RFC 10/12] mm/slab: reduce slabobj_ext memory with allocation profiling disabled Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-10-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 When memory allocation profiling is compiled in but permanently disabled on boot with (implicit or explicit) "never" parameter, stop allocating (wasting) memory for the codetag_ref parts of slabobj_ext metadata. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- mm/slab.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slab.h b/mm/slab.h index dcca86799fc9..a50347c9dbe3 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -587,7 +587,7 @@ static inline size_t static_obj_ext_size(void) if (IS_ENABLED(CONFIG_MEMCG)) sz +=3D 1; =20 - if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + if (slab_obj_ext_has_codetag()) sz +=3D 1; =20 return sizeof(struct slabobj_ext) * sz; --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E05143FD14; Wed, 15 Jul 2026 10:11:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110283; cv=none; b=ga7zcFDqEvvkljPNB8Bt7FNtuwrKZ4SA+3q3jqWpk1Qba+Vz2s7UHHNMQV6Y8yzRkC01lQIWJcOuvJHmVIeS1Sk+wpDc8klw80HQ70JBxAKDMv1yP06JCCRYgr7kBz8U5yMg/gimX0Va0u1taflWN6Fi1+S/eoXRju9tL1TxSYs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110283; c=relaxed/simple; bh=yL9qj2aMDxVh01ChtbbElt5w9xb9vxWtupcsfHusP5w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=fndNd49wHV8IVCdyNKl7/XPXNWCrFGKb9G64D7rmrFqqdYKd5HzWicifdmrLzzDt5IEa8mManKAWJ0mYPMPgvdSfDT35Ja3IEfKEI2HIm0G1Lr2akJDY8+jYd+fj0xOkQx4qlmoiRgosPwmyJZNs7y4E5SvQDrrUQ++D6JRH9y0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Olo3/0Ii; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Olo3/0Ii" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 621711F000E9; Wed, 15 Jul 2026 10:11:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110282; bh=PE3KTV7cHysrre7SDOgoC57Bp7TB5EzPerhfduGTM0M=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=Olo3/0Iiz7E7xxUfliQ4rw0McZ1OC9LZxrxLmUFBSHXcdcW024RB9Y/+dShJ2mOmW ycqXeuBNBGXZuMdlW0pBkoYC/nWOQbSDL8zBd98HxqkPkQ1aWh6Uvmgs5MpYcdKOD4 q8ZIxTEO9hMiAlBFovcPtPQ81SIENvs8/3F2dWA9f2iXUVb4dlETypYdQdminnGU9/ yrS+X9HayZM2kB/168UM9IPFLCyp0YPUIaG93Lva575xJ2t4Di9IVSGcI5IxuLJL7U MXcNKENijEuPjYGbkZ75UfPJtoj6+UXzrXa1Rex9HmsT8jqyFeDNXlQ5v3UJVa+OqA 2EcYBfZSVsxsg== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:51 +0200 Subject: [PATCH RFC 11/12] mm/slab: add slab_needs_objcg() helper Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-11-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Slabs of some caches never need the objcg part of struct slabobj_ext. Introduce a helper to query this for a slab. Currently only is_kmalloc_normal() caches are considered as not needing objcg's. We could also consider all kmem caches without SLAB_ACCOUNT, however some might be used with and without __GFP_ACCOUNT concurrently and we currently don't restrict that. This can be improved later. To make the evaluation of slab_needs_objcg() faster in the allocation and free fast paths, add a obj_exts_needs_objcg flag into slab itself. This optimization is only available on 64bit architectures where free bits are available for the flag. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Suren Baghdasaryan --- mm/kfence/core.c | 3 +++ mm/slab.h | 31 +++++++++++++++++++++++++++++-- mm/slub.c | 4 ++++ 3 files changed, 36 insertions(+), 2 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 717e8baf7e5d..afeaf80484ad 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -640,6 +640,9 @@ static unsigned long kfence_init_pool(void) struct slab *slab =3D page_slab(page); slab->obj_exts =3D (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_e= xts | MEMCG_DATA_OBJEXTS; +#ifdef CONFIG_64BIT + slab->obj_exts_needs_objcg =3D 1; +#endif #endif } =20 diff --git a/mm/slab.h b/mm/slab.h index a50347c9dbe3..948d075cdbef 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -81,10 +81,11 @@ struct freelist_counters { #ifdef CONFIG_64BIT /* * Some optimizations use free bits in 'counters' field - * to save memory. If these free bits are not available, - * such optimizations are disabled. + * to save memory or CPU. If these free bits are not + * available, such optimizations are disabled. */ unsigned obj_exts_in_object:1; + unsigned obj_exts_needs_objcg:1; #endif }; }; @@ -580,6 +581,32 @@ static inline bool slab_obj_ext_has_codetag(void) } #endif =20 +#ifdef CONFIG_MEMCG +static inline bool cache_needs_objcg(struct kmem_cache *cache) +{ + return !is_kmalloc_normal(cache); +} + +static inline bool slab_needs_objcg(struct slab *slab) +{ +#ifdef CONFIG_64BIT + return slab->obj_exts_needs_objcg; +#else + return cache_needs_objcg(slab->slab_cache); +#endif +} +#else +static inline bool cache_needs_objcg(struct kmem_cache *cache) +{ + return false; +} + +static inline bool slab_needs_objcg(struct slab *slab) +{ + return false; +} +#endif + static inline size_t static_obj_ext_size(void) { size_t sz =3D 0; diff --git a/mm/slub.c b/mm/slub.c index 4200e7105b30..771d73abacb6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3426,6 +3426,10 @@ static struct slab *allocate_slab(struct kmem_cache = *s, gfp_t flags, =20 slab->objects =3D oo_objects(oo); =20 +#ifdef CONFIG_64BIT + if (cache_needs_objcg(s)) + slab->obj_exts_needs_objcg =3D 1; +#endif slab->slab_cache =3D s; =20 kasan_poison_slab(slab); --=20 2.55.0 From nobody Sat Jul 25 17:33:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9057243F4BE; Wed, 15 Jul 2026 10:11:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110286; cv=none; b=GU2xHX4TKJ8BoAiJazN1unYUubBffC6Ta5B4oI80izDUtekIDlwqaO3sfi1TB5pfzHa+HOeZyMbDNDmGPHt/4h+YSfPYod8CAaGMfhc0JoZ2frT683FMNqlSX3iPBK/BwxJqsoBoFzYmGYWPtTlpFZqoXGB1qi/YtEhafVOBvXc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784110286; c=relaxed/simple; bh=O0CIznq26Ds2kUKsMvzrT8Dx5jQK5mnuzi2ty7qEc0g=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=HW/ggma22lmkTOrelZPYw1j+J6MWy4hpqDc6QB0duZRkDedNEHHOBYHVNwYWPqqv5ima3xBY3r60wgnGXi8xK+ASNFaJT0thW++rX0EERbqK217BDJBE16lym8fRISY2vYZma5MmRU3w/Y2ESXuI2FejRYbmQuOMIYxB+MGsfR4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lGHDqt5V; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lGHDqt5V" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 831751F00A3A; Wed, 15 Jul 2026 10:11:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784110285; bh=sxiKNmbJD7zBEkDQUyHwImzWMYztIT9SravkSnuQT8g=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=lGHDqt5VwPjEwT8kk2s3Ys3pWyu6FyVPdbFgHVC0vlQvErAZH6PxAPutt7waAvKb1 sPYK5/rI9F0AP/MopFYh0hscAsBpuqgr9stRYMtsJLwEx5cZFa2CqqG2BtumyMr4RQ /hdyfFlwdEVyK6CqDt6Aw3di232txGDH7dTfQYIH4cBQr03QpcwYOFZFhSRChqb3QA oKJxtdn3g/R09HDdzGF+jWLF6W1mxKlW5KU5aG4hwofplZUCD62EZib4C+NScRzNck jrdi2vF5fS7y4GuXcuUiRVEi3CLpi8hW41rXZHlemscDJhmv5TF+l4h6MGNDHxZNbj BMiRwr9Ga9v0g== From: "Vlastimil Babka (SUSE)" Date: Wed, 15 Jul 2026 12:10:52 +0200 Subject: [PATCH RFC 12/12] mm/slab: stop allocating objcg pointers when unnecessary Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260715-b4-objext_split-v1-12-9a49c4ccf4c3@kernel.org> References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Start using the slab_needs_objcg() helper to calculate slabobj_ext size. Caches that we know to never need objcg pointers (currently KMALLOC_NORMAL caches) will thus stop wasting memory on them when memory allocation profiling is enabled. For things to work properly, we need to also add slab_needs_objcg() checks to mem_cgroup_from_obj_slab() and memcg_slab_free_hook(), because when obj_exts array exists for a slab only due to mem_alloc profiling, we would otherwise attempt to access a non-existing objcg pointer in that slab. The function __memcg_slab_post_alloc_hook() should not be possible to call for a slab where slab_needs_objcg() is false, but add a DEBUG_VM check there to prevent breaking this assumption accidentally. Signed-off-by: Vlastimil Babka (SUSE) --- mm/memcontrol.c | 6 ++++++ mm/slab.h | 12 ++++++++++-- mm/slub.c | 3 +++ 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6303a2b1a9d0..09659722ec85 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2871,6 +2871,9 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct sl= ab *slab, void *p) if (!obj_exts) return NULL; =20 + if (!slab_needs_objcg(slab)) + return NULL; + get_slab_obj_exts(obj_exts); obj_ext =3D slab_obj_ext(slab->slab_cache, slab, obj_exts, p); objcg =3D *slab_obj_ext_objcgp(obj_ext); @@ -3581,6 +3584,9 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, =20 slab =3D virt_to_slab(p[i]); =20 + if (IS_ENABLED(CONFIG_DEBUG_VM) && WARN_ON_ONCE(!slab_needs_objcg(slab))) + continue; + if (!slab_obj_exts(slab) && alloc_slab_obj_exts(slab, s, flags, slab_alloc_flags)) { continue; diff --git a/mm/slab.h b/mm/slab.h index 948d075cdbef..072cc2506756 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -622,7 +622,15 @@ static inline size_t static_obj_ext_size(void) =20 static inline size_t slab_obj_ext_size(struct slab *slab) { - return static_obj_ext_size(); + size_t sz =3D 0; + + if (slab_needs_objcg(slab)) + sz +=3D 1; + + if (slab_obj_ext_has_codetag()) + sz +=3D 1; + + return sizeof(struct slabobj_ext) * sz; } =20 #ifdef CONFIG_SLAB_OBJ_EXT @@ -741,7 +749,7 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp(s= truct slabobj_ext *obj_ex static inline union codetag_ref * slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext) { - if (IS_ENABLED(CONFIG_MEMCG)) + if (slab_needs_objcg(slab)) obj_ext +=3D 1; =20 return &obj_ext->_ctref; diff --git a/mm/slub.c b/mm/slub.c index 771d73abacb6..09c4931e5435 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2503,6 +2503,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struc= t slab *slab, void **p, if (likely(!obj_exts)) return; =20 + if (!slab_needs_objcg(slab)) + return; + get_slab_obj_exts(obj_exts); __memcg_slab_free_hook(s, slab, p, objects, obj_exts); put_slab_obj_exts(obj_exts); --=20 2.55.0