From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3866242A166; Mon, 20 Jul 2026 14:16:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784556993; cv=none; b=US3d4lxCL8zCeFGqLDsyDA4rr8w+ZONapIRvpcIJljGzrgmWfkRTRDSKNR4NczATd2Q4xAr62dXHXjG20qpfrz2xhERFHYuLqwjLSX0VpD/o+JqK/gF+kx4B0cj1VwtOJvlaO4/ahbHkn31rr9WZRi4mItgXBFWI7vBUeh9d+uU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784556993; c=relaxed/simple; bh=ucImc0OiTptNhcepKrASGUox+8GSyWO/1rGFJyynGdY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=cHRJmmbAztLm4wv9G3MUFJi5beTkYNaG7E8pmZ4YiaVTtveOiQEXbcimszaPJw3rxSMlX0I+fbnCQcnOhttz+i1BZr2Y2fe8Oj4p6gbFrJtl2vcugwbZpyMuyEssVR1hqUg75HjuUz2ek/YrtC5Hqbzix0Y97VN3o92fXQMrbig= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=M9l2/0LE; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="M9l2/0LE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40FBC1F00AC4; Mon, 20 Jul 2026 14:16:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784556991; bh=bOrfQ3lYA2720DIdcg2lgwZAAppgr0gM7i8Eaojwjq8=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=M9l2/0LEB2GWQY1CtKBsbLdSJT/yAB7qZ29jms3ke7nnXI9Iv69tyJTQCnwDEuF2I BrW0Kie0BuwlqXg1NPRo2fO2PmnXhLHzLW1YjNPp4qxsIgQZiYT7T8nuwr6GrFaxvb IYlx8TawcJvWfkOd+jwUZCT5WS6xKpXWeZlK0nqMu5Zn+FN9AgyK+ep+18QhB/gGrV fq62Fo2D0oYsmXHTwrbNXOPeQWKL6T7CbmLqc0UNhSnyB8LpAs+ir0nrunWZwNMdAU aSUS9frX+xSXA5CsOg2JOpYPRvk0WyqAhdcBM0oWXGkyeXC8FtkybA6hvS3RNONaUk 3V2kLT6dZW8jQ== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:15 +0200 Subject: [PATCH v2 01/13] mm/slab: skip kfence objects in allocation profiling Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-1-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 struct kfence_metadata only contains struct slabobj_ext with CONFIG_MEMCG, which is then used for the "fake" slab's obj_exts field. If CONFIG_MEMCG is enabled, the struct can also end up used for memory allocation profiling. If CONFIG_MEMCG is disabled but profiling is enabled, it will end up allocating its obj_exts via prepare_slab_obj_exts_hook() and assigning them to the fake struct slab. These will probably then never be freed. So things sorta work, but not always in the intended and optimal way. The upcoming changes to slabobj_ext layout would additionally need a proper refactoring to keep working. However, there's little benefit in accounting KFENCE objects. KFENCE allocations are rare and there can be only CONFIG_KFENCE_NUM_OBJECTS (default to 255) outstanding ones at any time. For any callsite prominent enough in the memory allocation profiling stats, allocations served from KFENCE will be lost in the noise. Thus let's not complicate things and simply stop accounting KFENCE objects in allocation profiling and skip them in the related slab hooks. We also need to skip kfence objects in mark_obj_codetag_empty() in case a sheaf is allocated from kfence, per earlier sashiko review. Signed-off-by: Vlastimil Babka (SUSE) --- Documentation/mm/allocation-profiling.rst | 7 +++++++ mm/slub.c | 11 +++++++++++ 2 files changed, 18 insertions(+) diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/a= llocation-profiling.rst index 5389d241176a..d02eb54ee8f2 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -112,3 +112,10 @@ break it out by rhashtable type. =20 - Then, use the following form for your allocations: alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...)) + +Notes +=3D=3D=3D=3D=3D + +- When a slab object is allocated from KFENCE, its accounting is skipped. + KFENCE allocations are rare and limited to a small number, so this omiss= ion + is negligible. diff --git a/mm/slub.c b/mm/slub.c index 0337e60db5ac..76acb78f2655 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2076,6 +2076,11 @@ static inline void mark_obj_codetag_empty(const void= *obj) struct slabobj_ext *ext =3D slab_obj_ext(obj_slab, slab_exts, offs); =20 + if (is_kfence_address(obj)) { + put_slab_obj_exts(slab_exts); + return; + } + if (unlikely(is_codetag_empty(&ext->ref))) { put_slab_obj_exts(slab_exts); return; @@ -2352,6 +2357,9 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s,= void *object, gfp_t flags, if (alloc_flags & SLAB_ALLOC_NO_RECURSE) return; =20 + if (is_kfence_address(object)) + return; + slab =3D virt_to_slab(object); obj_exts =3D prepare_slab_obj_exts_hook(s, slab, flags, alloc_flags, obje= ct); /* @@ -2399,6 +2407,9 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s, = struct slab *slab, void **p for (i =3D 0; i < objects; i++) { unsigned int off =3D obj_to_index(s, slab, p[i]); =20 + if (is_kfence_address(p[i])) + continue; + alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size); } put_slab_obj_exts(obj_exts); --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E8FC42B315; Mon, 20 Jul 2026 14:16:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784556996; cv=none; b=EQPpupTkV0d40V6KqiFdsCF7NBqVXGY/Ms6p7mmK2m34jJ6PIS7yiThQ+wVswGXRr1cx1nXtGATaTTdP/25NP+nCTV9AeNfd539aI6MHG/3toeks6eg8jcsZNUzCke8NU7/Xk9cwCyKtJpZjXetmi+rTIYDB9zyFWn1n5RJ0pxs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784556996; c=relaxed/simple; bh=363oFxB2g1Jj/iW1/4CC4M+twglrj+8VmFQSOEIh1eM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=tqpsUKVEGGYUS1VTux+Z1AhEOzeeosxRmMQpXv9IwbDyR7e0bEgq2BL1aKzDv9M7J18FubzFM9Jr5kOI2SpWvn2smJWUW9IT5hXTC+/zaroKNhb+wiTp1IcKYkk3IPBix5SfzfYddXXQalj8vgP7I3VVjtd/mXp9mXpCiEDqjrw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UVbvJRN9; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UVbvJRN9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 509CB1F000E9; Mon, 20 Jul 2026 14:16:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784556994; bh=UPQckxndCEJ7gON+eXBCmL7lkfNrS7vyJZGPPaJdfGA=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=UVbvJRN9/886lD9oa0A2+WveR2vmD3E/+I7suSHl/DV4hQsc6R14wqnSnmUXna/XP g4+9TLjdZkn0KgzM3fJk4FOFQYSzTCvLZ2g3HRl/XRsTPUNv26iNFsYwdAstYntqsW PtkYejuJ/aC86tPbd3vYKnUl1TjnWaBbjn2FP+6v1i6yOCYWzMcFzpRw20by+R60CT 6uQGdpGO72JeOTRBAPYtmXobECi96RNQPviEKAYUYAZr/hdjmT9NRWOxTR+18lz63+ y2vZJdk/9DPTUN3G/CsCRA8V/4/0YYSfsb50D1VeODZQuyOmaa7q88kSHZxXU4tBhC bggomyliIG0bA== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:16 +0200 Subject: [PATCH v2 02/13] mm/slub: skip handle_failed_objexts_alloc() with profiling disabled Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-2-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 The function might get called with memory allocation profiling disabled, when the obj_ext array is allocated for objcg pointers only. The handling is however unnecessary in that case, so skip it. This would otherwise become a real bug later, as pointed out by sashiko. For now it's just an optimization. Link: https://sashiko.dev/#/patchset/20260715-b4-objext_split-v1-0-9a49c4cc= f4c3@kernel.org?part=3D10 Signed-off-by: Vlastimil Babka (SUSE) --- mm/slub.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 76acb78f2655..95fa6fbad11a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2101,15 +2101,16 @@ static inline bool mark_failed_objexts_alloc(struct= slab *slab) static inline void handle_failed_objexts_alloc(unsigned long obj_exts, struct slabobj_ext *vec, unsigned int objects) { + if (!mem_alloc_profiling_enabled()) + return; + /* * If vector previously failed to allocate then we have live * objects with no tag reference. Mark all references in this * vector as empty to avoid warnings later on. */ if (obj_exts =3D=3D OBJEXTS_ALLOC_FAIL) { - unsigned int i; - - for (i =3D 0; i < objects; i++) + for (unsigned int i =3D 0; i < objects; i++) set_codetag_empty(&vec[i].ref); } } --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 249FC42BC41; Mon, 20 Jul 2026 14:16:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784556999; cv=none; b=Eww23q/KjjltUHMvioSVrrLLuFqjRWtGHwRoQDJ1g+PQba0tOvzH61lwcH/E5UR99Ox6vKKQI3E+fpoyLOPiU5fAC7+GBzf+LkIAYE5vFQCQ6tvj06F4LZaEyOF1AN1kDd/8fH6upaDMSddTEa7W+7yhHze+wAX6Di3S9xnxAE4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784556999; c=relaxed/simple; bh=7lLFWB0Z3ynMsd8fncbOQ0vGhyyoswKScm3rzh7pxL4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ZLD3D/wcuiXAW6MiFoBypAh2NYXmWI5oCi7txAlg1DSGJN3sDHBUzcdjmqVH/UmV5YRCpOWWvK2LMiqm7PM2Qr0uS0G3ZGvM+brXSZfQKDgwQbQPM4Yqnck7AsBZi44YJcfayWQ0JxHbz1Vj1sTjRJo8jXaugiz//SXvOZiouNk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BvaFlNqC; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BvaFlNqC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 603041F00A3A; Mon, 20 Jul 2026 14:16:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784556998; bh=S3cjoXT+ctO1xn/lt8vvk/EJQ8UQyUkXc5d7Z/bIJ8Y=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=BvaFlNqCT3zRcpj5gpCsMysl7f1Z5x6KlIqglS3wL8dE7fDb4vfljJG69QgDI5aMG irsjdWDIFR4nZ5vEOphUnEvvRpnatSwcD3L+tSFQxnLwyibGDXScYKQP9xw505CO+z H0hbacgYBv9pURgI5jYIqD0qCXg5IiLDVdbPZgl3HguYxKGLm94s21tsc+ci5YagvQ 4J/mFiiJEZsj4PhXu94EfmM980cNBnzjoVjbDPS4Mt3jztWUM66oaQoOTqrHgVe9/e SiDqHSdH5SqzgWyksDDDO61RgEpDbwIU+Bi9S9P8fZ0Yt/lbMKgsA3tCmHspiFJhGx MDLFUHVSnvi9g== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:17 +0200 Subject: [PATCH v2 03/13] mm/slab: remove objs_per_slab() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-3-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 The function has an unused kmem_cache argument and almost nothing uses it anyway; doing slab->objects is simpler. Remove it with the last two users. KUNIT_EXPECT_EQ() needs a cast to avoid "error: =E2=80=98typeof=E2= =80=99 applied to a bit-field" but we don't need to keep a wrapper just for that. Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li Reviewed-by: Harry Yoo (Oracle) --- mm/kfence/kfence_test.c | 2 +- mm/slab.h | 6 ------ mm/slub.c | 3 +-- 3 files changed, 2 insertions(+), 9 deletions(-) diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c index de2d0f7d62b1..9867c03ef0ae 100644 --- a/mm/kfence/kfence_test.c +++ b/mm/kfence/kfence_test.c @@ -295,7 +295,7 @@ static void *test_alloc(struct kunit *test, size_t size= , gfp_t gfp, enum allocat * memcg accounting works correctly. */ KUNIT_EXPECT_EQ(test, obj_to_index(s, slab, alloc), 0U); - KUNIT_EXPECT_EQ(test, objs_per_slab(s, slab), 1); + KUNIT_EXPECT_EQ(test, ((unsigned int)slab->objects), 1); =20 if (policy =3D=3D ALLOCATE_ANY) return alloc; diff --git a/mm/slab.h b/mm/slab.h index f5e336b6b6b0..01535e1e2d3c 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -330,12 +330,6 @@ static inline unsigned int obj_to_index(const struct k= mem_cache *cache, return __obj_to_index(cache, slab_address(slab), obj); } =20 -static inline int objs_per_slab(const struct kmem_cache *cache, - const struct slab *slab) -{ - return slab->objects; -} - /* * State of the slab allocator. * diff --git a/mm/slub.c b/mm/slub.c index 95fa6fbad11a..8c1031989e41 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2133,7 +2133,6 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, gfp_t gfp, unsigned int alloc_flags) { const bool allow_spin =3D alloc_flags_allow_spinning(alloc_flags); - unsigned int objects =3D objs_per_slab(s, slab); bool new_slab =3D alloc_flags & SLAB_ALLOC_NEW_SLAB; unsigned long new_exts; unsigned long old_exts; @@ -2189,7 +2188,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, #endif retry: old_exts =3D READ_ONCE(slab->obj_exts); - handle_failed_objexts_alloc(old_exts, vec, objects); + handle_failed_objexts_alloc(old_exts, vec, slab->objects); =20 if (new_slab) { /* --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78345429817; Mon, 20 Jul 2026 14:16:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557002; cv=none; b=F1m0p4nlQs2e8CeJeVvdAxbMl2/y2zd+aYTZfT5ENWMCxffRNqNq0kctG5Q2sr9CeQjPKNcpNFcTmAyqtda78zUJEL7NFhRVAlMl46LsRfzxsPOZZmLZ6SXDI5VS/gRp2xGsB57W4PotLuaEVgwS2LTIlm8wRLyu+1wXfI+ya9g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557002; c=relaxed/simple; bh=uHuYlHIOrdo1xX2LkYws7+vgVsQOzkWSaSyhmIiiH7w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Ka1gza8C/27ITXFzrSrY/VPMSjXIGQT6Cl2tio3z7Oa9JMqdYPboUXjpuv8pRJbykmCmicosVJ4/j+t5UwIdz9F8Orlb+rY+FRcs7OYa7zBtrcMrmX6aogVZGsqNc2NaYiGTczIHXQQu5uAZ4Os26XHgrPo3bAygEos9wlobsTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Br9XN6gJ; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Br9XN6gJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6F65F1F00A3D; Mon, 20 Jul 2026 14:16:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557001; bh=hv2b9QbG8twJiZU//bTGb/DtRCdO7wG1c5Bfg2/CoNc=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=Br9XN6gJBq/DAksxVIjEvSqV/kxRrhJt2BSaai2pXAV2SyQ7+cQXYzdab7C2/MCPJ JKtpQdpnXaCdJCszSsNOW0vBJPwlcHu7/iWNjSKih5PMsSBIkpYTBA1NG9GZZnl3GI blHRKJHQGHn+Mvd5XEVY/UOjfiH9pk86iC+dxMGqDwfkBOgumDn61cPSf5pcP0ST/4 dec2G+CiZ0/pzr+0gjXsDyA0evWXgEecu7thWpvgReWZ/CqiuQtGBaQ/6lCaym1GfC ST2Ly5/crNwdUWH5LzWME1ybTIm6HUJ1/RSjLBMOo76MObVqqqvhbjQ+pvk3TatY4r LXCtocsX3g0/g== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:18 +0200 Subject: [PATCH v2 04/13] mm: move struct slabobj_ext to mm/slab.h Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-4-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Users of include/linux/memcontrol.h don't need to see this internal structure. Further changes to the struct will reduce recompiling. Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li Reviewed-by: Harry Yoo (Oracle) --- include/linux/memcontrol.h | 13 ------------- mm/slab.h | 13 +++++++++++++ 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e1f46a0016fc..93869cc35c25 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1440,19 +1440,6 @@ static inline void mem_cgroup_flush_workqueue(void) = { } static inline int mem_cgroup_init(void) { return 0; } #endif /* CONFIG_MEMCG */ =20 -/* - * Extended information for slab objects stored as an array in page->memcg= _data - * if MEMCG_DATA_OBJEXTS is set. - */ -struct slabobj_ext { -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; -#endif -#ifdef CONFIG_MEM_ALLOC_PROFILING - union codetag_ref ref; -#endif -} __aligned(8); - static inline struct lruvec *parent_lruvec(struct lruvec *lruvec) { struct mem_cgroup *memcg; diff --git a/mm/slab.h b/mm/slab.h index 01535e1e2d3c..7bd361447c54 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -549,6 +549,19 @@ static inline bool need_kmalloc_no_objext(void) return false; } =20 +/* + * Extended information for slab objects stored as an array in page->memcg= _data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { +#ifdef CONFIG_MEMCG + struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif +} __aligned(8); + #ifdef CONFIG_SLAB_OBJ_EXT =20 /* --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F1D842CAF9; Mon, 20 Jul 2026 14:16:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557005; cv=none; b=fzeoqmK1rkFIWGsVmyzBsY5T1iBJWzMZeYZ9llP0bXaFAHpSii8Xdinyb3wFDLy0HkhURCGLaFGUR7ZaWQdhgheG9nE8RdRGacrlifgyZhJsJbdf3M+5GtpbIiD5kP6c41NPrkG1RXFZL1fRksZi4DdzLiYuEGgl2ohvrnYs1T8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557005; c=relaxed/simple; bh=2E5NLURLqd5/RjNE+G0jpULeH5CsLkDEc9PUTQ9Vpvk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=tFfdDB/uDE2+gvkPIe9y+vBmbd5mUUTT1g7esmeixJY9jj8xeeAqrgwofhFgj1IRD2bXVTqXsL8cfPHw/f6jDFNg5ze5iR7derU8RxcYH5rWbvG9OeH/WctM2Pp7nPZ/uqAnf42btLeEB2ITLB85Fs6wTqDJ+RlTRlwVfj4BRG4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mKBvlIPO; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mKBvlIPO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E6201F000E9; Mon, 20 Jul 2026 14:16:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557004; bh=YDoguJb49gTUoNmifS0Dpu7IJ68YU71Jp33Sr8F+9ZU=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=mKBvlIPOQY+qIh+hV6wOGqLgdCnncDf+1ZqyMqL7VLG29XazRyXy9O2L3v0/1Ikxe AmixbS8Xm5vjAA+YYYmFXhEs543gug+5xNAJ5jqvomohb2KBwc6+QcwrSbflnDyX1r R4W7Rjttqwh7FzussL6Ki+DLHguSUHLc004b1xhy0xrKAofamYhxWprBFUjR6fXVwu EbwQxwdmOfqlZKfaXE16iGhggLQ9xq4NQUuENVqTWHf+GUbi+a+XByueXonX8mN5Vi L2shYE+Xu9oRlL/g27C7MibLRb1d+9I1HL/dC3FPMchISHTJ9j28ANhlrZ4DDDJXt4 XS6cR7SCY+Mow== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:19 +0200 Subject: [PATCH v2 05/13] mm/slab: make slab_obj_ext() determine object index Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-5-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 All callers perform the same obj_to_index() calculation to pass the index. Simplify by passing object pointer instead and determining the index by slab_obj_ext(). Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li Reviewed-by: Harry Yoo (Oracle) --- mm/memcontrol.c | 12 +++--------- mm/slab.h | 19 +++++++++++-------- mm/slub.c | 22 +++++++--------------- 3 files changed, 21 insertions(+), 32 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6dc4888a90f3..4e427286a88a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2865,15 +2865,13 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct = slab *slab, void *p) */ unsigned long obj_exts; struct slabobj_ext *obj_ext; - unsigned int off; =20 obj_exts =3D slab_obj_exts(slab); if (!obj_exts) return NULL; =20 get_slab_obj_exts(obj_exts); - off =3D obj_to_index(slab->slab_cache, slab, p); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(slab->slab_cache, slab, obj_exts, p); if (obj_ext->objcg) { struct obj_cgroup *objcg =3D obj_ext->objcg; =20 @@ -3541,7 +3539,6 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, size_t obj_size =3D obj_full_size(s); struct obj_cgroup *objcg; struct slab *slab; - unsigned long off; size_t i; =20 /* @@ -3616,8 +3613,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, =20 obj_exts =3D slab_obj_exts(slab); get_slab_obj_exts(obj_exts); - off =3D obj_to_index(s, slab, p[i]); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); obj_cgroup_get(objcg); obj_ext->objcg =3D objcg; put_slab_obj_exts(obj_exts); @@ -3635,10 +3631,8 @@ void __memcg_slab_free_hook(struct kmem_cache *s, st= ruct slab *slab, struct obj_cgroup *objcg; struct slabobj_ext *obj_ext; struct obj_stock_pcp *stock; - unsigned int off; =20 - off =3D obj_to_index(s, slab, p[i]); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); objcg =3D obj_ext->objcg; if (!objcg) continue; diff --git a/mm/slab.h b/mm/slab.h index 7bd361447c54..64cec02b5016 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -579,7 +579,7 @@ struct slabobj_ext { * obj_exts =3D slab_obj_exts(slab); * if (obj_exts) { * get_slab_obj_exts(obj_exts); - * obj_ext =3D slab_obj_ext(slab, obj_exts, obj_to_index(s, slab, = obj)); + * obj_ext =3D slab_obj_ext(s, slab, obj_exts, obj); * // do something with obj_ext * put_slab_obj_exts(obj_exts); * } @@ -639,21 +639,24 @@ static inline unsigned int slab_get_stride(struct sla= b *slab) /* * slab_obj_ext - get the pointer to the slab object extension metadata * associated with an object in a slab. + * @s: cache that the slab blongs to * @slab: a pointer to the slab struct * @obj_exts: a pointer to the object extension vector - * @index: an index of the object + * @obj: a pointer to the object * * Returns a pointer to the object extension associated with the object. * Must be called within a section covered by get/put_slab_obj_exts(). */ -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab, - unsigned long obj_exts, - unsigned int index) +static inline struct slabobj_ext * +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_ex= ts, + const void *obj) { struct slabobj_ext *obj_ext; + unsigned int index; =20 VM_WARN_ON_ONCE(obj_exts !=3D slab_obj_exts(slab)); =20 + index =3D obj_to_index(s, slab, obj); obj_ext =3D (struct slabobj_ext *)(obj_exts + slab_get_stride(slab) * index); return kasan_reset_tag(obj_ext); @@ -669,9 +672,9 @@ static inline unsigned long slab_obj_exts(struct slab *= slab) return 0; } =20 -static inline struct slabobj_ext *slab_obj_ext(struct slab *slab, - unsigned long obj_exts, - unsigned int index) +static inline struct slabobj_ext * +slab_obj_ext(struct kmem_cache *s, struct slab *slab, unsigned long obj_ex= ts, + const void *obj) { return NULL; } diff --git a/mm/slub.c b/mm/slub.c index 8c1031989e41..aa99d7eb6a4d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2070,11 +2070,10 @@ static inline void mark_obj_codetag_empty(const voi= d *obj) obj_slab =3D virt_to_slab(obj); slab_exts =3D slab_obj_exts(obj_slab); if (slab_exts) { + struct slabobj_ext *ext; + get_slab_obj_exts(slab_exts); - unsigned int offs =3D obj_to_index(obj_slab->slab_cache, - obj_slab, obj); - struct slabobj_ext *ext =3D slab_obj_ext(obj_slab, - slab_exts, offs); + ext =3D slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj); =20 if (is_kfence_address(obj)) { put_slab_obj_exts(slab_exts); @@ -2368,10 +2367,8 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s= , void *object, gfp_t flags, * check should be added before alloc_tag_add(). */ if (obj_exts) { - unsigned int obj_idx =3D obj_to_index(s, slab, object); - get_slab_obj_exts(obj_exts); - obj_ext =3D slab_obj_ext(slab, obj_exts, obj_idx); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, object); alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size); put_slab_obj_exts(obj_exts); } else { @@ -2392,7 +2389,6 @@ static noinline void __alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, vo= id **p, int objects) { - int i; unsigned long obj_exts; =20 /* slab->obj_exts might not be NULL if it was created for MEMCG accountin= g. */ @@ -2404,13 +2400,11 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s= , struct slab *slab, void **p return; =20 get_slab_obj_exts(obj_exts); - for (i =3D 0; i < objects; i++) { - unsigned int off =3D obj_to_index(s, slab, p[i]); - + for (int i =3D 0; i < objects; i++) { if (is_kfence_address(p[i])) continue; =20 - alloc_tag_sub(&slab_obj_ext(slab, obj_exts, off)->ref, s->size); + alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size); } put_slab_obj_exts(obj_exts); } @@ -2495,7 +2489,6 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) struct kmem_cache *s; struct page *page; struct slab *slab; - unsigned long off; =20 page =3D virt_to_page(p); if (PageLargeKmalloc(page)) { @@ -2535,8 +2528,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) obj_exts =3D slab_obj_exts(slab); if (obj_exts) { get_slab_obj_exts(obj_exts); - off =3D obj_to_index(s, slab, p); - obj_ext =3D slab_obj_ext(slab, obj_exts, off); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, p); if (unlikely(obj_ext->objcg)) { put_slab_obj_exts(obj_exts); return true; --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74B4842A17B; Mon, 20 Jul 2026 14:16:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557008; cv=none; b=pfrskSaKJP4BMhWN9j4u73Uu+xMNYcmRQhmh/qOVcIRRPnwPqrJH9OzQM3KTuHvOcDjAaEyeDoDZjAuQgYtaSnUoZ0iHBTsZrq+5MMqzpeV9aI32x8AK+htubPc9QXflGQg4Ca9NIkhvFaZqNBRLX+U76MK0vkIAo2VdfrTZADU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557008; c=relaxed/simple; bh=mcQgvQ3H084hVOKFHe4mvrHFvcTIirhGDX/vYCPlAwk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Z9nrDfzXovZhytBqsEvIzEq9OmruWqahVowH7G6pJsqWlEr8Kl8PNJO+fE+wv0mPc943GCX9uF4Ca3tHylpG+XnsuMzeCVTNr3FRP/zasSSlNSGxzcQ0J44mKo/BpW+uxsIw83YAHrFPv29s6eppldGqJYjDGoKjhuqYujeHRgU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=O6IlCXMd; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="O6IlCXMd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D66F1F00A3A; Mon, 20 Jul 2026 14:16:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557007; bh=D3U54YmCRY/GBvsj3bANv9/dOilD8BR4I8e1Q/zY4xQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=O6IlCXMdef44TD7zJDcvT/iuRPSZckHKHRWd2siui/LxQAjoo18KmQMbMT4tgXl1Q 1X5kY0W9M7JYIxRhNTGXKCEnC5UX2HkNMjlBvC6arD+edzA/ECMxxPXTLp8X49UX1X 9rG1xY2R069IV3CexOp8B1LPpZtquCZu49DQdp5FI72jor3t6olgYxL3klKFc+7Ivn sJgOjcCx8hnR7a4ZBe+GPek4JDGE2hScWhpZKI6dU+iwzwoyU0aCcaxJf2q5uYwF62 cJ4jGLvHqKi94GnJMWMn19jujSokl6DzGH4mCi+hjXVb9Wyxydr1tXjstWaMdzyhJt BbwufI9yG6TWA== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:20 +0200 Subject: [PATCH v2 06/13] mm/slab: abstract slabobj_ext.objcg access Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-6-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 In preparation for changes to the structure, abstract getting and setting the objcg field with slab_obj_ext_objcg() and slab_obj_ext_set_objcg(). Rename the field to _objcg to make an unexpected direct access a compile error. No functional change intended. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li Reviewed-by: Suren Baghdasaryan --- mm/kfence/core.c | 2 +- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 15 ++++++++++++++- mm/slub.c | 2 +- 4 files changed, 24 insertions(+), 9 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 6577bd76954e..05b5482c8150 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -1249,7 +1249,7 @@ void __kfence_free(void *addr) struct kfence_metadata *meta =3D addr_to_metadata((unsigned long)addr); =20 #ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->obj_exts.objcg); + KFENCE_WARN_ON(slab_obj_ext_objcg(&meta->obj_exts)); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4e427286a88a..cb1e97b4edc1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2865,6 +2865,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct sl= ab *slab, void *p) */ unsigned long obj_exts; struct slabobj_ext *obj_ext; + struct obj_cgroup *objcg; =20 obj_exts =3D slab_obj_exts(slab); if (!obj_exts) @@ -2872,9 +2873,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct sl= ab *slab, void *p) =20 get_slab_obj_exts(obj_exts); obj_ext =3D slab_obj_ext(slab->slab_cache, slab, obj_exts, p); - if (obj_ext->objcg) { - struct obj_cgroup *objcg =3D obj_ext->objcg; - + objcg =3D slab_obj_ext_objcg(obj_ext); + if (objcg) { put_slab_obj_exts(obj_exts); return obj_cgroup_memcg(objcg); } @@ -3614,8 +3614,10 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache = *s, struct list_lru *lru, obj_exts =3D slab_obj_exts(slab); get_slab_obj_exts(obj_exts); obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); + obj_cgroup_get(objcg); - obj_ext->objcg =3D objcg; + slab_obj_ext_set_objcg(obj_ext, objcg); + put_slab_obj_exts(obj_exts); } =20 @@ -3633,11 +3635,11 @@ void __memcg_slab_free_hook(struct kmem_cache *s, s= truct slab *slab, struct obj_stock_pcp *stock; =20 obj_ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); - objcg =3D obj_ext->objcg; + objcg =3D slab_obj_ext_objcg(obj_ext); if (!objcg) continue; =20 - obj_ext->objcg =3D NULL; + slab_obj_ext_set_objcg(obj_ext, NULL); =20 stock =3D trylock_stock(); __refill_obj_stock(objcg, stock, obj_size, true); diff --git a/mm/slab.h b/mm/slab.h index 64cec02b5016..1d1771f97aca 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -555,7 +555,7 @@ static inline bool need_kmalloc_no_objext(void) */ struct slabobj_ext { #ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; + struct obj_cgroup *_objcg; #endif #ifdef CONFIG_MEM_ALLOC_PROFILING union codetag_ref ref; @@ -662,6 +662,19 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, = unsigned long obj_exts, return kasan_reset_tag(obj_ext); } =20 +#ifdef CONFIG_MEMCG +static inline struct obj_cgroup *slab_obj_ext_objcg(struct slabobj_ext *ob= j_ext) +{ + return obj_ext->_objcg; +} + +static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext, + struct obj_cgroup *objcg) +{ + obj_ext->_objcg =3D objcg; +} +#endif + int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags); =20 diff --git a/mm/slub.c b/mm/slub.c index aa99d7eb6a4d..f2ffadde879d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2529,7 +2529,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) if (obj_exts) { get_slab_obj_exts(obj_exts); obj_ext =3D slab_obj_ext(s, slab, obj_exts, p); - if (unlikely(obj_ext->objcg)) { + if (unlikely(slab_obj_ext_objcg(obj_ext))) { put_slab_obj_exts(obj_exts); return true; } --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D69F42CB13; Mon, 20 Jul 2026 14:16:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557011; cv=none; b=Qu9r1F7A0sPCioOjbYcmV+ks/KWbKYzYAHaW+DdPGzf7Nz2eR/4057rnAjI4KBwbt4sAinPjXCC98/SORq/fUg1b8kElX7o2DWk0TexZO0RHT2+LP/o/8ZVPFzkkYlkk5eGprUFFBTeW3mmdlG1ITviOleDXVCz2/AzO8nJ+Whk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557011; c=relaxed/simple; bh=qRhbsPVpLn1S0jVnMzHPeZNIgXEVXf6DA1Hwaw/x/SE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=glPOY0X2KpCAGF7C3BHx+/ytsD4MV1D0J9/K9IMf+B+U8dRBZKdL1iZxR37CSEFI6vvDnsqIngkq/BQwhStBRgJeTggP6Rf6rHjnde2Cb1gLfxjwm9f0rwNR9UW5mIwr3Ww7Ve3PuZl+AVM22FWF5MXffShHQu36GbgvmHffFLA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FeexIb05; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FeexIb05" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9CC431F000E9; Mon, 20 Jul 2026 14:16:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557010; bh=6r95VsQz7/bTPUaUrteE7bqXPatxg+Tzf2bzkDOfpus=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=FeexIb05eHtmuOUiBfDtAWxk3PrPlHYamFu45sXhoPfPvLfZHK3B+P/3oLwfibj3F 2KM/AArbZ55tU1uwIesTVdWN2BsJV9KT2SqwUU2I3pTlWgAoC3yfkkTt3FnVETIfH6 8ETfJb47yFBxAdkpg3KmKOmKFV8jjB0KLX/geS+LghDHRmuPHWUcEJp8ntR8H8Xu5+ RxQ6aJZhttrV2b8/IcKp0pHMgDmq+GK/aKseQVharGFoYKdP7ntubaYo0UGSh8QE8d 6+M7ZI7lwH/Aq3mupNhevs/wzMenUWn5Xdbs3woHw5ZgEcr4G77yvJbbClkazgqFWL +R692rxxO6N4Q== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:21 +0200 Subject: [PATCH v2 07/13] mm/slab: abstract slabobj_ext.ref access Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-7-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 In preparation for changes to the structure, abstract access to the ref field with a slab_obj_ext_codetag_ref() function. Rename the field to _ctref to make an unexpected direct access a compile error. No functional change intended. Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li --- mm/slab.h | 10 +++++++++- mm/slub.c | 42 +++++++++++++++++++++++++++++------------- 2 files changed, 38 insertions(+), 14 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 1d1771f97aca..62d6366d7d79 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -558,7 +558,7 @@ struct slabobj_ext { struct obj_cgroup *_objcg; #endif #ifdef CONFIG_MEM_ALLOC_PROFILING - union codetag_ref ref; + union codetag_ref _ctref; #endif } __aligned(8); =20 @@ -675,6 +675,14 @@ static inline void slab_obj_ext_set_objcg(struct slabo= bj_ext *obj_ext, } #endif =20 +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline union codetag_ref * +slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext) +{ + return &obj_ext->_ctref; +} +#endif + int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, unsigned int alloc_flags); =20 diff --git a/mm/slub.c b/mm/slub.c index f2ffadde879d..cf999cbe75d1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2071,23 +2071,25 @@ static inline void mark_obj_codetag_empty(const voi= d *obj) slab_exts =3D slab_obj_exts(obj_slab); if (slab_exts) { struct slabobj_ext *ext; + union codetag_ref *ref; =20 get_slab_obj_exts(slab_exts); ext =3D slab_obj_ext(obj_slab->slab_cache, obj_slab, slab_exts, obj); + ref =3D slab_obj_ext_codetag_ref(obj_slab, ext); =20 if (is_kfence_address(obj)) { put_slab_obj_exts(slab_exts); return; } =20 - if (unlikely(is_codetag_empty(&ext->ref))) { + if (unlikely(is_codetag_empty(ref))) { put_slab_obj_exts(slab_exts); return; } =20 /* codetag should be NULL here */ - WARN_ON(ext->ref.ct); - set_codetag_empty(&ext->ref); + WARN_ON(ref->ct); + set_codetag_empty(ref); put_slab_obj_exts(slab_exts); } } @@ -2097,8 +2099,8 @@ static inline bool mark_failed_objexts_alloc(struct s= lab *slab) return cmpxchg(&slab->obj_exts, 0, OBJEXTS_ALLOC_FAIL) =3D=3D 0; } =20 -static inline void handle_failed_objexts_alloc(unsigned long obj_exts, - struct slabobj_ext *vec, unsigned int objects) +static inline void handle_failed_objexts_alloc(struct slab *slab, + unsigned long obj_exts, struct slabobj_ext *vec) { if (!mem_alloc_profiling_enabled()) return; @@ -2108,9 +2110,14 @@ static inline void handle_failed_objexts_alloc(unsig= ned long obj_exts, * objects with no tag reference. Mark all references in this * vector as empty to avoid warnings later on. */ - if (obj_exts =3D=3D OBJEXTS_ALLOC_FAIL) { - for (unsigned int i =3D 0; i < objects; i++) - set_codetag_empty(&vec[i].ref); + if (obj_exts !=3D OBJEXTS_ALLOC_FAIL) + return; + + for (unsigned int i =3D 0; i < slab->objects; i++) { + union codetag_ref *ref =3D slab_obj_ext_codetag_ref(slab, vec); + + set_codetag_empty(ref); + vec++; } } =20 @@ -2118,8 +2125,8 @@ static inline void handle_failed_objexts_alloc(unsign= ed long obj_exts, =20 static inline void mark_obj_codetag_empty(const void *obj) {} static inline bool mark_failed_objexts_alloc(struct slab *slab) { return f= alse; } -static inline void handle_failed_objexts_alloc(unsigned long obj_exts, - struct slabobj_ext *vec, unsigned int objects) {} +static inline void handle_failed_objexts_alloc(struct slab *slab, + unsigned long obj_exts, struct slabobj_ext *vec) {} =20 #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ =20 @@ -2187,7 +2194,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, #endif retry: old_exts =3D READ_ONCE(slab->obj_exts); - handle_failed_objexts_alloc(old_exts, vec, slab->objects); + handle_failed_objexts_alloc(slab, old_exts, vec); =20 if (new_slab) { /* @@ -2367,9 +2374,15 @@ __alloc_tagging_slab_alloc_hook(struct kmem_cache *s= , void *object, gfp_t flags, * check should be added before alloc_tag_add(). */ if (obj_exts) { + union codetag_ref *ref; + get_slab_obj_exts(obj_exts); + obj_ext =3D slab_obj_ext(s, slab, obj_exts, object); - alloc_tag_add(&obj_ext->ref, current->alloc_tag, s->size); + ref =3D slab_obj_ext_codetag_ref(slab, obj_ext); + + alloc_tag_add(ref, current->alloc_tag, s->size); + put_slab_obj_exts(obj_exts); } else { alloc_tag_set_inaccurate(current->alloc_tag); @@ -2401,10 +2414,13 @@ __alloc_tagging_slab_free_hook(struct kmem_cache *s= , struct slab *slab, void **p =20 get_slab_obj_exts(obj_exts); for (int i =3D 0; i < objects; i++) { + struct slabobj_ext *ext; + if (is_kfence_address(p[i])) continue; =20 - alloc_tag_sub(&slab_obj_ext(s, slab, obj_exts, p[i])->ref, s->size); + ext =3D slab_obj_ext(s, slab, obj_exts, p[i]); + alloc_tag_sub(slab_obj_ext_codetag_ref(slab, ext), s->size); } put_slab_obj_exts(obj_exts); } --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7222142E000; Mon, 20 Jul 2026 14:16:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557014; cv=none; b=Qh6obx2i3i8k1xKeCxsNwRV06K7oPp4sSQjORl38vD2HemQluWPNkiJFkfGwmCX1noBFJbLoOyLPoKMJivexHDMib70rigCSTav8NrCkmEOeH5sT/d7HOfuanYOFL5AKcXNtj+sCXfelZTdvfubFl5hegVBYS700I6GqQcb8j34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557014; c=relaxed/simple; bh=IC8TBEVEEa/sYw89rJKxlnP8ISZPRsKSX/WVJLgMsMs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=D+lTttE1EwkSo+Kiwhxyf7UysViolP4em5pEt28cFkqwrWrRkZN1k2plpQXXuX2pDqTUFU4KgyhjH4lEKWw5KHU9ZGHHO95AqkT84mjUPXrfC3kYZp6c8r58Dyg0FV6mLd4zI2toohpDm8Hsmm42U9AcPrTWwIaBzP12L74A2zM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ag50PqQ6; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ag50PqQ6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC3781F00A3A; Mon, 20 Jul 2026 14:16:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557013; bh=JVOSCtSkIl9pvKhPgb17Vi/B/0DgmPSgipmixRu4V7k=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=Ag50PqQ6TBFMx9wWb1CwjdwNFiyxlVEDAPuNyVJCmp8SwKzPz5D9IqsjkzT+S9uGD Hlzi0EUj39Xvc5J7KPXJBla6yj3isEcuaTB7t/1DeIqCayEkPnHzJrmJ4h4f4sk75e RRFsr8pvtcasjhBMcLv16O03pV5WL0NSGBK1x0Gx8I7Kzdhs1RRs0Icy+RKfVRa4+w Fcju+cDSZ4tIiRVTsAF5qwVLku11Ti1SuPJ2ZFLDz/uOhrgFgUux8lWGUmsSq5DwfF fKWy6RFbyFJOIqqoAGGXwECI65DtUOeO7h/jOOAFXHnYG3FUxT+lsHzvvV0zDyKF7l BRN6jR9J5stbQ== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:22 +0200 Subject: [PATCH v2 08/13] mm/slab: replace slab.stride with obj_exts_in_object Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-8-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 The stride field is used to convert object index to an slabobj_ext so both compact arrays (kmalloc() or in-slab-leftover) and spread in-object-padding obj_ext layouts are supported. In practice thus the stride is always sizeof(slabobj_ext) or s->size. This simplifies the calculations, but with the upcoming slabobj_ext handling changes, it will be easier to stop storing the stride and instead just have a flag whether obj_ext is in the object padding. obj_exts_in_object() can then rely on this flag and slab_obj_ext() can use that to determine the stride. No functional change intended. Performance impact TBD, hopefully in the noise. Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li --- mm/slab.h | 44 +++++++++++++++++++++++++------------------- mm/slub.c | 42 ++++++++++++++++-------------------------- 2 files changed, 41 insertions(+), 45 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 62d6366d7d79..e586798e4f16 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -81,10 +81,10 @@ struct freelist_counters { #ifdef CONFIG_64BIT /* * Some optimizations use free bits in 'counters' field - * to save memory. In case ->stride field is not available, + * to save memory. If these free bits are not available, * such optimizations are disabled. */ - unsigned int stride; + unsigned obj_exts_in_object:1; #endif }; }; @@ -617,22 +617,20 @@ static inline void put_slab_obj_exts(unsigned long ob= j_exts) } =20 #ifdef CONFIG_64BIT -static inline void slab_set_stride(struct slab *slab, unsigned int stride) +static inline bool obj_exts_in_object(struct slab *slab) { - slab->stride =3D stride; -} -static inline unsigned int slab_get_stride(struct slab *slab) -{ - return slab->stride; + /* + * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to + * check the per-slab bit. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but + * allocations within_slab_leftover are preferred. And those may be + * possible or not depending on the particular slab's size. + */ + return slab->obj_exts_in_object; } #else -static inline void slab_set_stride(struct slab *slab, unsigned int stride) +static inline bool obj_exts_in_object(struct slab *slab) { - VM_WARN_ON_ONCE(stride !=3D sizeof(struct slabobj_ext)); -} -static inline unsigned int slab_get_stride(struct slab *slab) -{ - return sizeof(struct slabobj_ext); + return false; } #endif =20 @@ -657,8 +655,15 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, = unsigned long obj_exts, VM_WARN_ON_ONCE(obj_exts !=3D slab_obj_exts(slab)); =20 index =3D obj_to_index(s, slab, obj); - obj_ext =3D (struct slabobj_ext *)(obj_exts + - slab_get_stride(slab) * index); + + if (!obj_exts_in_object(slab)) { + obj_ext =3D ((struct slabobj_ext *)obj_exts) + index; + } else { + unsigned int stride =3D s->size; + + obj_ext =3D (struct slabobj_ext *)(obj_exts + index * stride); + } + return kasan_reset_tag(obj_ext); } =20 @@ -700,9 +705,10 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab, = unsigned long obj_exts, return NULL; } =20 -static inline void slab_set_stride(struct slab *slab, unsigned int stride)= { } -static inline unsigned int slab_get_stride(struct slab *slab) { return 0; } - +static inline bool obj_exts_in_object(struct slab *slab) +{ + return false; +} =20 #endif /* CONFIG_SLAB_OBJ_EXT */ =20 diff --git a/mm/slub.c b/mm/slub.c index cf999cbe75d1..f5b3eb44cb8a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -870,18 +870,6 @@ static inline bool obj_exts_in_slab(struct kmem_cache = *s, struct slab *slab) #endif =20 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT) -static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab) -{ - /* - * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to - * check the stride. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but - * allocations within_slab_leftover are preferred. And those may be - * possible or not depending on the particular slab's size. - */ - return obj_exts_in_slab(s, slab) && - (slab_get_stride(slab) =3D=3D s->size); -} - static unsigned int obj_exts_offset_in_object(struct kmem_cache *s) { unsigned int offset =3D get_info_end(s); @@ -896,16 +884,20 @@ static unsigned int obj_exts_offset_in_object(struct = kmem_cache *s) =20 return offset; } -#else -static inline bool obj_exts_in_object(struct kmem_cache *s, struct slab *s= lab) + +static inline void slab_set_obj_exts_in_object(struct slab *slab) { - return false; + slab->obj_exts_in_object =3D 1; } - +#else static inline unsigned int obj_exts_offset_in_object(struct kmem_cache *s) { return 0; } + +static inline void slab_set_obj_exts_in_object(struct slab *slab) +{ +} #endif =20 #ifdef CONFIG_SLUB_DEBUG @@ -1206,7 +1198,7 @@ static void print_trailer(struct kmem_cache *s, struc= t slab *slab, u8 *p) =20 off +=3D kasan_metadata_size(s, false); =20 - if (obj_exts_in_object(s, slab)) + if (obj_exts_in_object(slab)) off +=3D sizeof(struct slabobj_ext); =20 if (off !=3D size_from_object(s)) @@ -1411,7 +1403,7 @@ static int check_pad_bytes(struct kmem_cache *s, stru= ct slab *slab, u8 *p) =20 off +=3D kasan_metadata_size(s, false); =20 - if (obj_exts_in_object(s, slab)) + if (obj_exts_in_object(slab)) off +=3D sizeof(struct slabobj_ext); =20 if (size_from_object(s) =3D=3D off) @@ -1439,7 +1431,7 @@ slab_pad_check(struct kmem_cache *s, struct slab *sla= b) length =3D slab_size(slab); end =3D start + length; =20 - if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(s, slab)) { + if (obj_exts_in_slab(s, slab) && !obj_exts_in_object(slab)) { remainder =3D length; remainder -=3D obj_exts_offset_in_slab(s, slab); remainder -=3D obj_exts_size_in_slab(slab); @@ -2261,9 +2253,6 @@ static void alloc_slab_obj_exts_early(struct kmem_cac= he *s, struct slab *slab) void *addr; unsigned long obj_exts; =20 - /* Initialize stride early to avoid memory ordering issues */ - slab_set_stride(slab, sizeof(struct slabobj_ext)); - if (!need_slab_obj_exts(s)) return; =20 @@ -2297,7 +2286,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cac= he *s, struct slab *slab) obj_exts |=3D MEMCG_DATA_OBJEXTS; #endif slab->obj_exts =3D obj_exts; - slab_set_stride(slab, s->size); + slab_set_obj_exts_in_object(slab); } } =20 @@ -3410,9 +3399,10 @@ static struct slab *allocate_slab(struct kmem_cache = *s, gfp_t flags, stat(s, ORDER_FALLBACK); } =20 + /* Initializes frozen, inuse, and any extra 64bit-only flags */ + slab->counters =3D 0; + slab->objects =3D oo_objects(oo); - slab->inuse =3D 0; - slab->frozen =3D 0; =20 slab->slab_cache =3D s; =20 @@ -6545,7 +6535,7 @@ static inline size_t slab_ksize(struct slab *slab) */ if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER)) return s->inuse; - else if (obj_exts_in_object(s, slab)) + else if (obj_exts_in_object(slab)) return s->inuse; /* * Else we can use all the padding etc for the allocation --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9A3A42E8DF; Mon, 20 Jul 2026 14:16:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557018; cv=none; b=CAPOJGXEXJR5K6egRfwVkRhO2Mt7HXD3q+lEpKa9h96QUaqD/BByvS1PZ5+Q9ONxqBIjJli/axVqvtuXqzfDFFJqejOCKJ29EuSdZPg1ClBVMPtLZdKqqb7T7ms8gPJSCK80MMTgJ489nJm2sfulG/dvmavk1k+zc1pyrVyBQ+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557018; c=relaxed/simple; bh=JSD3i3wxXmRaw3kP4Fra0mQWky7FxbV5dW+q+FeGfGU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=G86f/9s2uyrZQCdYWZdgeH8tQWSYMiX/IPX2xx8Q0abKTmwBW07XinvpOXy1A7dF2gfcweQNG82sFQlR9tACv1+QKIQFWx0RZ15i0rH+cb6PZI1Viz3CeK0Tgmqg3UANABxQcybUjEQM2cdW4cxYcKmoUsYmLfa5ojHghFGQk3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=a+9vjh6s; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="a+9vjh6s" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC1B01F000E9; Mon, 20 Jul 2026 14:16:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557016; bh=bjoSP6iddXq5PqiuJFnR36EEih/xwOGHSb79nndRhpk=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=a+9vjh6s3qu0BOkJaj3suWW2z1csNZFV0BtPZnhUHEIoGJBS5KSiGcwnk1Iks8gjB m2TT14Rmb2TkuX4tC/0hR2o6oKdj+ECei+hB7/v0JUFfK1a1Bjif5cRzPKzpArgJnB F/Eu9SyTDosipKO0glkdRQVWnKQVr2ph8jvPAHP2snGsgqAljhAESwJZm3oLK5zMjg 1lkq4VUhebyhhtNHmqRVKC50zawjTgscX15PrKG877KXQdPKJeIGLdfM+lr4FvoQZI 7hmvIQUmidozfjtyzthbhwlvNw/QHs4E8wumY8mGcwpUuEKUmDCUDc8iBPPFocVtJA uVTPgzFqJtSaQ== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:23 +0200 Subject: [PATCH v2 09/13] mm/slab: change struct slabobj_ext to a union Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-9-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Currently, struct slabobj_ext can hold both objcg pointer and codetag_ref (when both are compile-enabled) and there is an array of as many slabobj_ext instances as there are objects in a slab. This makes the layout fixed so even if codetag_ref is unused (because memory allocation profiling is disabled), the space for them is allocated and wasted. Similarly, some caches (currently kmalloc_normal) do not ever need objcg pointers, leading to wasted memory with memory allocation profiling enabled. To make this more flexible, change the layout so that struct slabobj_ext becomes a union of objcg pointer and codetag_ref (to ensure uniform size; in practice both are the same size anyway). The slabobj_ext array then can have twice as many elements as before. For cache locality purposes, the effective memory layout is unchanged, so objcg and codetag ref for a given object are still adjacent. cache_obj_ext_size() returns the effective size of (0-2) struct slabobj_ext's for a cache, slab_obj_ext_size() for a slab. Currently both return a constant value derived from the config options, but will be made dynamic later. Replace all sizeof(slabobj_ext) usage with these. No functional change intended, the layout is still effectively static. Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) --- mm/slab.h | 49 +++++++++++++++++++++++++++++++++++++++---------- mm/slub.c | 19 +++++++++++-------- 2 files changed, 50 insertions(+), 18 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index e586798e4f16..f8446167e175 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -550,18 +550,42 @@ static inline bool need_kmalloc_no_objext(void) } =20 /* - * Extended information for slab objects stored as an array in page->memcg= _data - * if MEMCG_DATA_OBJEXTS is set. + * Extended information for slab objects stored as a pointer to an array in + * slab->obj_exts (aliasing page->memcg_data) if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { + /* + * All elements of the union should be pointer-sized to avoid memory + * waste + */ + union { #ifdef CONFIG_MEMCG - struct obj_cgroup *_objcg; + struct obj_cgroup *_objcg; #endif #ifdef CONFIG_MEM_ALLOC_PROFILING - union codetag_ref _ctref; + union codetag_ref _ctref; #endif + }; } __aligned(8); =20 +static inline size_t cache_obj_ext_size(struct kmem_cache *s) +{ + size_t sz =3D 0; + + if (IS_ENABLED(CONFIG_MEMCG)) + sz +=3D 1; + + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + sz +=3D 1; + + return sizeof(struct slabobj_ext) * sz; +} + +static inline size_t slab_obj_ext_size(struct slab *slab) +{ + return cache_obj_ext_size(slab->slab_cache); +} + #ifdef CONFIG_SLAB_OBJ_EXT =20 /* @@ -651,18 +675,18 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab,= unsigned long obj_exts, { struct slabobj_ext *obj_ext; unsigned int index; + unsigned int stride; =20 VM_WARN_ON_ONCE(obj_exts !=3D slab_obj_exts(slab)); =20 index =3D obj_to_index(s, slab, obj); =20 - if (!obj_exts_in_object(slab)) { - obj_ext =3D ((struct slabobj_ext *)obj_exts) + index; - } else { - unsigned int stride =3D s->size; + if (!obj_exts_in_object(slab)) + stride =3D slab_obj_ext_size(slab); + else + stride =3D s->size; =20 - obj_ext =3D (struct slabobj_ext *)(obj_exts + index * stride); - } + obj_ext =3D (struct slabobj_ext *)(obj_exts + index * stride); =20 return kasan_reset_tag(obj_ext); } @@ -670,12 +694,14 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab,= unsigned long obj_exts, #ifdef CONFIG_MEMCG static inline struct obj_cgroup *slab_obj_ext_objcg(struct slabobj_ext *ob= j_ext) { + /* if objcg exists, it comes first, so we don't need to do anything */ return obj_ext->_objcg; } =20 static inline void slab_obj_ext_set_objcg(struct slabobj_ext *obj_ext, struct obj_cgroup *objcg) { + /* if objcg exists, it comes first, so we don't need to do anything */ obj_ext->_objcg =3D objcg; } #endif @@ -684,6 +710,9 @@ static inline void slab_obj_ext_set_objcg(struct slabob= j_ext *obj_ext, static inline union codetag_ref * slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext) { + if (IS_ENABLED(CONFIG_MEMCG)) + obj_ext +=3D 1; + return &obj_ext->_ctref; } #endif diff --git a/mm/slub.c b/mm/slub.c index f5b3eb44cb8a..287cbf3eddcc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -803,7 +803,7 @@ static inline bool need_slab_obj_exts(struct kmem_cache= *s) =20 static inline unsigned int obj_exts_size_in_slab(struct slab *slab) { - return sizeof(struct slabobj_ext) * slab->objects; + return slab_obj_ext_size(slab) * slab->objects; } =20 static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s, @@ -1199,7 +1199,7 @@ static void print_trailer(struct kmem_cache *s, struc= t slab *slab, u8 *p) off +=3D kasan_metadata_size(s, false); =20 if (obj_exts_in_object(slab)) - off +=3D sizeof(struct slabobj_ext); + off +=3D slab_obj_ext_size(slab); =20 if (off !=3D size_from_object(s)) /* Beginning of the filler is the free pointer */ @@ -1404,7 +1404,7 @@ static int check_pad_bytes(struct kmem_cache *s, stru= ct slab *slab, u8 *p) off +=3D kasan_metadata_size(s, false); =20 if (obj_exts_in_object(slab)) - off +=3D sizeof(struct slabobj_ext); + off +=3D slab_obj_ext_size(slab); =20 if (size_from_object(s) =3D=3D off) return 1; @@ -2094,6 +2094,8 @@ static inline bool mark_failed_objexts_alloc(struct s= lab *slab) static inline void handle_failed_objexts_alloc(struct slab *slab, unsigned long obj_exts, struct slabobj_ext *vec) { + unsigned int stride; + if (!mem_alloc_profiling_enabled()) return; =20 @@ -2105,11 +2107,13 @@ static inline void handle_failed_objexts_alloc(stru= ct slab *slab, if (obj_exts !=3D OBJEXTS_ALLOC_FAIL) return; =20 + stride =3D slab_obj_ext_size(slab) / sizeof(*vec); + for (unsigned int i =3D 0; i < slab->objects; i++) { union codetag_ref *ref =3D slab_obj_ext_codetag_ref(slab, vec); =20 set_codetag_empty(ref); - vec++; + vec +=3D stride; } } =20 @@ -2135,7 +2139,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kme= m_cache *s, unsigned long new_exts; unsigned long old_exts; struct slabobj_ext *vec; - size_t sz =3D sizeof(struct slabobj_ext) * slab->objects; + size_t sz =3D slab_obj_ext_size(slab) * slab->objects; =20 gfp &=3D ~OBJCGS_CLEAR_MASK; /* @@ -2278,8 +2282,7 @@ static void alloc_slab_obj_exts_early(struct kmem_cac= he *s, struct slab *slab) =20 get_slab_obj_exts(obj_exts); for_each_object(addr, s, slab_address(slab), slab->objects) - memset(kasan_reset_tag(addr) + offset, 0, - sizeof(struct slabobj_ext)); + memset(kasan_reset_tag(addr) + offset, 0, slab_obj_ext_size(slab)); put_slab_obj_exts(obj_exts); =20 #ifdef CONFIG_MEMCG @@ -7938,7 +7941,7 @@ static int calculate_sizes(struct kmem_cache_args *ar= gs, struct kmem_cache *s) aligned_size =3D ALIGN(size, s->align); #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT) if (slab_args_unmergeable(args, s->flags) && - (aligned_size - size >=3D sizeof(struct slabobj_ext))) + (aligned_size - size >=3D cache_obj_ext_size(s))) s->flags |=3D SLAB_OBJ_EXT_IN_OBJ; #endif size =3D aligned_size; --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C95EF42E8CF; Mon, 20 Jul 2026 14:16:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557020; cv=none; b=ED27wg8aDACmPKscMKz15C8UpwrSe+aqxIaeAO6zjFcyU8Bm5Frd8l1gvDie/2NMrt7r6mbKJf5Y7TxrrWGFi9JwfzJO09dApKZ8b/K0lM6pbNqbr3kiYe5NHs/YOYIEQS5qkkZ01Tu0hT8iuvK+OmGloSHSDGoxA5MUyvRmZfg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557020; c=relaxed/simple; bh=Rl7X5Rc2qzs8VSmnN6mZfUDi4ETsgSO0pJ6+SeGk3qc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=i8PxDLZuVnW2wfqPx0hYi1aSJltDnEKHSQodJ20lxz8/HVoDoR/7H54pqFW0251jrCctDFm/IwG/PL5xlGGZ/Yjh6TEARwnXd/4d0s537DYrxspohcz/FbXtpn62JyUl+vAVWKJ8hTYRDQeb50sg5jNANx2NpIPQ0zJkSkDPIy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=E/pkkBJT; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="E/pkkBJT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB6FF1F00A3A; Mon, 20 Jul 2026 14:16:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557019; bh=p2cb0SC7Ft0dMHSHileX02Rth4klsLBw5vrXUr6czBQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=E/pkkBJTKEu06X2E0MtqbAqYG9hOrori+/Hbw0Q15iKK2xoCJuysaSbUjQnTjr4v/ /sTpoSxqfoJHBP5gshywigUIIVPwl0y6AldIvzDUc1t7LW+t+/Wy2DKN3jL5k9+BZq iXnq+W94dP+4WXVmojEjornHrVlpBFywNqv2qk4gyo3snXvWHHi8OKPz5ov+g+vpgf lvtTEVgxdkGSoANsjGKvL28F1b71Tgmpzi6j2LGN7mDAHeEV0OiaP+PajPeWFhHk/q cQx4wexLQr5akB2uGT+LQagSMZfrbAFm7fJ3e+x4BVBeRZAcBvwIR4MvSxyMZX+joL 4LFq343++leMw== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:24 +0200 Subject: [PATCH v2 10/13] mm/slab: introduce slab_obj_ext_has_codetag() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-10-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 mem_alloc_profiling_enabled() allows evaluating (with a static key) if memory profiling is currently enabled. mem_profiling_support is a variable where false means it's not possible to enable it anymore, because the system was booted with "never" or it was later shut down. This is possible to query by mem_alloc_profiling_permanently_disabled(). To make slabobj_ext array size handling dynamic, we need a snapshot of mem_alloc_profiling_permanently_disabled() early in boot, so that's not affected by a later shutdown. We also need it to be static key based for performance. Neither mem_alloc_profiling_enabled() nor mem_alloc_profiling_permanently_disabled() satisfy this. Therefore introduce slab_obj_ext_has_codetag() with an underlying static key for that use case. Its state is made to reflect the result of mem_alloc_profiling_permanently_disabled() during kmem_cache_init(), which does happen after setup_early_mem_profiling(). Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) --- mm/slab.h | 16 ++++++++++++++++ mm/slub.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/mm/slab.h b/mm/slab.h index f8446167e175..983cb1119a76 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -568,6 +568,22 @@ struct slabobj_ext { }; } __aligned(8); =20 +#ifdef CONFIG_MEM_ALLOC_PROFILING +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + slab_obj_ext_has_codetag_key); + +static inline bool slab_obj_ext_has_codetag(void) +{ + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + &slab_obj_ext_has_codetag_key); +} +#else +static inline bool slab_obj_ext_has_codetag(void) +{ + return false; +} +#endif + static inline size_t cache_obj_ext_size(struct kmem_cache *s) { size_t sz =3D 0; diff --git a/mm/slub.c b/mm/slub.c index 287cbf3eddcc..d78d3e50c877 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -213,6 +213,11 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); static DEFINE_STATIC_KEY_FALSE(strict_numa); #endif =20 +#ifdef CONFIG_MEM_ALLOC_PROFILING +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + slab_obj_ext_has_codetag_key); +#endif + /* Structure holding extra parameters for slab allocations */ struct slab_alloc_context { unsigned long caller_addr; @@ -2425,6 +2430,25 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, s= truct slab *slab, void **p, __alloc_tagging_slab_free_hook(s, slab, p, objects); } =20 +/* + * Make sure the static key used by slab_obj_ext_has_codetag() reflects the + * value of !mem_alloc_profiling_permanently_disabled() + * + * Any later mem alloc profiling shutdown won't be reflected in the static= key + * because obj_exts with codetags might already exist. + */ +static void __init slab_obj_ext_has_codetag_init(void) +{ + bool need_codetag =3D !mem_alloc_profiling_permanently_disabled(); + + if (need_codetag !=3D static_key_enabled(&slab_obj_ext_has_codetag_key)) { + if (need_codetag) + static_branch_enable(&slab_obj_ext_has_codetag_key); + else + static_branch_disable(&slab_obj_ext_has_codetag_key); + } +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ =20 static inline void @@ -2439,6 +2463,10 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, s= truct slab *slab, void **p, { } =20 +static inline void slab_obj_ext_has_codetag_init(void) +{ +} + #endif /* CONFIG_MEM_ALLOC_PROFILING */ =20 =20 @@ -8551,6 +8579,8 @@ void __init kmem_cache_init(void) boot_kmem_cache_node; int node; =20 + slab_obj_ext_has_codetag_init(); + if (debug_guardpage_minorder()) slub_max_order =3D 0; =20 --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC65542F6F1; Mon, 20 Jul 2026 14:17:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557025; cv=none; b=O4zi1l3zLyzO0CcITY1rUBGvwQCXThm62ww8ZaEl4t7CYfyxWhTyteR5yltMUXVwudUMIYbmfnJ9nmMHOclc6QPJAxVMWvZNC2v8lFS6vYO27GVUKdCi1OnOodNcs1pBzB3N4ebL5PEHTS6HmPnU4RPlz4+qsp+0P1r2G/QBrxM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557025; c=relaxed/simple; bh=yOyOx1Af/unlEzIdYJngHCqv6wVo6jclyAheX9qKGk4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ehMY7B/Ppkn9xMaruEASCpr+HrTyAG0ZG+JQMpUoauhbDZBqOsZXCwx5rAqSFHDV40tviU5IfHxQg7VmBktjpq4Uv9sWG0QLk4CE4imTCyTdryUk0Wde/C4IrXOdOFI/EwC7rBtzsGxvzs5Fu7Yvum59UIvDvvnR3+Col8gT0+s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=egnIImWA; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="egnIImWA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DB38C1F000E9; Mon, 20 Jul 2026 14:16:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557022; bh=J2QHy+8aDvUuK1lYAmgQR7tOtfpX0R/pRz+K12b83rg=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=egnIImWAjekWM0hyh5G7FSbHakblC40grDwPP0aXTYu/nLY3GNOU4WJwFVBI8QzvF NpjLHTmr9QdDTg1pR3ejvqLf0NPsYbOGW74VEYYtfiVp1V4h5V1UWYu2nv+z0z5J0w XNAm20w1ucPg26aUEcjVxU0L20wNDXViVjPkoOnaBoKziKP6ZSZ2b3VZ6P8J5DUegw iaItZCo24oMkV/NtY1OO1xYeBqmQ1WpUgas+y4ly2QnGzsX6NkWIjxmHBb18fgDIZJ GwoTUKeNPa0trKdMm9oyPvukyGBv5Cc81mEQ7X9+0umyyPbdlGhBhH/RNCjSk54t8P qQPvI4ATQTkFQ== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:25 +0200 Subject: [PATCH v2 11/13] mm/slab: reduce slabobj_ext memory with allocation profiling disabled Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-11-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 When memory allocation profiling is compiled in but permanently disabled on boot with (implicit or explicit) "never" parameter, stop allocating (wasting) memory for the codetag_ref parts of slabobj_ext metadata. Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE) --- mm/slab.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slab.h b/mm/slab.h index 983cb1119a76..c5f37a46434e 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -591,7 +591,7 @@ static inline size_t cache_obj_ext_size(struct kmem_cac= he *s) if (IS_ENABLED(CONFIG_MEMCG)) sz +=3D 1; =20 - if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + if (slab_obj_ext_has_codetag()) sz +=3D 1; =20 return sizeof(struct slabobj_ext) * sz; --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADD7E42F6FF; Mon, 20 Jul 2026 14:17:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557026; cv=none; b=Au4tG82KzEkukCCP19sNb/5Hmb83M+O/ii05vGFikikHf/G5Vbd7f0QdDVA2yvLSJSxV70fWyXAZfWxaeO/ab3j+neEfJRo0jOQQlzCwMpFuKdgYPqJ5MfAk3ysqsC8k2iUeWtlMSxYTSIOj4HugcRAj7mbrgJuxE3TNKGLbQi0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557026; c=relaxed/simple; bh=Qc60nbMRVG9W/z1Hm48230q0MG2/ixzS/BSzEII/0EM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=NKY54fjzZupjfbLSYGDb/7/O5cBaeOBWQZrofYM++O63riUX+0mc1SbwvQEEiashxoEu2ufm6J7SOOXQaLu9N3GBYLbDkVEo2xrlhaUfai57LsSZgXduPEi/S8gMaITNGSVP5T4NNxykhnUn5YGE2dhUmzSyY1QiJDvN6a5BIRY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JVDub6Es; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JVDub6Es" Received: by smtp.kernel.org (Postfix) with ESMTPSA id ED0321F00A3D; Mon, 20 Jul 2026 14:17:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557025; bh=PKtmpQd7DPgnVftxCtGlnwK8PF1qRuxFiLgmmihblnM=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=JVDub6Es2QCeoRpdWlUpFhxs0iL6jDyZ7azuzPC5ljaXBYgFmorxwg8bg+KYnHvaD 7wxnO3abCFcoqdu1Jzt+bsn/FlqNOHn7D0WC/G2tkoh5SrKuc5H+oiwWm/15sgxMEL YjQe1TAJIF1xUG1YSv1RILv6Xzt/IQHkrcQGNXagTrxmfV6jI/loPXhRqfu7Y+aUT5 rDtx8bh51AFLM154oR+vbCr/nDSGnbvYAylExFkv2bJGdj7ukhuLYTUKrjF6iK9jfp Vhz69cZzCUHrW52DXwgTLSQBOU+5eBMMDvUCvcmqYxzbmTgQ7iyG5aj9BQvdR/sELg +t1fDBs++G6iw== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:26 +0200 Subject: [PATCH v2 12/13] mm/slab: add cache_ and slab_needs_objcg() helpers Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-12-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Slabs of some caches never need the objcg part of struct slabobj_ext. Introduce helpers to query this for a cache or a slab. Introduce SLAB_MAY_ACCOUNT flag that is currently only internal and only is_kmalloc_normal() caches don't have the flag. For named caches we currently can't derive SLAB_MAY_ACCOUNT from SLAB_ACCOUNT because some caches might be created without SLAB_ACCOUNT and then used both with and without __GFP_ACCOUNT concurrently, allocating obj_ext arrays on demand. So just add the SLAB_MAY_ACCOUNT to all kmem caches, unless kmem accounting is disabled. This can be improved later by finding out all caches used with __GFP_ACCOUNT, adding the SLAB_MAY_ACCOUNT flag to them explicitly and then ignoring __GFP_ACCOUNT for all other caches (with possible warning). To make the evaluation of slab_needs_objcg() faster in the allocation and free fast paths, add a obj_exts_needs_objcg flag into slab itself. This optimization is only available on 64bit architectures where free bits are available for the flag. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li --- include/linux/slab.h | 3 +++ mm/kfence/core.c | 3 +++ mm/slab.h | 31 +++++++++++++++++++++++++++++-- mm/slab_common.c | 25 ++++++++++++++++++++----- mm/slub.c | 6 +++++- 5 files changed, 60 insertions(+), 8 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 3a14df50766a..f9132902c5a6 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -45,6 +45,7 @@ enum _slab_flag_bits { #endif #ifdef CONFIG_MEMCG _SLAB_ACCOUNT, + _SLAB_MAY_ACCOUNT, #endif #ifdef CONFIG_KASAN_GENERIC _SLAB_KASAN, @@ -204,8 +205,10 @@ enum _slab_flag_bits { */ #ifdef CONFIG_MEMCG # define SLAB_ACCOUNT __SLAB_FLAG_BIT(_SLAB_ACCOUNT) +# define SLAB_MAY_ACCOUNT __SLAB_FLAG_BIT(_SLAB_MAY_ACCOUNT) #else # define SLAB_ACCOUNT __SLAB_FLAG_UNUSED +# define SLAB_MAY_ACCOUNT __SLAB_FLAG_UNUSED #endif =20 #ifdef CONFIG_KASAN_GENERIC diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 05b5482c8150..897ecf2594fb 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -640,6 +640,9 @@ static unsigned long kfence_init_pool(void) struct slab *slab =3D page_slab(page); slab->obj_exts =3D (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_e= xts | MEMCG_DATA_OBJEXTS; +#ifdef CONFIG_64BIT + slab->obj_exts_needs_objcg =3D 1; +#endif #endif } =20 diff --git a/mm/slab.h b/mm/slab.h index c5f37a46434e..65b44902c06e 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -81,10 +81,11 @@ struct freelist_counters { #ifdef CONFIG_64BIT /* * Some optimizations use free bits in 'counters' field - * to save memory. If these free bits are not available, - * such optimizations are disabled. + * to save memory or CPU. If these free bits are not + * available, such optimizations are disabled. */ unsigned obj_exts_in_object:1; + unsigned obj_exts_needs_objcg:1; #endif }; }; @@ -584,6 +585,32 @@ static inline bool slab_obj_ext_has_codetag(void) } #endif =20 +#ifdef CONFIG_MEMCG +static inline bool cache_needs_objcg(struct kmem_cache *cache) +{ + return (cache->flags & SLAB_MAY_ACCOUNT); +} + +static inline bool slab_needs_objcg(struct slab *slab) +{ +#ifdef CONFIG_64BIT + return slab->obj_exts_needs_objcg; +#else + return cache_needs_objcg(slab->slab_cache); +#endif +} +#else +static inline bool cache_needs_objcg(struct kmem_cache *cache) +{ + return false; +} + +static inline bool slab_needs_objcg(struct slab *slab) +{ + return false; +} +#endif + static inline size_t cache_obj_ext_size(struct kmem_cache *s) { size_t sz =3D 0; diff --git a/mm/slab_common.c b/mm/slab_common.c index 03ecac12cd86..4e983e250338 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -52,7 +52,7 @@ struct kmem_cache *kmem_cache; SLAB_OBJ_EXT_IN_OBJ) =20 #define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \ - SLAB_CACHE_DMA32 | SLAB_ACCOUNT) + SLAB_CACHE_DMA32 | SLAB_ACCOUNT | SLAB_MAY_ACCOUNT) =20 /* * Merge control. If this is set then no merging of slab caches will occur. @@ -359,6 +359,13 @@ struct kmem_cache *__kmem_cache_create_args(const char= *name, goto out_unlock; } =20 + /* + * For now we assume any cache can be used with __GFP_ACCOUNT and thus + * may need to store objcg pointers for objects + */ + if (!mem_cgroup_kmem_disabled()) + flags |=3D SLAB_MAY_ACCOUNT; + /* Fail closed on bad usersize of useroffset values. */ if (!IS_ENABLED(CONFIG_HARDENED_USERCOPY) || WARN_ON(!args->usersize && args->useroffset) || @@ -984,11 +991,19 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type ty= pe) #endif =20 /* - * If CONFIG_MEMCG is enabled, disable cache merging for - * KMALLOC_NORMAL caches. + * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not + * aliased with any other type, make sure it's never merged with any other + * cache. + * + * In other cases the kmalloc cache may end up being used for a + * __GFP_ACCOUNT allocation so mark it as such */ - if (IS_ENABLED(CONFIG_MEMCG) && (type =3D=3D KMALLOC_NORMAL)) - flags |=3D SLAB_NO_MERGE; + if (!mem_cgroup_kmem_disabled()) { + if (type =3D=3D KMALLOC_NORMAL && KMALLOC_RECLAIM !=3D KMALLOC_NORMAL) + flags |=3D SLAB_NO_MERGE; + else + flags |=3D SLAB_MAY_ACCOUNT; + } =20 if (minalign > ARCH_KMALLOC_MINALIGN) { aligned_size =3D ALIGN(aligned_size, minalign); diff --git a/mm/slub.c b/mm/slub.c index d78d3e50c877..30435e2509ea 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2557,7 +2557,7 @@ bool memcg_slab_post_charge(void *p, gfp_t flags) * of slab_obj_exts being allocated from the same slab and thus the slab * becoming effectively unfreeable. */ - if (is_kmalloc_normal(s)) + if (!cache_needs_objcg(s)) return true; =20 /* Ignore already charged objects. */ @@ -3435,6 +3435,10 @@ static struct slab *allocate_slab(struct kmem_cache = *s, gfp_t flags, =20 slab->objects =3D oo_objects(oo); =20 +#ifdef CONFIG_64BIT + if (cache_needs_objcg(s)) + slab->obj_exts_needs_objcg =3D 1; +#endif slab->slab_cache =3D s; =20 kasan_poison_slab(slab); --=20 2.55.0 From nobody Sat Jul 25 02:10:52 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E2D242B726; Mon, 20 Jul 2026 14:17:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557030; cv=none; b=h4getuWQJV4yGDCDyBd/LNI3XQs7RJLN73xSswESLLnujk3u6H8jU6i+Ia/8LlW/Oq5xyjlSptmtuN4iLZvdWKZS2CNcp1buTPcpujvF29PkBePuTBZVL6WkMdufTRMD4EPEWu1ib6cnr7Ev/x1d71QXvLwNJXpfz+XSxcTwZVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784557030; c=relaxed/simple; bh=ebUJQq50hWtJS9nsVENCUjdOnHucNEhidB+p61SyW3c=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=MFH+SRehte1HV8TW2Satxf0hJs0Ch692gWoNQygQfX52UNb6RsQoUgoIWwxNc9+1VbLKvr8v31Pb+21fqh3bqFusRPiyNhBKZ4PgFYPDm9lGx/4E+zsigs0c4DVkbDEPQ2sjslU6LI+iZ3x6em83tSUAn+LC4vDqlyv0Ym3dFUo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KIzxc26D; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KIzxc26D" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 080271F000E9; Mon, 20 Jul 2026 14:17:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784557028; bh=zNnHaVXxj/mvFCRCqUFR4sjoNHHhPJC/qg6efDIn92M=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=KIzxc26DhH1045e+ExrzClszygTXStN9ECH4U/XdwnS8Yusfg53YVrEKplbAB88oc 4X+Df9BjB+OdTt+/vJnXkTl4OX5SBWervBjCoAA748KRgWIZhdBX/djV/P7R8F1YM/ tIsk4nrzCIQ1zt8B/G5PRTp/VhZHNtwvuUhscDZsaHynVXz3lybUDVJRWT842UFc5I H8DRaiXxjTn0fWntsAXEQq0EPBS4RMl0wgaTYpU1fHKm1VIZAkmg2bjW3Em7oGJfA8 LZT3aAjEa3WdIoDoB+W6P0Scq2wxGAktVuXYVe9SBwmjsnNgQS5VD5Cy3uV/D2FTaX MUHPhfGNWXbng== From: "Vlastimil Babka (SUSE)" Date: Mon, 20 Jul 2026 16:16:27 +0200 Subject: [PATCH v2 13/13] mm/slab: stop allocating objcg pointers when unnecessary Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-objext_split-v2-13-2fa7c6f60dbe@kernel.org> References: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> In-Reply-To: <20260720-b4-objext_split-v2-0-2fa7c6f60dbe@kernel.org> To: Harry Yoo , Suren Baghdasaryan Cc: Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 Start using the slab_needs_objcg() helper to calculate slabobj_ext size. Caches that we know to never need objcg pointers (currently KMALLOC_NORMAL caches) will thus stop wasting memory on them when memory allocation profiling is enabled. For things to work properly, we need to also add slab_needs_objcg() checks to mem_cgroup_from_obj_slab() and memcg_slab_free_hook(), because when obj_exts array exists for a slab only due to mem_alloc profiling, we would otherwise attempt to access a non-existing objcg pointer in that slab. The function __memcg_slab_post_alloc_hook() should not be possible to call for a slab where slab_needs_objcg() is false, but add a DEBUG_VM check there to prevent breaking this assumption accidentally. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Hao Li --- mm/memcontrol.c | 6 ++++++ mm/slab.h | 14 +++++++++++--- mm/slub.c | 3 +++ 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cb1e97b4edc1..aace85fb99f9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2871,6 +2871,9 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct sl= ab *slab, void *p) if (!obj_exts) return NULL; =20 + if (!slab_needs_objcg(slab)) + return NULL; + get_slab_obj_exts(obj_exts); obj_ext =3D slab_obj_ext(slab->slab_cache, slab, obj_exts, p); objcg =3D slab_obj_ext_objcg(obj_ext); @@ -3580,6 +3583,9 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, =20 slab =3D virt_to_slab(p[i]); =20 + if (IS_ENABLED(CONFIG_DEBUG_VM) && WARN_ON_ONCE(!slab_needs_objcg(slab))) + continue; + if (!slab_obj_exts(slab) && alloc_slab_obj_exts(slab, s, flags, slab_alloc_flags)) { continue; diff --git a/mm/slab.h b/mm/slab.h index 65b44902c06e..18c2a807f023 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -615,7 +615,7 @@ static inline size_t cache_obj_ext_size(struct kmem_cac= he *s) { size_t sz =3D 0; =20 - if (IS_ENABLED(CONFIG_MEMCG)) + if (cache_needs_objcg(s)) sz +=3D 1; =20 if (slab_obj_ext_has_codetag()) @@ -626,7 +626,15 @@ static inline size_t cache_obj_ext_size(struct kmem_ca= che *s) =20 static inline size_t slab_obj_ext_size(struct slab *slab) { - return cache_obj_ext_size(slab->slab_cache); + size_t sz =3D 0; + + if (slab_needs_objcg(slab)) + sz +=3D 1; + + if (slab_obj_ext_has_codetag()) + sz +=3D 1; + + return sizeof(struct slabobj_ext) * sz; } =20 #ifdef CONFIG_SLAB_OBJ_EXT @@ -753,7 +761,7 @@ static inline void slab_obj_ext_set_objcg(struct slabob= j_ext *obj_ext, static inline union codetag_ref * slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext) { - if (IS_ENABLED(CONFIG_MEMCG)) + if (slab_needs_objcg(slab)) obj_ext +=3D 1; =20 return &obj_ext->_ctref; diff --git a/mm/slub.c b/mm/slub.c index 30435e2509ea..e6148b73b92f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2512,6 +2512,9 @@ void memcg_slab_free_hook(struct kmem_cache *s, struc= t slab *slab, void **p, if (likely(!obj_exts)) return; =20 + if (!slab_needs_objcg(slab)) + return; + get_slab_obj_exts(obj_exts); __memcg_slab_free_hook(s, slab, p, objects, obj_exts); put_slab_obj_exts(obj_exts); --=20 2.55.0