From nobody Thu Dec 25 10:50:18 2025 Received: from out-181.mta0.migadu.com (out-181.mta0.migadu.com [91.218.175.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10AF41DFE2 for ; Wed, 17 Jan 2024 11:46:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705491995; cv=none; b=didOk11mZAsM9T82CALuJE3zBhIzXiwcK3f6chsL5W9UdaBk6KfHxWqzeeIv2ByC3hFNIHUUWKYk0sGIezLeiyRRNuNMPCWXA1IRDAN4fuC0Qnx+uvdwA8tGSvMiVwOxCyK6BQM+aOidxRUrP0SwALYtSqNoZ2atjD+ph4SRMPE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705491995; c=relaxed/simple; bh=1cV2tT3H0/m/Zwl/1edZJTWNsxvFuwH6aWvUH2ceDxA=; h=X-Report-Abuse:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Developer-Signature:X-Developer-Key:X-Migadu-Flow; b=H+LcoAritebrwHgDouzMybxhsP5Ts3iUmfw4DbyJ1Skv5XRKe7HyH9O7i0fKWOssty1c207B0uu/OALNQrVP3sSApIers+cTDChXmhtpgv2sszPhBHAllF5pCIBgTIQhC58M4UpbRAPiY1yZe4BHeVqbsLHQiMDDaZhAMw572T0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=linux.dev; arc=none smtp.client-ip=91.218.175.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 17 Jan 2024 11:45:58 +0000 Subject: [PATCH 1/3] mm/slub: directly load freelist from cpu partial slab in the likely case Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240117-slab-misc-v1-1-fd1c49ccbe70@bytedance.com> References: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com> In-Reply-To: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com> To: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Joonsoo Kim , Vlastimil Babka , Christoph Lameter , Pekka Enberg , Andrew Morton , Roman Gushchin , David Rientjes Cc: linux-mm@kvack.org, Chengming Zhou , linux-kernel@vger.kernel.org X-Developer-Signature: v=1; a=ed25519-sha256; t=1705491984; l=1967; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=1cV2tT3H0/m/Zwl/1edZJTWNsxvFuwH6aWvUH2ceDxA=; b=lspzQdoDP02OxesbrkhjrNtCmxWtOMg+h1t6s9v2piAwijqH3Tc7Q1b13y4pdQjtZJBFFuN85 MSi/7Nl/He4A7iY8555VHvt+jjR22vlwQ2KgFZAuvEDcaZ+skfH0ZEc X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT The likely case is that we get a usable slab from the cpu partial list, we can directly load freelist from it and return back, instead of going the other way that need more work, like reenable interrupt and recheck. But we need to remove the "VM_BUG_ON(!new.frozen)" in get_freelist() for reusing it, since cpu partial slab is not frozen. It seems acceptable since it's only for debug purpose. There is some small performance improvement too, which shows by: perf bench sched messaging -g 5 -t -l 100000 mm-stable slub-optimize Total time 7.473 7.209 Signed-off-by: Chengming Zhou --- mm/slub.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..20c03555c97b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3326,7 +3326,6 @@ static inline void *get_freelist(struct kmem_cache *s= , struct slab *slab) counters =3D slab->counters; =20 new.counters =3D counters; - VM_BUG_ON(!new.frozen); =20 new.inuse =3D slab->objects; new.frozen =3D freelist !=3D NULL; @@ -3498,18 +3497,19 @@ static void *___slab_alloc(struct kmem_cache *s, gf= p_t gfpflags, int node, =20 slab =3D slub_percpu_partial(c); slub_set_percpu_partial(c, slab); - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - stat(s, CPU_PARTIAL_ALLOC); =20 - if (unlikely(!node_match(slab, node) || - !pfmemalloc_match(slab, gfpflags))) { - slab->next =3D NULL; - __put_partials(s, slab); - continue; + if (likely(node_match(slab, node) && + pfmemalloc_match(slab, gfpflags))) { + c->slab =3D slab; + freelist =3D get_freelist(s, slab); + stat(s, CPU_PARTIAL_ALLOC); + goto load_freelist; } =20 - freelist =3D freeze_slab(s, slab); - goto retry_load_slab; + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + + slab->next =3D NULL; + __put_partials(s, slab); } #endif =20 --=20 b4 0.10.1 From nobody Thu Dec 25 10:50:18 2025 Received: from out-179.mta0.migadu.com (out-179.mta0.migadu.com [91.218.175.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B18B41EA74 for ; Wed, 17 Jan 2024 11:46:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705491999; cv=none; b=D0EY42Ep+PxMVIzGQGluhunf3aD8aFiKagmo+9I6TiN8r1KNelpy6nRKGnJUlbYnqQF/y/J9aQrv0XY0rPXbEnOFmAEnkkv6H1s8tC4Ubism1CZvP3GPWRhn5dnNx75TkBf2ZWzfwPHeep0ipoZqTjft73vuU66sE5xD22fFSXM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705491999; c=relaxed/simple; bh=dSgFg48t7LupTsKBO2gZsJttDy0+Yco9PB4eWVCuRJk=; h=X-Report-Abuse:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Developer-Signature:X-Developer-Key:X-Migadu-Flow; b=EWwdEYYLiKqurzLTK++xWd5ey2ipob9Fbtqr41i6yQPofI2AwqulpDkv5K2lV7zTk4TREAMMIN0bciHY59GJDuu4v/GPT2ekPPNTkz4AZyE0XXZ217/YXx8zmLx0b+NSq13QCLHUp36zA7+OuVOGtG3q3mnJ8wLqKGBfScTr6AM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=linux.dev; arc=none smtp.client-ip=91.218.175.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 17 Jan 2024 11:45:59 +0000 Subject: [PATCH 2/3] mm/slub: remove full list manipulation for non-debug slab Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240117-slab-misc-v1-2-fd1c49ccbe70@bytedance.com> References: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com> In-Reply-To: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com> To: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Joonsoo Kim , Vlastimil Babka , Christoph Lameter , Pekka Enberg , Andrew Morton , Roman Gushchin , David Rientjes Cc: linux-mm@kvack.org, Chengming Zhou , linux-kernel@vger.kernel.org X-Developer-Signature: v=1; a=ed25519-sha256; t=1705491984; l=1053; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=dSgFg48t7LupTsKBO2gZsJttDy0+Yco9PB4eWVCuRJk=; b=5XIIuX0qwsSqEvymGfIXuXLWpVtl9zXEjYel4Re50np/cxam9armg3d5VQKRVueB0DNreHkfP hgVbFAM35ZaDZEywwRZZ52cNqr8+6l7gbh1JIGosavcoqYyq5KQue2/ X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT Since debug slab is processed by free_to_partial_list(), and only debug slab which has SLAB_STORE_USER flag would care about the full list, we can remove these unrelated full list manipulations from __slab_free(). Signed-off-by: Chengming Zhou Acked-by: Christoph Lameter (Ampere) Reviewed-by: Vlastimil Babka --- mm/slub.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 20c03555c97b..f0307e8b4cd2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4187,7 +4187,6 @@ static void __slab_free(struct kmem_cache *s, struct = slab *slab, * then add it. */ if (!kmem_cache_has_cpu_partial(s) && unlikely(!prior)) { - remove_full(s, n, slab); add_partial(n, slab, DEACTIVATE_TO_TAIL); stat(s, FREE_ADD_PARTIAL); } @@ -4201,9 +4200,6 @@ static void __slab_free(struct kmem_cache *s, struct = slab *slab, */ remove_partial(n, slab); stat(s, FREE_REMOVE_PARTIAL); - } else { - /* Slab must be on the full list */ - remove_full(s, n, slab); } =20 spin_unlock_irqrestore(&n->list_lock, flags); --=20 b4 0.10.1 From nobody Thu Dec 25 10:50:18 2025 Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EC2F1EB31 for ; Wed, 17 Jan 2024 11:46:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705492002; cv=none; b=tP/tjJ50G4jnCfL6KtpugbIzbxn0RqrQs4zTWtt8x/+FbRZeA6QU81G47Jub3jNe2WzmBtbXTvwfMuZHFguiBFbsy1l3ATi+T7rBOzuE+JFWN7uHhcN0NSXqasifX4SehG+sl5ekzuNxqUQ3BW7d3DCDx2xp3EHYE24m2B5UQGE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705492002; c=relaxed/simple; bh=FCOnwYrhZkluxCwU1rI7qym40IkKZ74sKbmGZdbV/NU=; h=X-Report-Abuse:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Developer-Signature:X-Developer-Key:X-Migadu-Flow; b=q3euE1rv+7uIfFUJM+PCw6z+aBGxv4pEr2MD/NLZpqoxorUXfj4SDYxZcocEP/rK4HdwfCZNXpmLw5kgPlVCLLAfa+UR5SQgESSDfD1Bn0bgMLeqA5Oh41GrrMvSsl7u+oU2RzX32MOllKOQuByuB8AJdg9sPDUWgI7YfZBKtcs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=linux.dev; arc=none smtp.client-ip=91.218.175.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Wed, 17 Jan 2024 11:46:00 +0000 Subject: [PATCH 3/3] mm/slub: remove unused parameter in next_freelist_entry() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240117-slab-misc-v1-3-fd1c49ccbe70@bytedance.com> References: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com> In-Reply-To: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com> To: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Joonsoo Kim , Vlastimil Babka , Christoph Lameter , Pekka Enberg , Andrew Morton , Roman Gushchin , David Rientjes Cc: linux-mm@kvack.org, Chengming Zhou , linux-kernel@vger.kernel.org X-Developer-Signature: v=1; a=ed25519-sha256; t=1705491984; l=1440; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=FCOnwYrhZkluxCwU1rI7qym40IkKZ74sKbmGZdbV/NU=; b=r60bI1xFOSY0XMd/T4Vsn0QUucBv9bV6X7Ko6gDXvzMuaW9qAxNwy6xtQTq9LHk4bfdsUq04n Izus8lbaiz1CHUbUv63tGpbiP1Tp3eAQ0hAti23U2ipWl75vInBFP5w X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT The parameter "struct slab *slab" is unused in next_freelist_entry(), so just remove it. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f0307e8b4cd2..3a4e2f8d341c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2243,7 +2243,7 @@ static void __init init_freelist_randomization(void) } =20 /* Get the next entry on the pre-computed freelist randomized */ -static void *next_freelist_entry(struct kmem_cache *s, struct slab *slab, +static void *next_freelist_entry(struct kmem_cache *s, unsigned long *pos, void *start, unsigned long page_limit, unsigned long freelist_count) @@ -2282,13 +2282,12 @@ static bool shuffle_freelist(struct kmem_cache *s, = struct slab *slab) start =3D fixup_red_left(s, slab_address(slab)); =20 /* First entry is used as the base of the freelist */ - cur =3D next_freelist_entry(s, slab, &pos, start, page_limit, - freelist_count); + cur =3D next_freelist_entry(s, &pos, start, page_limit, freelist_count); cur =3D setup_object(s, cur); slab->freelist =3D cur; =20 for (idx =3D 1; idx < slab->objects; idx++) { - next =3D next_freelist_entry(s, slab, &pos, start, page_limit, + next =3D next_freelist_entry(s, &pos, start, page_limit, freelist_count); next =3D setup_object(s, next); set_freepointer(s, cur, next); --=20 b4 0.10.1