From nobody Thu Dec 25 01:30:58 2025 Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3624A5A0EA for ; Tue, 23 Jan 2024 09:33:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706002436; cv=none; b=NTnBxZNjJeDZVHUvQDgFmpChY8gxDZbWLJ0aMz0YAdTHly+caH00nrOgpf7gfVbE6XqiSVqHW0VEQkuXZPAXAhi/wFCgc8edexsctWFwjlY/n20e/ktX2qP97FB7gcndLhEsQvm7f3mvXDPTUtQs8W6X25FHapPh0WE42DFF+w4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706002436; c=relaxed/simple; bh=RxryMU48iVU4pnALkZXNrPy7bZDqLdaz2AEx4p3Ji5Q=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=INOmBbQUA8jqguzGAUJtfx8joJcoWd1/ogekWUedhgr0oGQ/K4ZnpmAPm3+/57cvh7YmFEL+rUlUVsIQK1MfohSEKIXQf6J+b02idxNRmVvpr5MHev86HG8XtRHNyPOLroBjazmP4R1p59JCN0rVS5SXtg8lC87soviN9CjnBnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=linux.dev; arc=none smtp.client-ip=95.215.58.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 23 Jan 2024 09:33:29 +0000 Subject: [PATCH v2 1/3] mm/slub: directly load freelist from cpu partial slab in the likely case Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240117-slab-misc-v2-1-81766907896e@bytedance.com> References: <20240117-slab-misc-v2-0-81766907896e@bytedance.com> In-Reply-To: <20240117-slab-misc-v2-0-81766907896e@bytedance.com> To: Joonsoo Kim , Vlastimil Babka , David Rientjes , Roman Gushchin , Pekka Enberg , Christoph Lameter , Andrew Morton , Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Vlastimil Babka , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, "Christoph Lameter (Ampere)" X-Developer-Signature: v=1; a=ed25519-sha256; t=1706002427; l=2205; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=RxryMU48iVU4pnALkZXNrPy7bZDqLdaz2AEx4p3Ji5Q=; b=378yGXlTZX4OD1uYzqIEJLiYJiKugeTycFRTFrJPpwzMaIF7wNnhWL/62RFkkKgtmJbEawLdi XuPcUkfqgQcAYhCGSKbsn/raWfAV9nLnWfdn7JURg7bvvxzMP/mmbwK X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT The likely case is that we get a usable slab from the cpu partial list, we can directly load freelist from it and return back, instead of going the other way that need more work, like reenable interrupt and recheck. But we need to remove the "VM_BUG_ON(!new.frozen)" in get_freelist() for reusing it, since cpu partial slab is not frozen. It seems acceptable since it's only for debug purpose. And get_freelist() also assumes it can return NULL if the freelist is empty, which is not possible for the cpu partial slab case, so we add "VM_BUG_ON(!freelist)" after get_freelist() to make it explicit. There is some small performance improvement too, which shows by: perf bench sched messaging -g 5 -t -l 100000 mm-stable slub-optimize Total time 7.473 7.209 Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..fda402b2d649 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3326,7 +3326,6 @@ static inline void *get_freelist(struct kmem_cache *s= , struct slab *slab) counters =3D slab->counters; =20 new.counters =3D counters; - VM_BUG_ON(!new.frozen); =20 new.inuse =3D slab->objects; new.frozen =3D freelist !=3D NULL; @@ -3498,18 +3497,20 @@ static void *___slab_alloc(struct kmem_cache *s, gf= p_t gfpflags, int node, =20 slab =3D slub_percpu_partial(c); slub_set_percpu_partial(c, slab); - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - stat(s, CPU_PARTIAL_ALLOC); =20 - if (unlikely(!node_match(slab, node) || - !pfmemalloc_match(slab, gfpflags))) { - slab->next =3D NULL; - __put_partials(s, slab); - continue; + if (likely(node_match(slab, node) && + pfmemalloc_match(slab, gfpflags))) { + c->slab =3D slab; + freelist =3D get_freelist(s, slab); + VM_BUG_ON(!freelist); + stat(s, CPU_PARTIAL_ALLOC); + goto load_freelist; } =20 - freelist =3D freeze_slab(s, slab); - goto retry_load_slab; + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + + slab->next =3D NULL; + __put_partials(s, slab); } #endif =20 --=20 b4 0.10.1 From nobody Thu Dec 25 01:30:58 2025 Received: from out-176.mta1.migadu.com (out-176.mta1.migadu.com [95.215.58.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 970AB5C5E1 for ; Tue, 23 Jan 2024 09:33:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706002439; cv=none; b=SOg4cWxOF3ysqJ0IPiGKqWHhbWB0gpv5W822Ff2jNy7HvR9r/wHKmj5dHpV+PzrPCNidQq8o08OvtQKoV4TRmC3CubYVzolrlymPKiD/LEEShK2uuJ16AVtLioJAjdZRf+uRs36ByOBADwB+DU/y9OeA8kQ8gRG5xgNfIA5negs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706002439; c=relaxed/simple; bh=cuwSKTPtiNqZTZ2D9xT7/HRhzSDVJccC9FWGQ4Ep0yw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=IHjkiNIMSKozfrwDzPAtpEvP8LJIljWX5feNXbIuPPiuR8NCxw1p1ugs8AGLEy9Y2n3NkGV0+tSuq50deZZxyQtsb55zSN9rTC2wVpRHA6N8xRjz0HCHAbBZGfu+QyCrd/ixAURrjbECTPo94SM7YD5yxl3oouKOgGRPrBzQmt4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=linux.dev; arc=none smtp.client-ip=95.215.58.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 23 Jan 2024 09:33:30 +0000 Subject: [PATCH v2 2/3] mm/slub: remove full list manipulation for non-debug slab Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240117-slab-misc-v2-2-81766907896e@bytedance.com> References: <20240117-slab-misc-v2-0-81766907896e@bytedance.com> In-Reply-To: <20240117-slab-misc-v2-0-81766907896e@bytedance.com> To: Joonsoo Kim , Vlastimil Babka , David Rientjes , Roman Gushchin , Pekka Enberg , Christoph Lameter , Andrew Morton , Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Vlastimil Babka , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, "Christoph Lameter (Ampere)" X-Developer-Signature: v=1; a=ed25519-sha256; t=1706002427; l=1153; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=cuwSKTPtiNqZTZ2D9xT7/HRhzSDVJccC9FWGQ4Ep0yw=; b=mJdc+YnrEwU86ZPAWNO2hIOji1KtwvraB80SdFXY8/3S14XjUgouN7U9QY4VsPfbv0iWMF2qa jNF2VdpUcx9CAUqOdk6g8VKSrJo7sg3GFd7D/n6FIaKYhdVOjnloqeD X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT Since debug slab is processed by free_to_partial_list(), and only debug slab which has SLAB_STORE_USER flag would care about the full list, we can remove these unrelated full list manipulations from __slab_free(). Acked-by: Christoph Lameter (Ampere) Reviewed-by: Vlastimil Babka Signed-off-by: Chengming Zhou --- mm/slub.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index fda402b2d649..5c6fbeef05a8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4188,7 +4188,6 @@ static void __slab_free(struct kmem_cache *s, struct = slab *slab, * then add it. */ if (!kmem_cache_has_cpu_partial(s) && unlikely(!prior)) { - remove_full(s, n, slab); add_partial(n, slab, DEACTIVATE_TO_TAIL); stat(s, FREE_ADD_PARTIAL); } @@ -4202,9 +4201,6 @@ static void __slab_free(struct kmem_cache *s, struct = slab *slab, */ remove_partial(n, slab); stat(s, FREE_REMOVE_PARTIAL); - } else { - /* Slab must be on the full list */ - remove_full(s, n, slab); } =20 spin_unlock_irqrestore(&n->list_lock, flags); --=20 b4 0.10.1 From nobody Thu Dec 25 01:30:58 2025 Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34F8A5C60B for ; Tue, 23 Jan 2024 09:34:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706002441; cv=none; b=soAlfNT7Xebd5hEzed/LB9AAXZigBbU/TW3svVkn8uv+Ry6oWxRBniEywPpGY4aaeUdr0XFS6DzJOPeBDIKFK2szQiq0YkVGrs1+om8iMEmFs7h9OlHmHqsce2A3moaEjDMozismDM5dygx7Rg0cKJqu7qNZfIieCtYNGK3m4+Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706002441; c=relaxed/simple; bh=ZXzd5ZM1kPfc29nku387qFGBNfMPhmyqnVdI70QYk5o=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Xb7lwhhICTDvoxEZF38oWVYN/kcZ2Hwcssl7DOb0VyNKjuoSpBe47Y4bboapkHCJ1yrhbjzWfxBCyc19FK4yCcCnu15sGqtSZiEU1RWgh+lSPx40dl3S4cO2oeVN7nsjUKQ62bMQeFutGg6GRX/O8jbMu8Cqek1Je5UDd7Y0ssg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=linux.dev; arc=none smtp.client-ip=95.215.58.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Tue, 23 Jan 2024 09:33:31 +0000 Subject: [PATCH v2 3/3] mm/slub: remove unused parameter in next_freelist_entry() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240117-slab-misc-v2-3-81766907896e@bytedance.com> References: <20240117-slab-misc-v2-0-81766907896e@bytedance.com> In-Reply-To: <20240117-slab-misc-v2-0-81766907896e@bytedance.com> To: Joonsoo Kim , Vlastimil Babka , David Rientjes , Roman Gushchin , Pekka Enberg , Christoph Lameter , Andrew Morton , Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Vlastimil Babka , linux-kernel@vger.kernel.org, Chengming Zhou , linux-mm@kvack.org, "Christoph Lameter (Ampere)" X-Developer-Signature: v=1; a=ed25519-sha256; t=1706002427; l=1540; i=zhouchengming@bytedance.com; s=20231204; h=from:subject:message-id; bh=ZXzd5ZM1kPfc29nku387qFGBNfMPhmyqnVdI70QYk5o=; b=eO3C/qpMGEax+MfzVtLNHcbDe+c64BixN/5g6zrbrjsUMaxGOraPVsgUT0dI0grRB+ucrDO2l j/WoJtVo5iuD79mh2KmwA9Hx/yNMXxarI/hcbm5mwl8odmEhDnXWi2x X-Developer-Key: i=zhouchengming@bytedance.com; a=ed25519; pk=xFTmRtMG3vELGJBUiml7OYNdM393WOMv0iWWeQEVVdA= X-Migadu-Flow: FLOW_OUT The parameter "struct slab *slab" is unused in next_freelist_entry(), so just remove it. Acked-by: Christoph Lameter (Ampere) Reviewed-by: Vlastimil Babka Signed-off-by: Chengming Zhou --- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 5c6fbeef05a8..7f235fa6592d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2243,7 +2243,7 @@ static void __init init_freelist_randomization(void) } =20 /* Get the next entry on the pre-computed freelist randomized */ -static void *next_freelist_entry(struct kmem_cache *s, struct slab *slab, +static void *next_freelist_entry(struct kmem_cache *s, unsigned long *pos, void *start, unsigned long page_limit, unsigned long freelist_count) @@ -2282,13 +2282,12 @@ static bool shuffle_freelist(struct kmem_cache *s, = struct slab *slab) start =3D fixup_red_left(s, slab_address(slab)); =20 /* First entry is used as the base of the freelist */ - cur =3D next_freelist_entry(s, slab, &pos, start, page_limit, - freelist_count); + cur =3D next_freelist_entry(s, &pos, start, page_limit, freelist_count); cur =3D setup_object(s, cur); slab->freelist =3D cur; =20 for (idx =3D 1; idx < slab->objects; idx++) { - next =3D next_freelist_entry(s, slab, &pos, start, page_limit, + next =3D next_freelist_entry(s, &pos, start, page_limit, freelist_count); next =3D setup_object(s, next); set_freepointer(s, cur, next); --=20 b4 0.10.1