From nobody Thu Dec 18 00:47:01 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17E50C004C0 for ; Sat, 21 Oct 2023 14:44:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231442AbjJUOoT (ORCPT ); Sat, 21 Oct 2023 10:44:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231429AbjJUOoR (ORCPT ); Sat, 21 Oct 2023 10:44:17 -0400 Received: from out-210.mta1.migadu.com (out-210.mta1.migadu.com [IPv6:2001:41d0:203:375::d2]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CBC0D7C for ; Sat, 21 Oct 2023 07:44:13 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fc6duZ/q+apDqgoP4cLzy3p5t8cD8/eKarQEAhD6cwA=; b=VwdHbGN6iqPqOPPmjkoCZCqeya5Wrnfy29DfCh17oso4mTQaIrjGQ/bBj7cuImP6Ak0ZDt 74Ji4pLY0KWggdSHOhw8II53OTlr2K3tbgg2NViY04lO63dYbYrl8s/FsQ+pKx4DbOjdEG yYcnIac3YU/gH6s55G+958BKYZfu70Y= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 3/6] slub: Don't freeze slabs for cpu partial Date: Sat, 21 Oct 2023 14:43:14 +0000 Message-Id: <20231021144317.3400916-4-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Chengming Zhou Now we will freeze slabs when moving them out of node partial list to cpu partial list, this method needs two cmpxchg_double operations: 1. freeze slab (acquire_slab()) under the node list_lock 2. get_freelist() when pick used in ___slab_alloc() Actually we don't need to freeze when moving slabs out of node partial list, we can delay freeze to use slab freelist in ___slab_alloc(), so we can save one cmpxchg_double(). And there are other good points: 1. The moving of slabs between node partial list and cpu partial list becomes simpler, since we don't need to freeze or unfreeze at all. 2. The node list_lock contention would be less, since we only need to freeze one slab under the node list_lock. (In fact, we can first move slabs out of node partial list, don't need to freeze any slab at all, so the contention on slab won't transfer to the node list_lock contention.) We can achieve this because there is no concurrent path would manipulate the partial slab list except the __slab_free() path, which is serialized now. Note this patch just change the parts of moving the partial slabs for easy code review, we will fix other parts in the following patches. Specifically this patch change three paths: 1. get partial slab from node: get_partial_node() 2. put partial slab to node: __unfreeze_partials() 3. cache partail slab on cpu when __slab_free() Signed-off-by: Chengming Zhou --- mm/slub.c | 63 +++++++++++++++++-------------------------------------- 1 file changed, 19 insertions(+), 44 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index adeff8df85ec..61ee82ea21b6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2277,7 +2277,9 @@ static void *get_partial_node(struct kmem_cache *s, s= truct kmem_cache_node *n, struct slab *slab, *slab2; void *object =3D NULL; unsigned long flags; +#ifdef CONFIG_SLUB_CPU_PARTIAL unsigned int partial_slabs =3D 0; +#endif =20 /* * Racy check. If we mistakenly see no partial slabs then we @@ -2303,20 +2305,22 @@ static void *get_partial_node(struct kmem_cache *s,= struct kmem_cache_node *n, continue; } =20 - t =3D acquire_slab(s, n, slab, object =3D=3D NULL); - if (!t) - break; - if (!object) { - *pc->slab =3D slab; - stat(s, ALLOC_FROM_PARTIAL); - object =3D t; - } else { - put_cpu_partial(s, slab, 0); - stat(s, CPU_PARTIAL_NODE); - partial_slabs++; + t =3D acquire_slab(s, n, slab, object =3D=3D NULL); + if (t) { + *pc->slab =3D slab; + stat(s, ALLOC_FROM_PARTIAL); + object =3D t; + continue; + } } + #ifdef CONFIG_SLUB_CPU_PARTIAL + remove_partial(n, slab); + put_cpu_partial(s, slab, 0); + stat(s, CPU_PARTIAL_NODE); + partial_slabs++; + if (!kmem_cache_has_cpu_partial(s) || partial_slabs > s->cpu_partial_slabs / 2) break; @@ -2606,9 +2610,6 @@ static void __unfreeze_partials(struct kmem_cache *s,= struct slab *partial_slab) unsigned long flags =3D 0; =20 while (partial_slab) { - struct slab new; - struct slab old; - slab =3D partial_slab; partial_slab =3D slab->next; =20 @@ -2621,23 +2622,7 @@ static void __unfreeze_partials(struct kmem_cache *s= , struct slab *partial_slab) spin_lock_irqsave(&n->list_lock, flags); } =20 - do { - - old.freelist =3D slab->freelist; - old.counters =3D slab->counters; - VM_BUG_ON(!old.frozen); - - new.counters =3D old.counters; - new.freelist =3D old.freelist; - - new.frozen =3D 0; - - } while (!__slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); - - if (unlikely(!new.inuse && n->nr_partial >=3D s->min_partial)) { + if (unlikely(!slab->inuse && n->nr_partial >=3D s->min_partial)) { slab->next =3D slab_to_discard; slab_to_discard =3D slab; } else { @@ -3634,18 +3619,8 @@ static void __slab_free(struct kmem_cache *s, struct= slab *slab, was_frozen =3D new.frozen; new.inuse -=3D cnt; if ((!new.inuse || !prior) && !was_frozen) { - - if (kmem_cache_has_cpu_partial(s) && !prior) { - - /* - * Slab was on no list before and will be - * partially empty - * We can defer the list move and instead - * freeze it. - */ - new.frozen =3D 1; - - } else { /* Needs to be taken off a list */ + /* Needs to be taken off a list */ + if (!kmem_cache_has_cpu_partial(s) || prior) { =20 n =3D get_node(s, slab_nid(slab)); /* @@ -3675,7 +3650,7 @@ static void __slab_free(struct kmem_cache *s, struct = slab *slab, * activity can be necessary. */ stat(s, FREE_FROZEN); - } else if (new.frozen) { + } else if (kmem_cache_has_cpu_partial(s) && !prior) { /* * If we just froze the slab then put it onto the * per cpu partial list. --=20 2.20.1