From nobody Sun May 24 19:33:20 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B53EE41B36A for ; Fri, 22 May 2026 14:23:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779459824; cv=none; b=YYM2ZI8SzJBb5vUDCe3eJAtHgNB5HsGjSgjqNUXRSDDh0YEBb8OpnRA4gfmdFkYKMS5EUAHo84lwYkkXWvgmEMyK+GXV10rQy0xv6pHCO22c49o0SleZcUEGukiyba9dsEF4hlrMg+LJbuGgp0nhwtC7JW2ACqEP7MRbtBYVjW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779459824; c=relaxed/simple; bh=4r4E8fF8xlQ2gCDGhw11J9DZ3L6ugESsDuH1k5p0Uwc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=sz38ySGexHdwcUd5zr1PMqTAPCOJ3tbNsYleD+Ecyjive4kXPOIt07AjaCWoUDfQu9GaoHKdXWC69HkbE+Nzv+MFXTM8AYTzto0yAgv217/elcHz5hveLjgrtndNw1wttIsL50yTH07pA/aOPsKoCjDNyaWjtGZVQd35dp//tYk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IaWqHuSX; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IaWqHuSX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09D471F00A3D; Fri, 22 May 2026 14:23:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1779459821; bh=4b73HeUFL1BJFb9tKUFnj3I3fQCi1XJtQNXmwPghYI8=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=IaWqHuSXUgpMe4/lDFIwXOEhByFwpTQJaGOyoVuTTXq7iBljnkVbkJHrK3k1w1z8a iWq1g9DpzaWmI8n5ccngszzrJqVorI6HR8jmW0wmeDAMfMu3rO8d+QFbcbZwbbYap7 MZYSPhnjKzv2J6o2oA1R2rsn+ob2bQdl4Od5BvSDe87J+19Ke/4pngaa+9xZ3Hh6yS nkOMARNQXcCIGZyVXflVbpThl6fFscdog6OXvf64ecEJvNNJIi6dgfsA6MWA3PgLaP 30jl0Zrrvyngu8BjJh4T52YzUbi6H/TyRO5aMRfjXxiaqps/bFyAizuyxsT3okgZRG gZJwDS9T7TjaQ== From: "Vlastimil Babka (SUSE)" Date: Fri, 22 May 2026 16:23:20 +0200 Subject: [PATCH v3 1/2] mm, slab: add an optimistic __slab_try_return_freelist() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260522-b4-refill-optimistic-return-v3-1-2ba78ec1c6ed@kernel.org> References: <20260522-b4-refill-optimistic-return-v3-0-2ba78ec1c6ed@kernel.org> In-Reply-To: <20260522-b4-refill-optimistic-return-v3-0-2ba78ec1c6ed@kernel.org> To: Harry Yoo Cc: Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, hu.shengming@zte.com.cn, Vinicius Costa Gomes , "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=5302; i=vbabka@kernel.org; h=from:subject:message-id; bh=4r4E8fF8xlQ2gCDGhw11J9DZ3L6ugESsDuH1k5p0Uwc=; b=owGbwMvMwMG4+8GG0kuuHbMYT6slMWQJpD1Ku5u7zqSOO09x126BR3/k6jPvtQpVcZWV93Z53 8pv8VDtZPRnYWDkYLAUU2Sp3n3CUXSmssc0D9+PMINYmUCmSIs0MAABCwNfbmJeqZGOkZ6ptqGe IZChY8TAxSkAU13Syv6/4MLRy/3pejvmnVy5gF2SZeaR2Oov/Tvu+9jnrTHSyJeyNHETbPoYe6n xkY7Pu7X8U46lLc7XDhFVLH5/XH3hJxUfccG2cr64+SULO5cIM8zj8eb4EHxFl2H7ggqmaZzsWX n3rLbZRy0+vLNV1DKUZQ/LTd7+OaqiZ1UKeVyvBy7NvOCdt+HXikOHw/ZUxkjlPHyhv4NPYEEQr 7Kf9/y98w4Hv1op7VN9Sz5lI+8GqTeCt6ZcL/V8+rpUxNNX++65r5sjlj9SWbLnsMi3rn/R2YnP +5mlCmOniqXE8iycmHB0laOCCUPxY2bxAoXFEe9vKXQkCF/tOi519My785omRfuPlG1LkAk+cOB /yEQA X-Developer-Key: i=vbabka@kernel.org; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 When we end up returning extraneous objects during refill to a slab where we just did a get_freelist_nofreeze(), it is likely no other CPU has freed objects to it meanwhile. We can then reattach the remainder of the freelist without having to walk the (potentially cache cold) freelist for finding its tail to connect slab->freelist to it. Add a __slab_try_return_freelist() function that does that. As suggested by Hao Li, it doesn't need to also return the slab to the partial list, because there's code in __refill_objects_node() that already does that for any slabs where we don't detach the freelist in the first place. So we just put the slab back to the pc.slabs list. It's no longer likely that the list will be empty now, so remove the unlikely() annotation. However, also change that code to add to the tail of the partial list instead of head to match what __slab_free() did and avoid a regression, that was reported for the earlier version by the kernel test robot [1]. This change will also affect slabs which were grabbed from the partial list and not refilled from even partially, but those should be much more rare than a partial refill. [1] https://lore.kernel.org/all/202605112204.9382cecf-lkp@intel.com/ Reviewed-by: Hao Li Signed-off-by: Vlastimil Babka (SUSE) --- mm/slub.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index a7bd929bcb26..5816fcfc7a90 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4326,7 +4326,8 @@ static inline bool pfmemalloc_match(struct slab *slab= , gfp_t gfpflags) * Assumes this is performed only for caches without debugging so we * don't need to worry about adding the slab to the full list. */ -static inline void *get_freelist_nofreeze(struct kmem_cache *s, struct sla= b *slab) +static inline void *get_freelist_nofreeze(struct kmem_cache *s, struct sla= b *slab, + unsigned int *count) { struct freelist_counters old, new; =20 @@ -4342,6 +4343,7 @@ static inline void *get_freelist_nofreeze(struct kmem= _cache *s, struct slab *sla =20 } while (!slab_update_freelist(s, slab, &old, &new, "get_freelist_nofreez= e")); =20 + *count =3D old.objects - old.inuse; return old.freelist; } =20 @@ -5509,6 +5511,34 @@ static noinline void free_to_partial_list( } } =20 +/* + * Try returning (remainder of) the freelist that we just detached from the + * slab. Optimistically assume the slab is still full, so we don't need t= o find + * the tail of the detached freelist. + * + * Fail if the slab isn't full anymore due to a cocurrent free. + */ +static bool __slab_try_return_freelist(struct kmem_cache *s, struct slab *= slab, + void *head, int cnt) +{ + struct freelist_counters old, new; + + old.freelist =3D slab->freelist; + old.counters =3D slab->counters; + + if (old.freelist) + return false; + + new.freelist =3D head; + new.counters =3D old.counters; + new.inuse -=3D cnt; + + if (!slab_update_freelist(s, slab, &old, &new, "__slab_try_return_freelis= t")) + return false; + + return true; +} + /* * Slow path handling. This may still be called frequently since objects * have a longer lifetime than the cpu slabs in most processing loads. @@ -7120,41 +7150,48 @@ __refill_objects_node(struct kmem_cache *s, void **= p, gfp_t gfp, unsigned int mi =20 list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) { =20 + unsigned int count; + list_del(&slab->slab_list); =20 - object =3D get_freelist_nofreeze(s, slab); + object =3D get_freelist_nofreeze(s, slab, &count); =20 - while (object && refilled < max) { + while (count && refilled < max) { p[refilled] =3D object; object =3D get_freepointer(s, object); maybe_wipe_obj_freeptr(s, p[refilled]); =20 refilled++; + count--; } =20 /* * Freelist had more objects than we can accommodate, we need to - * free them back. We can treat it like a detached freelist, just - * need to find the tail object. + * free them back. First we try to be optimistic and assume the + * slab is stil full since we just detached its freelist. + * Otherwise we must need to find the tail object. */ - if (unlikely(object)) { + if (unlikely(count)) { void *head =3D object; void *tail; - int cnt =3D 0; + + if (__slab_try_return_freelist(s, slab, head, count)) { + list_add(&slab->slab_list, &pc.slabs); + break; + } =20 do { tail =3D object; - cnt++; object =3D get_freepointer(s, object); } while (object); - __slab_free(s, slab, head, tail, cnt, _RET_IP_); + __slab_free(s, slab, head, tail, count, _RET_IP_); } =20 if (refilled >=3D max) break; } =20 - if (unlikely(!list_empty(&pc.slabs))) { + if (!list_empty(&pc.slabs)) { spin_lock_irqsave(&n->list_lock, flags); =20 list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) { @@ -7163,7 +7200,7 @@ __refill_objects_node(struct kmem_cache *s, void **p,= gfp_t gfp, unsigned int mi continue; =20 list_del(&slab->slab_list); - add_partial(n, slab, ADD_TO_HEAD); + add_partial(n, slab, ADD_TO_TAIL); } =20 spin_unlock_irqrestore(&n->list_lock, flags); --=20 2.54.0 From nobody Sun May 24 19:33:20 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A03636606A for ; Fri, 22 May 2026 14:23:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779459825; cv=none; b=IVrB17kccslLmgne30bWAGqL948HSKHmuEFay8LiFs6Pc/5FgcU6qCSlk0TWr7VRQJ0LqpT0n1348RdqzGzn6dizLOf5cbZ3IE9If4Vo/aJI608g1feYJp6Cq7MxeKDqIRv+8RF69qL80LxYC0zmx07VdOufftcQ5t4vZL79bq8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779459825; c=relaxed/simple; bh=5xU7bvNWPz0EcVzac20UtCjNot5RIW9j5Mp9rjIj3Ow=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=myuINjw9ZP6HWvC/FqTQ5nGNrZ2/qeWMIE3tf91vpFENzg9QozMcCWTVEE88zf0cvshwauvue3s6u8inVKCkNEoJVzPTHgzL+P4B/HTZ7v3qokZ36+mNAHqz+1oLHf5yAA/UN+88uTn37HSeiWc1iLtCYPw3mNc3gtPWHIzuGEo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hbkvlrx+; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hbkvlrx+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8712E1F00A3E; Fri, 22 May 2026 14:23:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1779459823; bh=vNUnwd0Z1xsF+oZJGcRcFicg2hDM3WcwB/9gZhQZ+wk=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=hbkvlrx+xq2crqJZLJ8xlwsNZcEtQ7vIF/I3PYTemtAs3R1Gquyl5NdzyrbocQTo6 eRX4jdQdt+fcIVAm8Ibaqi/wHJ43w1bYBgmKv88TV4yuUx9qGLwrZyFLEV8xtcVmab XPcxZRpfYTaASTlNmAPEmxzNLy7aE2ZPbyldH/zgp3OQZM1pYJfle5FokVDdtHAQmY 5XD7UBP3RQ3EDaKEPmSZLlIMtrgHCmJuZE/8dI2LVzDE0SZnCBY+icVKORIIjitSP9 raJ2RchT6+8RL0HizNZlLwm7g9WAGnr43dDRIqg01EmgqIpdbKpJVu6FM8ioJv4VdB 2Oo4ZYAAYqZ9Q== From: "Vlastimil Babka (SUSE)" Date: Fri, 22 May 2026 16:23:21 +0200 Subject: [PATCH v3 2/2] mm, slab: simplify returning slab in __refill_objects_node() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260522-b4-refill-optimistic-return-v3-2-2ba78ec1c6ed@kernel.org> References: <20260522-b4-refill-optimistic-return-v3-0-2ba78ec1c6ed@kernel.org> In-Reply-To: <20260522-b4-refill-optimistic-return-v3-0-2ba78ec1c6ed@kernel.org> To: Harry Yoo Cc: Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, hu.shengming@zte.com.cn, Vinicius Costa Gomes , "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=1282; i=vbabka@kernel.org; h=from:subject:message-id; bh=5xU7bvNWPz0EcVzac20UtCjNot5RIW9j5Mp9rjIj3Ow=; b=owGbwMvMwMG4+8GG0kuuHbMYT6slMWQJpD19dNRAbVL2OX3jTvaMuGjjO1XlVuaXut63X89ne 5IiIn2rk9GfhYGRg8FSTJGlevcJR9GZyh7TPHw/wgxiZQKZIi3SwAAELAx8uYl5pUY6Rnqm2oZ6 hkCGjhEDF6cATHWIKPtvVp09K3OqG7SO6R+pzfKMbD+2/0YmF3t9/PbIdbcVbn9Xzpz+VXXlj+e Lj3P80d18PEQ8VVH45Am5d9e6VDtXzRT4Z3jlzQ4hl+LuqWm8rStM01MakjgPH1yy8mDnHGumuF n9FZt3LNXgf9h3Se1RNNddsf/NxzVj7Z5z10Zx3gqt3eK5qrkxUuOJepf70X+6hS2Xrx9+9zjm2 /uQvKN7X6r1z3yw3ld3j5Fi/frzaz0euMx1sNn9wUuBnVHN1lYp4ZKOR+liMz97A9d/Yiv7ay5K mNwTilrHOyubUWOWRUHGHqeAR1zTa3Ucn22o+5Kmfc5V5fwXR++MVTtuntuhtLL6osfsSRXbdDW /vN8PAA== X-Developer-Key: i=vbabka@kernel.org; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 When we return slabs to the partial list because we didn't fully refill from them, we observe the min_partial limit when the returned slab is empty, and discard it when over the limit. But it's unlikely for the limit to be reached while we were refilling, and the worst outcome is to have temporarily more free slabs on the list than necessary. So just drop that code and simplify the function. Signed-off-by: Vlastimil Babka (SUSE) --- mm/slub.c | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 5816fcfc7a90..074d57c0390b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -7196,21 +7196,11 @@ __refill_objects_node(struct kmem_cache *s, void **= p, gfp_t gfp, unsigned int mi =20 list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) { =20 - if (unlikely(!slab->inuse && n->nr_partial >=3D s->min_partial)) - continue; - list_del(&slab->slab_list); add_partial(n, slab, ADD_TO_TAIL); } =20 spin_unlock_irqrestore(&n->list_lock, flags); - - /* any slabs left are completely free and for discard */ - list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) { - - list_del(&slab->slab_list); - discard_slab(s, slab); - } } =20 return refilled; --=20 2.54.0