From nobody Tue Feb 10 00:44:34 2026 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E689307AC3 for ; Tue, 2 Dec 2025 09:00:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764666024; cv=none; b=Uh7Ygi13zoScy6nwD5GuP61FIrWh91UdE3oJvLs/yE7I+M8/XX+5Sgam9EVDXOMyNPl/lpEi3uHn1Um93CONeMzL/PmfAdObx7ta7/xW3F8gepRCLNqL0Nmo63PiUsWqzfhfOiG9RTEJHFLfCBcWXsoQVT95zXXAhI4o4TYGZfc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764666024; c=relaxed/simple; bh=6U0wc5j7itDMrkk9cvcAeQZkYbCa1ocpdtIssIpDS5k=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=glv0FrPqmrwZ1hu/Paxm8mSrYV9O06oxGeH+NGvr//mlvkNpcTNwEUMwMHqkJ2WEsqDTBXRRxgbyND6nsB0hCWVbBWj0lTj4B4uZP32Ed82dJRRdJwnS8fp9UWeClofSPd+2stL3dIie+xR/AajmG5l/50VLHfu/CNB1slLp5hQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=grMAoV1/; arc=none smtp.client-ip=209.85.210.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="grMAoV1/" Received: by mail-pf1-f194.google.com with SMTP id d2e1a72fcca58-7ba92341f83so7908955b3a.0 for ; Tue, 02 Dec 2025 01:00:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764666021; x=1765270821; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=PDtCgcKoqebXyo+jxENam+HXtRH1ghw4HvSFLZRxozg=; b=grMAoV1/0CN5JyiyEB8PhZk8cWMvAd7TJqWOdHjHs5DQovH52GT5X7/erJFNnmHUij hbBgGT5IaUmTi6BIHv9OIKcgw4B1rDigQu43Iv9o0kersd85zh9nM3NOVLxNKrPL28J5 ECaD7/keElej8CyrAiexFNNFfeU+mG4uBOWBfpPsQnIG13lw94hK2PnsuwLcdQL2ekzV 3T+U1K+QaWKOZ5zlIfxTk2C44BtNiTV0id99w/OfX6SKoX+EWnGY4P0ue1qprQ0vybuo XtwNPRDM73pxPNUPBlHAlmWCZnCAjIsuoTSOnosH/0D3jTSVAUDcMdM+OnVhq+gLkmOd aIJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764666021; x=1765270821; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PDtCgcKoqebXyo+jxENam+HXtRH1ghw4HvSFLZRxozg=; b=q51ZsSzbLpRoN3Ou3fsiFTUbZYiCWyb137s9BFMq2xFOxeLjxprNzzyNhfo4bLMhFY 7mnMhTDE7VKbYnAvY3t0AOYjHXYlP93QHjJIIUb0TmjY30CZ2bwOAOdEMiEFIs2jc6c2 h/5VlRlTEwpO1oKaClFAqRzUFgzzMAjnTp4HPGjUmQkXDwoRNNXSguv+DjBeTtdWG/PM YRfhCZOGUf1jAavtLKp17ZZdusmB11xEcwOR96gu4lABq8uX2t1RDi5BXum6QbJVr5ue JZbxuX7eu0mjlxIwrE11DZPeB1itUMXprZBAKjJa1yJCEet944w4vV4UqMGi8UMBfzmH 7O2w== X-Forwarded-Encrypted: i=1; AJvYcCXYUZBTxHF27GMy/ea5cKC1gaWKXDDGSM1PTKU0pfKQuCgSX/Gvrq4lo152JjIpA9U+73hTlwXgJSguJso=@vger.kernel.org X-Gm-Message-State: AOJu0YzvQgOqSw6THty6t2I7rX/sioN5BLhjCqV4pT9QhsuoGIJe4mDx kJ3UiLQ7xUYUNBPBvb1BtzB4RvB9hN2alAMbvKeezsH3dH79Ze2lp5Vn X-Gm-Gg: ASbGnctsllFRdtnr8QKKB2DfeBOjl0s4MgQzDtRfUs2rEb3biZg0fep5BSXzpw+hVex szZ6w1xzu6FVWBSt/f8C8qCtasSd8Tu1mxThqiOksDWlcj+i2f9uiIF+wkiVoE5GYuhjzQLSxQ/ VkJqpyZ8ngt2crbSAqyhKuiWSMp8Zl87tASBxScPJe7guNDQIfezQLNscDlUm2m93I46nq20NZN XJNm+GZTm4bIwM0lf+pw3LDsxhgqO9c9R0GHO0QWr9icRpK8ownunAx8Ha9eJVv35A63cwQE9px 9KcF4zbx7kPwbP7OdoEO90ABlEEwcjSA/WA0KsU5Of4uAGANJtx07J+kaoLd8LI1+k2v1El4k3l kE0ER5586gLgfax17LwVajXGWmRiOl3VrwCsFuolOwazheziEBbw5E0AZAZuuUtoEsoYVXwU4OC yuV/hr X-Google-Smtp-Source: AGHT+IGBTBGTbAUOxAz4LdYDBbQj7h1lJpYS6bSIIDqI1Q6AFazDjtAmDH/mUQRFh29VnLmHXujLxA== X-Received: by 2002:a05:6a20:548d:b0:35d:3523:c90d with SMTP id adf61e73a8af0-3637d8d6789mr32627334637.0.1764666021403; Tue, 02 Dec 2025 01:00:21 -0800 (PST) Received: from fedora ([183.241.171.104]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7d150b68367sm16385575b3a.12.2025.12.02.01.00.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Dec 2025 01:00:21 -0800 (PST) Date: Tue, 2 Dec 2025 17:00:08 +0800 From: Hao Li To: Vlastimil Babka Cc: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo , Uladzislau Rezki , Sidhartha Kumar , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Venkat Rao Bagalkote Subject: slub: add barn_get_full_sheaf() and refine empty-main sheaf replacement Message-ID: References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz> Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce barn_get_full_sheaf(), a helper that detaches a full sheaf from the per-node barn without requiring an empty sheaf in exchange. Use this helper in __pcs_replace_empty_main() to change how an empty main per-CPU sheaf is handled: - If pcs->spare is NULL and pcs->main is empty, first try to obtain a full sheaf from the barn via barn_get_full_sheaf(). On success, park the empty main sheaf in pcs->spare and install the full sheaf as the new pcs->main. - If pcs->spare already exists and has objects, keep the existing behavior of simply swapping pcs->main and pcs->spare. - Only when both pcs->main and pcs->spare are empty do we fall back to barn_replace_empty_sheaf() and trade the empty main sheaf into the barn in exchange for a full one. This makes the empty-main path more symmetric with __pcs_replace_full_main(= ), which for a full main sheaf parks the full sheaf in pcs->spare and pulls an empty sheaf from the barn. It also matches the documented design more close= ly: "When both percpu sheaves are found empty during an allocation, an empty sheaf may be replaced with a full one from the per-node barn." Signed-off-by: Hao Li --- * This patch is based on b4/sheaves-for-all branch mm/slub.c | 50 +++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 43 insertions(+), 7 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index a94c64f56504..1fd28aa204e1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2746,6 +2746,32 @@ static void pcs_destroy(struct kmem_cache *s) s->cpu_sheaves =3D NULL; } =20 +static struct slab_sheaf *barn_get_full_sheaf(struct node_barn *barn, + bool allow_spin) +{ + struct slab_sheaf *full =3D NULL; + unsigned long flags; + + if (!data_race(barn->nr_full)) + return NULL; + + if (likely(allow_spin)) + spin_lock_irqsave(&barn->lock, flags); + else if (!spin_trylock_irqsave(&barn->lock, flags)) + return NULL; + + if (likely(barn->nr_full)) { + full =3D list_first_entry(&barn->sheaves_full, + struct slab_sheaf, barn_list); + list_del(&full->barn_list); + barn->nr_full--; + } + + spin_unlock_irqrestore(&barn->lock, flags); + + return full; +} + static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn, bool allow_spin) { @@ -4120,7 +4146,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct= slub_percpu_sheaves *pcs, struct slab_sheaf *empty =3D NULL; struct slab_sheaf *full; struct node_barn *barn; - bool can_alloc; + bool can_alloc, allow_spin; =20 lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); =20 @@ -4130,10 +4156,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struc= t slub_percpu_sheaves *pcs, return NULL; } =20 - if (pcs->spare && pcs->spare->size > 0) { - swap(pcs->main, pcs->spare); - return pcs; - } + allow_spin =3D gfpflags_allow_spinning(gfp); =20 barn =3D get_barn(s); if (!barn) { @@ -4141,8 +4164,21 @@ __pcs_replace_empty_main(struct kmem_cache *s, struc= t slub_percpu_sheaves *pcs, return NULL; } =20 - full =3D barn_replace_empty_sheaf(barn, pcs->main, - gfpflags_allow_spinning(gfp)); + if (!pcs->spare) { + full =3D barn_get_full_sheaf(barn, allow_spin); + if (full) { + pcs->spare =3D pcs->main; + pcs->main =3D full; + return pcs; + } + } else if (pcs->spare->size > 0) { + swap(pcs->main, pcs->spare); + return pcs; + } + + /* both main and spare are empty */ + + full =3D barn_replace_empty_sheaf(barn, pcs->main, allow_spin); =20 if (full) { stat(s, BARN_GET); --=20 2.50.1