From nobody Fri Oct 3 08:51:18 2025 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8539A301027 for ; Wed, 3 Sep 2025 13:00:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756904445; cv=none; b=qjh53EcAlQvuS0GhXgbS1/iBXLZMB6z+Q2AFenXZWmmP1n/WIiq8vwKZfSaovfmzQgI/BhuwjPSRObHSMwrORuMjg/DBPJ/Pys8x1+fvPDzFqT6rJFg+bPbWmgB+qBubAXi/uD+jjg5QFVYw58kmP9FtlBDYnystbxJgJJueY08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756904445; c=relaxed/simple; bh=7CCSDJsDSo53rJ2Xve2xGVYoT/NsP8GXnfH4mSKfrok=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ufZzENv0DM6SV0Non8HtZOxx2CU3aVyQCYIgjSCAub8V0C+wrvuVz3i3duu/dok8uwvWtn/UAnU5Ra1R5aIPDjxR5PMAKgLVAnEamXaYlHw7xhtKVPxF70NH07FvObhJv0mYzLOufvPODqjHWIw3PC+ZBV2iNqUTRkrB8ULXAD0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=LoETShIM; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=MyQghC5D; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=LoETShIM; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=MyQghC5D; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="LoETShIM"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="MyQghC5D"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="LoETShIM"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="MyQghC5D" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 4F49F21227; Wed, 3 Sep 2025 13:00:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1756904410; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvueOpBRqUSJBVmPF3IuGLVWneYsx8iJYPivjgPVl+E=; b=LoETShIMk0pCx5L816uJyvFy7c4G6hgcTgDNeP0wdw3yv0H1gKUdk/cqAYnCOz9H7IfMBQ HDr9bFcRAknhxCqHENhCd3m3YzDFv71MZmJZCdVgC1HKbU0Y75RhdAXAhziIlsleurg51T VNyNCxItm6H6usKuYzzlrfROk7g2Vzw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1756904410; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvueOpBRqUSJBVmPF3IuGLVWneYsx8iJYPivjgPVl+E=; b=MyQghC5DsjfSLaNXl3OCffrJh1mZIzymp5cPWwwkaVoofGSGp5XZAZ+nQGdVjDXWQmPKlF PfA/Q+4MQ1QPxkDQ== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=LoETShIM; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=MyQghC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1756904410; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvueOpBRqUSJBVmPF3IuGLVWneYsx8iJYPivjgPVl+E=; b=LoETShIMk0pCx5L816uJyvFy7c4G6hgcTgDNeP0wdw3yv0H1gKUdk/cqAYnCOz9H7IfMBQ HDr9bFcRAknhxCqHENhCd3m3YzDFv71MZmJZCdVgC1HKbU0Y75RhdAXAhziIlsleurg51T VNyNCxItm6H6usKuYzzlrfROk7g2Vzw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1756904410; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvueOpBRqUSJBVmPF3IuGLVWneYsx8iJYPivjgPVl+E=; b=MyQghC5DsjfSLaNXl3OCffrJh1mZIzymp5cPWwwkaVoofGSGp5XZAZ+nQGdVjDXWQmPKlF PfA/Q+4MQ1QPxkDQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8B0FB13AFB; Wed, 3 Sep 2025 13:00:09 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id YPyuIdk7uGitOAAAD6G6ig (envelope-from ); Wed, 03 Sep 2025 13:00:09 +0000 From: Vlastimil Babka Date: Wed, 03 Sep 2025 14:59:48 +0200 Subject: [PATCH v7 06/21] slab: determine barn status racily outside of lock Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250903-slub-percpu-caches-v7-6-71c114cdefef@suse.cz> References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz> In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Harry Yoo , Uladzislau Rezki , Sidhartha Kumar , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, vbabka@suse.cz X-Mailer: b4 0.14.2 X-Spam-Level: X-Spam-Flag: NO X-Rspamd-Queue-Id: 4F49F21227 X-Rspamd-Action: no action X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.51 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; FUZZY_RATELIMITED(0.00)[rspamd.com]; ARC_NA(0.00)[]; RCPT_COUNT_TWELVE(0.00)[13]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; FREEMAIL_CC(0.00)[linux.dev,oracle.com,gmail.com,kvack.org,vger.kernel.org,lists.infradead.org,suse.cz]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RECEIVED_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:106:10:150:64:167:received]; DKIM_TRACE(0.00)[suse.cz:+]; R_RATELIMIT(0.00)[to_ip_from(RLfsjnp7neds983g95ihcnuzgq)]; DBL_BLOCKED_OPENRESOLVER(0.00)[oracle.com:email,suse.cz:dkim,suse.cz:mid,suse.cz:email,imap1.dmz-prg2.suse.org:rdns,imap1.dmz-prg2.suse.org:helo] X-Spam-Score: -4.51 The possibility of many barn operations is determined by the current number of full or empty sheaves. Taking the barn->lock just to find out that e.g. there are no empty sheaves results in unnecessary overhead and lock contention. Thus perform these checks outside of the lock with a data_race() annotated variable read and fail quickly without taking the lock. Checks for sheaf availability that racily succeed have to be obviously repeated under the lock for correctness, but we can skip repeating checks if there are too many sheaves on the given list as the limits don't need to be strict. Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo Signed-off-by: Vlastimil Babka --- mm/slub.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index c6ca9b60acd15520410ac08d252bb09e111db6f1..d078630f7211b617f390a16fa08= 0da8a1df45355 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2801,9 +2801,12 @@ static struct slab_sheaf *barn_get_empty_sheaf(struc= t node_barn *barn) struct slab_sheaf *empty =3D NULL; unsigned long flags; =20 + if (!data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); =20 - if (barn->nr_empty) { + if (likely(barn->nr_empty)) { empty =3D list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); @@ -2850,6 +2853,9 @@ static struct slab_sheaf *barn_get_full_or_empty_shea= f(struct node_barn *barn) struct slab_sheaf *sheaf =3D NULL; unsigned long flags; =20 + if (!data_race(barn->nr_full) && !data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); =20 if (barn->nr_full) { @@ -2880,9 +2886,12 @@ barn_replace_empty_sheaf(struct node_barn *barn, str= uct slab_sheaf *empty) struct slab_sheaf *full =3D NULL; unsigned long flags; =20 + if (!data_race(barn->nr_full)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); =20 - if (barn->nr_full) { + if (likely(barn->nr_full)) { full =3D list_first_entry(&barn->sheaves_full, struct slab_sheaf, barn_list); list_del(&full->barn_list); @@ -2906,19 +2915,23 @@ barn_replace_full_sheaf(struct node_barn *barn, str= uct slab_sheaf *full) struct slab_sheaf *empty; unsigned long flags; =20 + /* we don't repeat this check under barn->lock as it's not critical */ + if (data_race(barn->nr_full) >=3D MAX_FULL_SHEAVES) + return ERR_PTR(-E2BIG); + if (!data_race(barn->nr_empty)) + return ERR_PTR(-ENOMEM); + spin_lock_irqsave(&barn->lock, flags); =20 - if (barn->nr_full >=3D MAX_FULL_SHEAVES) { - empty =3D ERR_PTR(-E2BIG); - } else if (!barn->nr_empty) { - empty =3D ERR_PTR(-ENOMEM); - } else { + if (likely(barn->nr_empty)) { empty =3D list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); list_add(&full->barn_list, &barn->sheaves_full); barn->nr_empty--; barn->nr_full++; + } else { + empty =3D ERR_PTR(-ENOMEM); } =20 spin_unlock_irqrestore(&barn->lock, flags); --=20 2.51.0