From nobody Mon Feb 9 14:02:28 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 371CF1D31B9 for ; Fri, 7 Nov 2025 13:52:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523526; cv=none; b=OyS7U2pWAUDXfmDCN/+Za1hGsHzL4xzFxWir+LhvytDnVxGdEE4Y/qNkasDh9JNJ1DAep9dT84ienARZtWFTOGQTKAHsMpugghiFlcczdvCahNMa3IcWTWHbq/a2tEFvI0Tzbc7EdSIyQpygtIFKNHfAHNwgSYDVGJBfClxpagM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523526; c=relaxed/simple; bh=hSWEdMLovcP9bQkB7Wk8xHcoJA3zCHtq+wyLjs21pmI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mgYruuENNhU567qbDhwk9uMRqzNCzFELO/gTD66DWCu9ehMhTZA59UQ9wcEeRSDZ+h5F1OcisojyMvQYSutPvs5a6UueIlUxSFvRuqbfeFDV5Z1L+rgntwol3Cacv+8VU1eVF7uth/uSVOMziMaBf7XL/lT1na16PU52w/xGvZs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=fYxK1G/W; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=TGWKXhah; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="fYxK1G/W"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="TGWKXhah" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 93CCC1F395; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DpbVwPnigYZBYPBD0v1fEcmXhvJqKrRei+EvKd6N3nY=; b=fYxK1G/WAs29MTQdyG4dByVDgnPXCPDtfebWTTPlRWgieWXFzAFXw5gmwUHNimGlMgfOTV 2r86Q1/yrh36ncnuN6hV56lyUwNj9a1voTpQbsiA4laNyWs7Jdf+SgWB+79gG3jcoLEEqq tf7sjPmO3wXCl8lYtteBzP0il4wq4RU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DpbVwPnigYZBYPBD0v1fEcmXhvJqKrRei+EvKd6N3nY=; b=TGWKXhahJY6XQ/0Q8hvOY1jKzklVoLltGsdlgtHSsy3J5hkAgZl1Rib1VN2cDJP4cAnpGI 7BCDMGm4DtQ4hzBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8053513A60; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id yP8JH3L5DWlFCwAAD6G6ig (envelope-from ); Fri, 07 Nov 2025 13:51:46 +0000 From: Vlastimil Babka Date: Fri, 07 Nov 2025 14:51:23 +0100 Subject: [PATCH 1/4] slab: separate struct freelist_tid from kmem_cache_cpu Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251107-slab-fms-cleanup-v1-1-650b1491ac9e@suse.cz> References: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> In-Reply-To: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo , linux-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.3 X-Rspamd-Queue-Id: 93CCC1F395 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Spam-Flag: NO X-Spam-Score: -4.00 X-Spam-Level: X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[] In kmem_cache_cpu we currently have a union of the freelist+tid pair with freelist_aba_t, relying implicitly on the type compatibility with the freelist+counters pair used in freelist_aba_t. To allow further changes to freelist_aba_t, we can instead define a separate struct freelist_tid (instead of a typedef, per the coding style) for kmem_cache_cpu, as that affects only a single helper __update_cpu_freelist_fast(). We can add the resulting struct freelist_tid to kmem_cache_cpu as unnamed field thanks to -fms-extensions, so that freelist and tid fields can still be accessed directly. Signed-off-by: Vlastimil Babka Reviewed-by: Harry Yoo --- mm/slub.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 074abe8e79f8..5f6408c9e0fd 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -411,18 +411,22 @@ enum stat_item { }; =20 #ifndef CONFIG_SLUB_TINY -/* - * When changing the layout, make sure freelist and tid are still compatib= le - * with this_cpu_cmpxchg_double() alignment requirements. - */ -struct kmem_cache_cpu { +struct freelist_tid { union { struct { - void **freelist; /* Pointer to next available object */ + void *freelist; /* Pointer to next available object */ unsigned long tid; /* Globally unique transaction id */ }; - freelist_aba_t freelist_tid; + freelist_full_t freelist_tid; }; +}; + +/* + * When changing the layout, make sure freelist and tid are still compatib= le + * with this_cpu_cmpxchg_double() alignment requirements. + */ +struct kmem_cache_cpu { + struct freelist_tid; struct slab *slab; /* The slab from which we are allocating */ #ifdef CONFIG_SLUB_CPU_PARTIAL struct slab *partial; /* Partially allocated slabs */ @@ -4367,11 +4371,11 @@ __update_cpu_freelist_fast(struct kmem_cache *s, void *freelist_old, void *freelist_new, unsigned long tid) { - freelist_aba_t old =3D { .freelist =3D freelist_old, .counter =3D tid }; - freelist_aba_t new =3D { .freelist =3D freelist_new, .counter =3D next_ti= d(tid) }; + struct freelist_tid old =3D { .freelist =3D freelist_old, .tid =3D tid }; + struct freelist_tid new =3D { .freelist =3D freelist_new, .tid =3D next_t= id(tid) }; =20 - return this_cpu_try_cmpxchg_freelist(s->cpu_slab->freelist_tid.full, - &old.full, new.full); + return this_cpu_try_cmpxchg_freelist(s->cpu_slab->freelist_tid, + &old.freelist_tid, new.freelist_tid); } =20 /* --=20 2.51.1 From nobody Mon Feb 9 14:02:28 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4A991D31B9 for ; Fri, 7 Nov 2025 13:51:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523516; cv=none; b=EMjadKBMSh+A5AFTnE5PeWllauiijudyvl5PlGXmzShl+Z8DBSOj5BuAiP82QzoiTKH0Z8SSZlToXuhZM8yFBIa7iqXI6s+kB9ZiAZmnf/hLwiaZgYAyDxd7rYZa1Wpi0e6mLrCuB89PJWgIPWXbyd3Q2OJiyWsYM1myI4b9Ymg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523516; c=relaxed/simple; bh=zOhrY5GaYQzKSZYFPVRNOLzmbILZydfiFpKWaRoudBI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=aaMHpOCzY/5a3WvRYp/nkB8nAUqx/NqlMgL++TwG3t6flsSnaZKB39AMY84BlngIZ37dGDpWYmuyF4XA9iTFuVcdlm1Z56K4sUJEpR/Qm4/kjMsww/YQvoNgPln/1SpuZ7gpoNXRSKw93XI76V13yL5F56KS8gmLxTe8v2yzSZ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=hv/VOq3E; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=IcTGpFhS; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="hv/VOq3E"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="IcTGpFhS" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A28711F457; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7qvlsbcBZ089j/NzKsFpZxKXLw4TpiqdOIdH9JEv6Go=; b=hv/VOq3EaIjugnpTHlyuexxWXtm8g/2kpUkHkEFcLqzbntwk2VQbs/0UADiHRwjhEtXgVe 9m0saOZMtKmuS66+fWVSxLn8YvoHdn0DqSwX8bkqgKhL1V+eDBzv9HHlmxCsDMi4/Rb8i2 fNLiaPvX+3KLs5VK5onNE6RPbRYJnMI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7qvlsbcBZ089j/NzKsFpZxKXLw4TpiqdOIdH9JEv6Go=; b=IcTGpFhS0Io25VB39gRKl5Bm2aB2cTMdBHcJp96/4WXPT1fzF1fsjhk6rcG35ZFRNzav70 D1mjH+OKDBcGa2Cw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 8E64F13A67; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 8C5zInL5DWlFCwAAD6G6ig (envelope-from ); Fri, 07 Nov 2025 13:51:46 +0000 From: Vlastimil Babka Date: Fri, 07 Nov 2025 14:51:24 +0100 Subject: [PATCH 2/4] slab: turn freelist_aba_t to a struct and fully define counters there Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251107-slab-fms-cleanup-v1-2-650b1491ac9e@suse.cz> References: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> In-Reply-To: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo , linux-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.3 X-Spam-Level: X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[] X-Rspamd-Queue-Id: A28711F457 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Spam-Flag: NO X-Spam-Score: -4.00 In struct slab we currently have freelist and counters pair, where counters itself is a union of unsigned long with a sub-struct of several smaller fields. Then for the usage with double cmpxchg we have freelist_aba_t that duplicates the definition of the freelist+counters with implicitly the same layout as the full definition in struct slab. Thanks to -fms-extension we can now move the full counters definition to freelist_aba_t (while changing it to struct freelist_counters as a typedef is unnecessary and discouraged) and replace the relevant part in struct slab to an unnamed reference to it. The immediate benefit is the removal of duplication and no longer relying on the same layout implicitly. It also allows further cleanups thanks to having the full definition of counters in struct freelist_counters. Signed-off-by: Vlastimil Babka Reviewed-by: Harry Yoo --- mm/slab.h | 52 ++++++++++++++++++++++++---------------------------- mm/slub.c | 8 +++++--- 2 files changed, 29 insertions(+), 31 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 078daecc7cf5..42627b87d50c 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -40,13 +40,29 @@ typedef u64 freelist_full_t; * Freelist pointer and counter to cmpxchg together, avoids the typical ABA * problems with cmpxchg of just a pointer. */ -typedef union { - struct { - void *freelist; - unsigned long counter; +struct freelist_counters { + union { + struct { + void *freelist; + union { + unsigned long counters; + struct { + unsigned inuse:16; + unsigned objects:15; + /* + * If slab debugging is enabled then the + * frozen bit can be reused to indicate + * that the slab was corrupted + */ + unsigned frozen:1; + }; + }; + }; +#ifdef system_has_freelist_aba + freelist_full_t freelist_counters; +#endif }; - freelist_full_t full; -} freelist_aba_t; +}; =20 /* Reuses the bits in struct page */ struct slab { @@ -69,27 +85,7 @@ struct slab { #endif }; /* Double-word boundary */ - union { - struct { - void *freelist; /* first free object */ - union { - unsigned long counters; - struct { - unsigned inuse:16; - unsigned objects:15; - /* - * If slab debugging is enabled then the - * frozen bit can be reused to indicate - * that the slab was corrupted - */ - unsigned frozen:1; - }; - }; - }; -#ifdef system_has_freelist_aba - freelist_aba_t freelist_counter; -#endif - }; + struct freelist_counters; }; struct rcu_head rcu_head; }; @@ -114,7 +110,7 @@ SLAB_MATCH(_unused_slab_obj_exts, obj_exts); #undef SLAB_MATCH static_assert(sizeof(struct slab) <=3D sizeof(struct page)); #if defined(system_has_freelist_aba) -static_assert(IS_ALIGNED(offsetof(struct slab, freelist), sizeof(freelist_= aba_t))); +static_assert(IS_ALIGNED(offsetof(struct slab, freelist), sizeof(struct fr= eelist_counters))); #endif =20 /** diff --git a/mm/slub.c b/mm/slub.c index 5f6408c9e0fd..8330e4f8b3b2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -764,10 +764,12 @@ __update_freelist_fast(struct slab *slab, void *freelist_new, unsigned long counters_new) { #ifdef system_has_freelist_aba - freelist_aba_t old =3D { .freelist =3D freelist_old, .counter =3D counter= s_old }; - freelist_aba_t new =3D { .freelist =3D freelist_new, .counter =3D counter= s_new }; + struct freelist_counters old =3D { .freelist =3D freelist_old, .counters = =3D counters_old }; + struct freelist_counters new =3D { .freelist =3D freelist_new, .counters = =3D counters_new }; =20 - return try_cmpxchg_freelist(&slab->freelist_counter.full, &old.full, new.= full); + return try_cmpxchg_freelist(&slab->freelist_counters, + &old.freelist_counters, + new.freelist_counters); #else return false; #endif --=20 2.51.1 From nobody Mon Feb 9 14:02:28 2026 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7758731A813 for ; Fri, 7 Nov 2025 13:51:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523516; cv=none; b=oWyi7p8RTKIEuEMLFZ+PsHUthlYTbpBgXIb6S1Txm+V+C27NiKY7tEEg2cmRAEwUE47y0P6KXvCG5aLNKSRxmULikkeKEoMPp4XAFhhqO7cul+4HXB104ytSm9AxPOeFCvLL5Rhd6k+nOJLpCFkF+6Szd6+YMxkQU81VREXkyXI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523516; c=relaxed/simple; bh=OOJrymWY/Gu0Gj4MYTTXnOnrPjcJyZkkFclG4H5WBuI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Hslx8hSmdoHS4Ra21IdzIIrdWGQ4AC7Gqcbhq3rvsJ/+do25xxBCRhp7zER0iQZCJzfxnK88fHm/UJfBBJN6v0LL5dwVFMUrg8HIpZJxJ/rbKBSxPEryYNtDTI70NkBKXKmMu3DSKSps67ZuTB3ruEHAyle8QXSe68STH6cXmTY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B2AC9211D6; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) Authentication-Results: smtp-out1.suse.de; none Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 9D1CB13A88; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id IH4OJnL5DWlFCwAAD6G6ig (envelope-from ); Fri, 07 Nov 2025 13:51:46 +0000 From: Vlastimil Babka Date: Fri, 07 Nov 2025 14:51:25 +0100 Subject: [PATCH 3/4] slab: use struct freelist_counters for local variables instead of struct slab Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251107-slab-fms-cleanup-v1-3-650b1491ac9e@suse.cz> References: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> In-Reply-To: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo , linux-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.3 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: B2AC9211D6 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Spam-Flag: NO X-Spam-Score: -4.00 X-Spam-Level: X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[] In several functions we declare local struct slab variables so we can work with the freelist and counters fields (including the sub-counters that are in the union) comfortably. With struct freelist_counters containing the full counters definition, we can now reduce the local variables to that type as we don't need the other fields in struct slab. Signed-off-by: Vlastimil Babka Reviewed-by: Harry Yoo --- mm/slub.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 8330e4f8b3b2..a55e0af26ec7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3725,8 +3725,7 @@ static void deactivate_slab(struct kmem_cache *s, str= uct slab *slab, void *nextfree, *freelist_iter, *freelist_tail; int tail =3D DEACTIVATE_TO_HEAD; unsigned long flags =3D 0; - struct slab new; - struct slab old; + struct freelist_counters old, new; =20 if (READ_ONCE(slab->freelist)) { stat(s, DEACTIVATE_REMOTE_FREES); @@ -4390,7 +4389,7 @@ __update_cpu_freelist_fast(struct kmem_cache *s, */ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) { - struct slab new; + struct freelist_counters new; unsigned long counters; void *freelist; =20 @@ -4418,7 +4417,7 @@ static inline void *get_freelist(struct kmem_cache *s= , struct slab *slab) */ static inline void *freeze_slab(struct kmem_cache *s, struct slab *slab) { - struct slab new; + struct freelist_counters new; unsigned long counters; void *freelist; =20 @@ -5867,7 +5866,7 @@ static void __slab_free(struct kmem_cache *s, struct = slab *slab, { void *old_head; bool was_frozen, was_full; - struct slab new; + struct freelist_counters new; unsigned long counters; struct kmem_cache_node *n =3D NULL; unsigned long flags; --=20 2.51.1 From nobody Mon Feb 9 14:02:28 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12E6E3168FD for ; Fri, 7 Nov 2025 13:52:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523532; cv=none; b=J4jRD17OOongI35z6wtJstm95w7/0/5YTnTIX+ThoVlFFdTFHtJRjgstGVXu+d8uHFGhcvzFXIjFstZkL3YojBNeaNBg4btghY989RTMegFFj/rG03KMSOnjNBIuvbC4kyu9lHJTiKikJOWjma8065ejfsmHJtVwu+b9DzczWk4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762523532; c=relaxed/simple; bh=VlcyFpyyN8/dJ59tQZJAyvizdyQCo7jTGGmdC2FCilQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Eh8itZNpDE7G9f1YIISrkkJDFSXA9TEgx3YTJs38K7Pw737h1Y0PAgQbAvRMjxu/PNSv8tk8GukWRSZ7TkM4fWE63pdW50UBXIvN7ehO+TwEXRva3JbY38bCnYmtJcDZPq8yX86R237qa8PQIb2IOyYoZPJYfn7oxiXuEHJyQSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=DZNvXY9Y; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=XJP9ZOW+; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=DZNvXY9Y; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=XJP9ZOW+; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="DZNvXY9Y"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="XJP9ZOW+"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="DZNvXY9Y"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="XJP9ZOW+" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id BF08A1F461; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lbjp9eBdivzL6P8LhBfaJka806WwCcsNNUIAdQf/UyI=; b=DZNvXY9Y5Cevgwr5I91ouI7W/s/MxZbXP47IXlE4UjiYOYdRSlRE1YWxmZagun2nQqh7IR Szu4FwLkhjMCz+IjYVZPQskmg7DTl9BN01feerW2us4n6UlLWRnQKnxgIaDhs91AZHGQcj xqWgOCjPJfqVc2VGPrQQ+qd+kFS8eQc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lbjp9eBdivzL6P8LhBfaJka806WwCcsNNUIAdQf/UyI=; b=XJP9ZOW+umo9kwWSM++dRKu32L8N+xdF/kBMG28busVSzwXF+bFkYhd5BPtNlqnjqDG8w4 Cr/QFzMCa3zh6GBA== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lbjp9eBdivzL6P8LhBfaJka806WwCcsNNUIAdQf/UyI=; b=DZNvXY9Y5Cevgwr5I91ouI7W/s/MxZbXP47IXlE4UjiYOYdRSlRE1YWxmZagun2nQqh7IR Szu4FwLkhjMCz+IjYVZPQskmg7DTl9BN01feerW2us4n6UlLWRnQKnxgIaDhs91AZHGQcj xqWgOCjPJfqVc2VGPrQQ+qd+kFS8eQc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1762523506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lbjp9eBdivzL6P8LhBfaJka806WwCcsNNUIAdQf/UyI=; b=XJP9ZOW+umo9kwWSM++dRKu32L8N+xdF/kBMG28busVSzwXF+bFkYhd5BPtNlqnjqDG8w4 Cr/QFzMCa3zh6GBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id ABD0913C01; Fri, 7 Nov 2025 13:51:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 2G+nKXL5DWlFCwAAD6G6ig (envelope-from ); Fri, 07 Nov 2025 13:51:46 +0000 From: Vlastimil Babka Date: Fri, 07 Nov 2025 14:51:26 +0100 Subject: [PATCH 4/4] slab: use struct freelist_counters as parameters in relevant functions Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251107-slab-fms-cleanup-v1-4-650b1491ac9e@suse.cz> References: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> In-Reply-To: <20251107-slab-fms-cleanup-v1-0-650b1491ac9e@suse.cz> To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo , linux-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.3 X-Spam-Level: X-Spamd-Result: default: False [-8.30 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[8]; FUZZY_RATELIMITED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.cz:email,suse.cz:mid] X-Spam-Flag: NO X-Spam-Score: -8.30 In functions such as [__]slab_update_freelist() and __slab_update_freelist_fast/slow() we pass old and new freelist and counters as 4 separate parameters. The underlying __update_freelist_fast() then constructs struct freelist_counters variables for passing the full freelist+counter combinations to cmpxchg double. In most cases we actually start with struct freelist_counters variables, but then pass the individual fields, only to construct new struct freelist_counters variables. While it's all inlined and thus should be efficient, we can simplify this code. Thus replace the 4 parameters for individual fields with two freelist_aba_t pointers wherever applicable. __update_freelist_fast() can then pass them directly to try_cmpxchg_freelist(). The code is also more obvious as the pattern becomes unified such that we set up "old" and "new" struct freelist_counters variables upfront as we fully need them to be, and simply call [__]slab_update_freelist() on them. Previously some of the "new" values would be hidden as one of the many parameters and thus make it harder to figure out what the code does. Signed-off-by: Vlastimil Babka Reviewed-by: Harry Yoo --- mm/slub.c | 126 ++++++++++++++++++++++++++--------------------------------= ---- 1 file changed, 52 insertions(+), 74 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index a55e0af26ec7..ddd71f4937fa 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -759,34 +759,29 @@ static __always_inline void slab_unlock(struct slab *= slab) } =20 static inline bool -__update_freelist_fast(struct slab *slab, - void *freelist_old, unsigned long counters_old, - void *freelist_new, unsigned long counters_new) +__update_freelist_fast(struct slab *slab, struct freelist_counters *old, + struct freelist_counters *new) { #ifdef system_has_freelist_aba - struct freelist_counters old =3D { .freelist =3D freelist_old, .counters = =3D counters_old }; - struct freelist_counters new =3D { .freelist =3D freelist_new, .counters = =3D counters_new }; - return try_cmpxchg_freelist(&slab->freelist_counters, - &old.freelist_counters, - new.freelist_counters); + &old->freelist_counters, + new->freelist_counters); #else return false; #endif } =20 static inline bool -__update_freelist_slow(struct slab *slab, - void *freelist_old, unsigned long counters_old, - void *freelist_new, unsigned long counters_new) +__update_freelist_slow(struct slab *slab, struct freelist_counters *old, + struct freelist_counters *new) { bool ret =3D false; =20 slab_lock(slab); - if (slab->freelist =3D=3D freelist_old && - slab->counters =3D=3D counters_old) { - slab->freelist =3D freelist_new; - slab->counters =3D counters_new; + if (slab->freelist =3D=3D old->freelist && + slab->counters =3D=3D old->counters) { + slab->freelist =3D new->freelist; + slab->counters =3D new->counters; ret =3D true; } slab_unlock(slab); @@ -802,22 +797,18 @@ __update_freelist_slow(struct slab *slab, * interrupt the operation. */ static inline bool __slab_update_freelist(struct kmem_cache *s, struct sla= b *slab, - void *freelist_old, unsigned long counters_old, - void *freelist_new, unsigned long counters_new, - const char *n) + struct freelist_counters *old, struct freelist_counters *new, const char= *n) { bool ret; =20 if (USE_LOCKLESS_FAST_PATH()) lockdep_assert_irqs_disabled(); =20 - if (s->flags & __CMPXCHG_DOUBLE) { - ret =3D __update_freelist_fast(slab, freelist_old, counters_old, - freelist_new, counters_new); - } else { - ret =3D __update_freelist_slow(slab, freelist_old, counters_old, - freelist_new, counters_new); - } + if (s->flags & __CMPXCHG_DOUBLE) + ret =3D __update_freelist_fast(slab, old, new); + else + ret =3D __update_freelist_slow(slab, old, new); + if (likely(ret)) return true; =20 @@ -832,21 +823,17 @@ static inline bool __slab_update_freelist(struct kmem= _cache *s, struct slab *sla } =20 static inline bool slab_update_freelist(struct kmem_cache *s, struct slab = *slab, - void *freelist_old, unsigned long counters_old, - void *freelist_new, unsigned long counters_new, - const char *n) + struct freelist_counters *old, struct freelist_counters *new, const char= *n) { bool ret; =20 if (s->flags & __CMPXCHG_DOUBLE) { - ret =3D __update_freelist_fast(slab, freelist_old, counters_old, - freelist_new, counters_new); + ret =3D __update_freelist_fast(slab, old, new); } else { unsigned long flags; =20 local_irq_save(flags); - ret =3D __update_freelist_slow(slab, freelist_old, counters_old, - freelist_new, counters_new); + ret =3D __update_freelist_slow(slab, old, new); local_irq_restore(flags); } if (likely(ret)) @@ -3774,10 +3761,7 @@ static void deactivate_slab(struct kmem_cache *s, st= ruct slab *slab, } else { new.freelist =3D old.freelist; } - } while (!slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); + } while (!slab_update_freelist(s, slab, &old, &new, "unfreezing slab")); =20 /* * Stage three: Manipulate the slab list based on the updated state. @@ -4389,27 +4373,24 @@ __update_cpu_freelist_fast(struct kmem_cache *s, */ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) { - struct freelist_counters new; - unsigned long counters; - void *freelist; + struct freelist_counters old, new; =20 lockdep_assert_held(this_cpu_ptr(&s->cpu_slab->lock)); =20 do { - freelist =3D slab->freelist; - counters =3D slab->counters; + old.freelist =3D slab->freelist; + old.counters =3D slab->counters; =20 - new.counters =3D counters; + new.freelist =3D NULL; + new.counters =3D old.counters; =20 - new.inuse =3D slab->objects; - new.frozen =3D freelist !=3D NULL; + new.inuse =3D old.objects; + new.frozen =3D old.freelist !=3D NULL; =20 - } while (!__slab_update_freelist(s, slab, - freelist, counters, - NULL, new.counters, - "get_freelist")); =20 - return freelist; + } while (!__slab_update_freelist(s, slab, &old, &new, "get_freelist")); + + return old.freelist; } =20 /* @@ -4417,26 +4398,22 @@ static inline void *get_freelist(struct kmem_cache = *s, struct slab *slab) */ static inline void *freeze_slab(struct kmem_cache *s, struct slab *slab) { - struct freelist_counters new; - unsigned long counters; - void *freelist; + struct freelist_counters old, new; =20 do { - freelist =3D slab->freelist; - counters =3D slab->counters; + old.freelist =3D slab->freelist; + old.counters =3D slab->counters; =20 - new.counters =3D counters; + new.freelist =3D NULL; + new.counters =3D old.counters; VM_BUG_ON(new.frozen); =20 - new.inuse =3D slab->objects; + new.inuse =3D old.objects; new.frozen =3D 1; =20 - } while (!slab_update_freelist(s, slab, - freelist, counters, - NULL, new.counters, - "freeze_slab")); + } while (!slab_update_freelist(s, slab, &old, &new, "freeze_slab")); =20 - return freelist; + return old.freelist; } =20 /* @@ -5864,10 +5841,8 @@ static void __slab_free(struct kmem_cache *s, struct= slab *slab, unsigned long addr) =20 { - void *old_head; bool was_frozen, was_full; - struct freelist_counters new; - unsigned long counters; + struct freelist_counters old, new; struct kmem_cache_node *n =3D NULL; unsigned long flags; bool on_node_partial; @@ -5891,13 +5866,19 @@ static void __slab_free(struct kmem_cache *s, struc= t slab *slab, spin_unlock_irqrestore(&n->list_lock, flags); n =3D NULL; } - old_head =3D slab->freelist; - counters =3D slab->counters; - set_freepointer(s, tail, old_head); - new.counters =3D counters; - was_frozen =3D !!new.frozen; - was_full =3D (old_head =3D=3D NULL); + + old.freelist =3D slab->freelist; + old.counters =3D slab->counters; + + was_full =3D (old.freelist =3D=3D NULL); + was_frozen =3D old.frozen; + + set_freepointer(s, tail, old.freelist); + + new.freelist =3D head; + new.counters =3D old.counters; new.inuse -=3D cnt; + /* * Might need to be taken off (due to becoming empty) or added * to (due to not being full anymore) the partial list. @@ -5926,10 +5907,7 @@ static void __slab_free(struct kmem_cache *s, struct= slab *slab, } } =20 - } while (!slab_update_freelist(s, slab, - old_head, counters, - head, new.counters, - "__slab_free")); + } while (!slab_update_freelist(s, slab, &old, &new, "__slab_free")); =20 if (likely(!n)) { =20 --=20 2.51.1