From nobody Tue Sep 29 03:55:25 2026 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83B403DAAC5 for ; Sat, 15 Aug 2026 06:54:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786776869; cv=none; b=qzJj9o2Bv3xOb7yi9iagKoxDmyAwC9l2q7nkJR4A5NnstH/I8HRzta1RhUNcvh/BNtgX5sGzmgnWHQiInifvFQjQvy4H7TBDr7q7XIBhX8kBq5c7htL1jh+sNqNaj8CahDyVQmuEO02nx+KCThZes6lVVmPuaKqcURc9DGfM/G4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786776869; c=relaxed/simple; bh=S7E5nbBY6/hJJLAkyWwanqtFvXLl1dY3D7LGfRRuIcw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=dLaVkc483hspa5PuL2rRlWQB8CTvBdNQwsU3tccnOXYgfGzUqIEVtw4VYpijrAWUBMvlXHs5/99qjvzBuWulrRZtgOvEpyE9Zujmz6QO/usNTnzy76kkMBXkxsTk+ptqtVJfndTvSmDQBecc3La7uGMn9CGT8YB/shGwScLU/k4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OWyLwyr6; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OWyLwyr6" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-38125cebfdaso2703138a91.1 for ; Fri, 14 Aug 2026 23:54:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786776858; x=1787381658; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EFTk5A+CpTEE2+SBU0kZXE2vNyQzqzpJb7ipur+kflw=; b=OWyLwyr6rSTBYAatMNaVYSyI57qA6AG3F/1tWyfXC97KpBvlAAJlSarYkdg7RZuxAS x85PqBoLT+Y4/7TJWoJy4vcAe6ZutBlul3KbDcGBFFK92VtxWbBL2IE8nrO7KPGvhjiC m2gAneTAgBDVKYcRed0X1HvVrngpn1hgBgdNe00ABgv5EM9emP4EIO6iMYIUtuB3Ut2z J6tU+EpddpOuO+4Vx2RB23gpPdkQ6onf1RNuxARDA7Z/GxnMsmE2U8r8qy3CdEFjtsNR UihvXUVth1/H2xYmWGm6Cjxlbp7W82oSUmdAwwUS2UZv3vZa+AuA3YcVUEPtoolXJ0qv SCPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786776858; x=1787381658; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=EFTk5A+CpTEE2+SBU0kZXE2vNyQzqzpJb7ipur+kflw=; b=Tw+t2RHDZ5df5TrMe7nJSv6BJ0tcu8iU4qhNJGcZcOsWt2jg5F0PZJ4CbTXn6eeMnR 2klrBkGH5XRoVlJgYdbl7hc1wNtKe52KhqPB+yOMngBHa9/fTLBv2r7KYfY6jBtRSfIC VgRWz+mDI92ANL+x/mnLJYfl3nEuKgej+pwEvdsz7t9Iq9ICmUdKjmycP7hZxHX+s3uR J6i017MPxFxgb0oWoZjxp5fv+dL04efNwZ+OIwKpH1MhrDwy0fFZkeUUg58qFjTGleqI 3v+lF8jMNMk43ZlxiCrJSP5N3M9ehxMmkOff2KdTIbgKJdFjNjIEcrCVTt2wrO+7Mtvc KXWQ== X-Forwarded-Encrypted: i=1; AHgh+RpUPunmI5CnMrD+Zw2k5VFdTNV35F1Y33NtMAbD34aIA4zJ2drQaxB1ln6lGHuBNfhQskqeVrlOwBRqPOQ=@vger.kernel.org X-Gm-Message-State: AOJu0YzmELV3aNJ+Rc/44nHoNd49UJMe2fSe7NXH8oh1GMwIPo+YcmcJ sgNrDvTY4Xl3S2x8UmWXbOok/k3+tHfWJAuvcrYKCna6oSHOk75tPogjPdxlk8ZX X-Gm-Gg: AR+sD13sxjEaj2rJ6SWycBxx/qvIbtw5JPufGOQFPYr+0uCZtIzKuItMjLYXqhqO9i/ OYrb8i32G3xGJGj1g8d11TLeVwvQkM5rpEJQIqwcm8s882RQJZ1lTSMNzNa+yLQOCNqT8ZiGHNT 9r0SLvMyIeP0wBUMHsUhd/QwCHKsPETHs32iJ3ezRPt+DYAsEuSVqV8GYgfOrMFllqv7oLt8dwb idmSUip7OKGPCW6AVmFX6+zOQTFpsIXxuMmg+H+cCJIxs0wMP0LdDu7UHivd7vMF8pr6C8w4fJ6 /BDZmK6O+XERXbPi74IdPqftGqP3bZwz8VUrI7QkebQbovYbi+3uLNbnyybHdDMUxiCelAATcWm hHpP8etR/T7sAVQ6mLIA5sW28HESe+gLQvQHJU582sgFQ+pbD5Lca0vtEh2lUAIE++L6pYZ0TJ6 gB6DcizZmk0JW99rtLV6AAux6tklqlvvPttq5Y0GKYfiGTWjOhIl9Ie34gdvl5LSJ6BqZNG8Zcz 4n58rrcHhYtlcCHPIw75wAK/mi+y+1wHCah2pt8uRSUKZHiaA== X-Received: by 2002:a17:90b:4c48:b0:38e:488f:7068 with SMTP id 98e67ed59e1d1-3933b6d0d30mr11800411a91.2.1786776857744; Fri, 14 Aug 2026 23:54:17 -0700 (PDT) Received: from NV-J4GCB44.localdomain ([103.74.125.162]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-394eb752fe2sm5691360a91.14.2026.08.14.23.54.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Aug 2026 23:54:17 -0700 (PDT) From: Jianyue Wu Date: Sat, 15 Aug 2026 14:54:07 +0800 Subject: [PATCH RFC v3 1/3] mm/zswap: release retired pools via call_rcu() instead of synchronize_rcu() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260815-shrink_zswap_entry_0815_v2-v3-1-0171bd86a667@gmail.com> References: <20260815-shrink_zswap_entry_0815_v2-v3-0-0171bd86a667@gmail.com> In-Reply-To: <20260815-shrink_zswap_entry_0815_v2-v3-0-0171bd86a667@gmail.com> To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton Cc: Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jianyue Wu X-Mailer: b4 0.13.0 When a pool's last reference is dropped, __zswap_pool_empty() removes it from the pool list and schedules __zswap_pool_release(), which calls synchronize_rcu() to wait for readers before tearing the pool down. synchronize_rcu() is a synchronous, potentially long wait. Replace it with an asynchronous call_rcu(): __zswap_pool_empty() now hands the pool to call_rcu(), and the RCU callback defers the sleepable teardown to the release worker (the callback itself runs in softirq context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it. Suggested-by: Yosry Ahmed Signed-off-by: Jianyue Wu --- mm/zswap.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 37f34e406c8e..cc4243356e21 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -155,6 +155,7 @@ struct zswap_pool { struct crypto_acomp_ctx __percpu *acomp_ctx; struct percpu_ref ref; struct list_head list; + struct rcu_head rcu_head; struct work_struct release_work; struct hlist_node node; char tfm_name[CRYPTO_MAX_ALG_NAME]; @@ -382,8 +383,6 @@ static void __zswap_pool_release(struct work_struct *wo= rk) struct zswap_pool *pool =3D container_of(work, typeof(*pool), release_work); =20 - synchronize_rcu(); - /* nobody should have been able to get a ref... */ WARN_ON(!percpu_ref_is_zero(&pool->ref)); percpu_ref_exit(&pool->ref); @@ -392,6 +391,20 @@ static void __zswap_pool_release(struct work_struct *w= ork) zswap_pool_destroy(pool); } =20 +static void __zswap_pool_release_rcu(struct rcu_head *head) +{ + struct zswap_pool *pool =3D container_of(head, typeof(*pool), rcu_head); + + /* + * The grace period has elapsed, so no RCU reader can still observe the + * pool through the list it was removed from in __zswap_pool_empty(). + * Hand off to a worker for the sleepable teardown, since this callback + * runs in softirq context. + */ + INIT_WORK(&pool->release_work, __zswap_pool_release); + schedule_work(&pool->release_work); +} + static struct zswap_pool *zswap_pool_current(void); =20 static void __zswap_pool_empty(struct percpu_ref *ref) @@ -406,8 +419,7 @@ static void __zswap_pool_empty(struct percpu_ref *ref) =20 list_del_rcu(&pool->list); =20 - INIT_WORK(&pool->release_work, __zswap_pool_release); - schedule_work(&pool->release_work); + call_rcu(&pool->rcu_head, __zswap_pool_release_rcu); =20 spin_unlock_bh(&zswap_pools_lock); } --=20 2.43.0 From nobody Tue Sep 29 03:55:25 2026 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47B223C0A04 for ; Sat, 15 Aug 2026 06:54:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786776875; cv=none; b=D5A19PGrn00Re3qswg8ies3ss8AnBO0Pjp/ZHdf2qCxvF2ka5w3UzZke50lHWOaXBK9kD4c5CbtzFcBNuFE0aRU1pd3170u8UcB+9eYWkgHgh7p4KXRk+oQ4a9eiRN7mt0aWS2YWjXfvV3KnjDBzvfiy67HaLYat2AlHAKaQG5A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786776875; c=relaxed/simple; bh=bgvpeNwnfkovXqPI1/l9XpXxKkf1TQAg+SLYDbUGTqA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=iu3dgqMMM3hr+PMKvODnEmZQuCh28SPz0wort4BFPk95C6t4rrNXwCeiM+8cDQtP+Zt0b1BQrIm2zz/5Fr0LZQd4gjLVBSoosGQDqXkoenMSLIvQLTYSF3Uj3sTUKy0j8AAuck3g4Vkjj2jgxmFEDx8W+VBoOelLCsxlywt9QZU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XFhAkHev; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XFhAkHev" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-38125cebfdaso2703176a91.1 for ; Fri, 14 Aug 2026 23:54:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786776861; x=1787381661; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=L88HWI4nOnXjNaN6Z4WUAG6hZj/n7nHW2N2BDrBtNXg=; b=XFhAkHevlRk7y2XD4IMNHWoHrfXW+g+etAxou84VuRWB4h16wggT/KMOSYgn1pRjoC mitVO7Ndnk842FrqcBFSdWciIiI0AQRmiYK70Qy87kH9maHJ2QorKx8a1cRh3EjJzXxb Qq+0Or1qNytA750vbCZTQvtZRX5BtUrdpvJhDGYi8ReKddZUHsIrHG+g5lNDua6dSHy6 hd1kMN+U6d1xpghEVOSoGQccdtq12Mx6l7TELAc/i7MCjF1s5eJ2n+NH0GIchVwpuH/1 z1q6/qCJZAR4rIR7Sxonf9O5wSw1xSBMqJxQDOIQhuPyu3whe3/IAFULPvpbNVh21/dP WvRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786776861; x=1787381661; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=L88HWI4nOnXjNaN6Z4WUAG6hZj/n7nHW2N2BDrBtNXg=; b=pM4WrkkvHLjrJskypdsmzB35vlrbKEpDj6gAU3NfaQ3l2hd+6chj3F8tqiGaTmjcDk Vwj8SCJc8/jvGPg3Z0LbkqfWVt/EL4DWxVgGGlIxsXKx+MgVInb4HOV6DYtWh8i5LhpZ vfdebq5+sz0+HE8YNKWLJhgnGDEtoyZlDC/uW+8Ag8xEpiZz8w2qpOJGVyk5hyNfB2tp w7LNSunhmu8bhJQLfX73MOfTbodjXA7+njW4bCqA7qKDHoUPQFcu7pl6qVGRw8D1tWFR obdwbn2D+YuE2ZwzhDVBcfL+ki8COVM5K/GVsCDEM3T/up5JNME5mgdejscUx2FZjayG 8iPw== X-Forwarded-Encrypted: i=1; AHgh+RofEDHreQywyfnZg3fM8VIsK+BpVq6qkhFk2qVqc8vn37rv2yH4cAqphxHYjYdvLwM5DxLrAY9lp3+YZwk=@vger.kernel.org X-Gm-Message-State: AOJu0YzFaRLSmUinWqH7whWFeuYCLk3h9/1Y8FH5L9CZ5evvHVj7trVF TbcqJAUZ6fhdbO7c351v8GCT/sOqgbHAvLyqKUZ6IH+goJNWSu1pFrmt X-Gm-Gg: AR+sD12SjxBg8en69vR3OvylpDLV+yrkbznLHMEbXAuih6zXmlwuBykfdEBbGtNLe3y dcb2Ymjp3Cnmy0GL6PbU0z5ARItvJdBaPLSdZEhIIkxNU48ZqMIdPEOkpEhdgOlYf9Fpf/5zkuH iQu7WU41pgs1ZMoA9NzhxmhnDUmKcUM60qZEhnuo1feXQ8AsWRM+owjcORSjrRUgK3fD9MLmvB9 iOfDeI+QSCuf7lRt5NUD6bDih6D4bBuc1I8mld7mBO2fANpdCXQgfC1z7pmDA1Sis46SJcYZ/nH nE3qRgTNDqMXfxKtZmfBLCXdXxx2E5/vmuQd7lqBlqSNQBaOJSiAj2l3ctxPjYBGmALJB/gNsee l2+CZ1mhbr/IgjviVNQjJ92jKDTLwBSoiXlvc0eXMGcPYZwRLoGep0JmDe1xs9pIz9bk1TZFaJT iCFo3Pb5kCkwxiT2gxMjFNT+aZb/++/414vHAZ9+gLFYITti+XL/+oABgjiX0fWZYtkOAavc+lH DGnhFj0y5hHYZ/eUNFN3gqrH80uk3Q8jwGFV6A= X-Received: by 2002:a17:90b:4cc4:b0:38e:101f:c867 with SMTP id 98e67ed59e1d1-3933b927a55mr13212040a91.18.1786776860731; Fri, 14 Aug 2026 23:54:20 -0700 (PDT) Received: from NV-J4GCB44.localdomain ([103.74.125.162]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-394eb752fe2sm5691360a91.14.2026.08.14.23.54.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Aug 2026 23:54:20 -0700 (PDT) From: Jianyue Wu Date: Sat, 15 Aug 2026 14:54:08 +0800 Subject: [PATCH RFC v3 2/3] mm/zswap: replace the zswap_pools list with a fixed pools array Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260815-shrink_zswap_entry_0815_v2-v3-2-0171bd86a667@gmail.com> References: <20260815-shrink_zswap_entry_0815_v2-v3-0-0171bd86a667@gmail.com> In-Reply-To: <20260815-shrink_zswap_entry_0815_v2-v3-0-0171bd86a667@gmail.com> To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton Cc: Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jianyue Wu X-Mailer: b4 0.13.0 Originally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches. Hold the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and track the current pool with a separate rcu-protected pointer. Slot 0 is intentionally left unused (always NULL): a zeroed or incorrectly initialized pool index then resolves to NULL and trips a WARN rather than silently aliasing a live pool in another slot. The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. A pool first reserves its slot with a placeholder marker and only stores the real pointer once it is committed as the current pool, so array walkers never observe a not-yet-ready pool as live. Behavior change: the fixed array bounds the number of simultaneously live pools at ZSWAP_MAX_POOLS - 1 (15, since slot 0 is reserved), whereas the old list was unbounded. A pool is only live while it is the current pool or still has stored pages referencing it, and pools are reused across compressor switches, so 15 is far more than any real configuration needs. Once all slots are occupied, creating a pool for a 16th distinct compressor fails: zswap_pool_create() errors and returns NULL, and the compressor switch is rejected with -EINVAL rather than silently succeeding. The cap can be raised by increasing ZSWAP_MAX_POOLS (bounded by the u8 slot index, so up to 256). Suggested-by: Nhat Pham Suggested-by: Yosry Ahmed Signed-off-by: Jianyue Wu --- mm/zswap.c | 140 +++++++++++++++++++++++++++++++++++++++++++++++++--------= ---- 1 file changed, 114 insertions(+), 26 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index cc4243356e21..603fdc418041 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -13,6 +13,7 @@ =20 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt =20 +#include #include #include #include @@ -154,13 +155,42 @@ struct zswap_pool { struct zs_pool *zs_pool; struct crypto_acomp_ctx __percpu *acomp_ctx; struct percpu_ref ref; - struct list_head list; struct rcu_head rcu_head; struct work_struct release_work; struct hlist_node node; + u8 idx; char tfm_name[CRYPTO_MAX_ALG_NAME]; }; =20 +#define ZSWAP_MAX_POOLS 16 +/* + * Slot 0 is intentionally never used: it stays NULL so that a zeroed or + * incorrectly initialized pool->idx resolves to NULL (and trips a WARN) + * instead of silently aliasing a live pool in another slot. + */ +#define ZSWAP_FIRST_POOL_SLOT 1 +static struct zswap_pool __rcu *zswap_pools[ZSWAP_MAX_POOLS]; +static_assert(ZSWAP_MAX_POOLS - 1 <=3D U8_MAX); +/* + * The current pool (NULL if none): an alias of one zswap_pools[] slot. + * It always holds a ref, so a pool is never retired while it is current. + */ +static struct zswap_pool __rcu *zswap_current_pool; + +/* + * A slot placeholder used to reserve an index before the pool is committe= d. + * A reserving pool publishes this marker first and only stores the real + * pointer once it is ready to become current; walkers of zswap_pools[] tr= eat + * a reserved slot as empty and skip it, so a not-yet-ready pool is never + * observed as live. + */ +#define ZSWAP_SLOT_RESERVED ((struct zswap_pool *)-1UL) + +static inline bool zswap_slot_is_pool(struct zswap_pool *pool) +{ + return pool && pool !=3D ZSWAP_SLOT_RESERVED; +} + /* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru; =20 @@ -201,9 +231,7 @@ struct zswap_entry { static struct xarray *zswap_trees[MAX_SWAPFILES]; static unsigned int nr_zswap_trees[MAX_SWAPFILES]; =20 -/* RCU-protected iteration */ -static LIST_HEAD(zswap_pools); -/* protects zswap_pools list modification */ +/* protects the zswap_pools array and zswap_current_pool */ static DEFINE_SPINLOCK(zswap_pools_lock); /* pool counter to provide unique names to zsmalloc */ static atomic_t zswap_pools_count =3D ATOMIC_INIT(0); @@ -271,6 +299,40 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *ac= omp_ctx) acomp_ctx->buffer =3D NULL; } =20 +static int zswap_pool_reserve_slot(struct zswap_pool *pool) +{ + int i; + + guard(spinlock_bh)(&zswap_pools_lock); + for (i =3D ZSWAP_FIRST_POOL_SLOT; i < ZSWAP_MAX_POOLS; i++) { + if (!rcu_access_pointer(zswap_pools[i])) { + /* + * Reserve the slot with a placeholder rather than the + * pool itself: the pool is not ready to be current yet + * and must not be observed as live by array walkers. + * Set idx before publishing so readers never see it + * stale. zswap_pool_publish_slot() stores the real + * pointer once the pool is committed. + */ + pool->idx =3D i; + rcu_assign_pointer(zswap_pools[i], ZSWAP_SLOT_RESERVED); + return i; + } + } + + return -ENOSPC; +} + +static void zswap_pool_publish_slot(struct zswap_pool *pool) +{ + assert_spin_locked(&zswap_pools_lock); + + if (rcu_dereference_protected(zswap_pools[pool->idx], + lockdep_is_held(&zswap_pools_lock)) =3D=3D + ZSWAP_SLOT_RESERVED) + rcu_assign_pointer(zswap_pools[pool->idx], pool); +} + static struct zswap_pool *zswap_pool_create(char *compressor) { struct zswap_pool *pool; @@ -314,19 +376,28 @@ static struct zswap_pool *zswap_pool_create(char *com= pressor) if (ret) goto cpuhp_add_fail; =20 - /* being the current pool takes 1 ref; this func expects the - * caller to always add the new pool as the current pool + /* + * The initial ref is for the current-pool role. On success the + * caller must install this pool as the current pool. */ ret =3D percpu_ref_init(&pool->ref, __zswap_pool_empty, PERCPU_REF_ALLOW_REINIT, GFP_KERNEL); if (ret) goto ref_fail; - INIT_LIST_HEAD(&pool->list); + + ret =3D zswap_pool_reserve_slot(pool); + if (ret < 0) { + pr_err("cannot create more than %d pools\n", + ZSWAP_MAX_POOLS - ZSWAP_FIRST_POOL_SLOT); + goto slot_fail; + } =20 zswap_pool_debug("created", pool); =20 return pool; =20 +slot_fail: + percpu_ref_exit(&pool->ref); ref_fail: cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); =20 @@ -387,7 +458,7 @@ static void __zswap_pool_release(struct work_struct *wo= rk) WARN_ON(!percpu_ref_is_zero(&pool->ref)); percpu_ref_exit(&pool->ref); =20 - /* pool is now off zswap_pools list and has no references. */ + /* readers drained by the grace period */ zswap_pool_destroy(pool); } =20 @@ -397,9 +468,9 @@ static void __zswap_pool_release_rcu(struct rcu_head *h= ead) =20 /* * The grace period has elapsed, so no RCU reader can still observe the - * pool through the list it was removed from in __zswap_pool_empty(). - * Hand off to a worker for the sleepable teardown, since this callback - * runs in softirq context. + * pool through the array slot cleared in __zswap_pool_empty(). Hand off + * to a worker for the sleepable teardown, since this callback runs in + * softirq context. */ INIT_WORK(&pool->release_work, __zswap_pool_release); schedule_work(&pool->release_work); @@ -417,7 +488,13 @@ static void __zswap_pool_empty(struct percpu_ref *ref) =20 WARN_ON(pool =3D=3D zswap_pool_current()); =20 - list_del_rcu(&pool->list); + /* + * Clear the slot before retiring the pool so new readers cannot see + * it; the call_rcu() below drains readers that already observed it. + * The slot may still hold the ZSWAP_SLOT_RESERVED placeholder if the + * pool is torn down before it was ever published as current. + */ + rcu_assign_pointer(zswap_pools[pool->idx], NULL); =20 call_rcu(&pool->rcu_head, __zswap_pool_release_rcu); =20 @@ -447,7 +524,8 @@ static struct zswap_pool *__zswap_pool_current(void) { struct zswap_pool *pool; =20 - pool =3D list_first_or_null_rcu(&zswap_pools, typeof(*pool), list); + pool =3D rcu_dereference_check(zswap_current_pool, + lockdep_is_held(&zswap_pools_lock)); WARN_ONCE(!pool && zswap_has_pool, "%s: no page storage pool!\n", __func__); =20 @@ -480,11 +558,15 @@ static struct zswap_pool *zswap_pool_current_get(void) static struct zswap_pool *zswap_pool_find_get(char *compressor) { struct zswap_pool *pool; + int i; =20 assert_spin_locked(&zswap_pools_lock); =20 - list_for_each_entry_rcu(pool, &zswap_pools, list) { - if (strcmp(pool->tfm_name, compressor)) + for (i =3D ZSWAP_FIRST_POOL_SLOT; i < ZSWAP_MAX_POOLS; i++) { + pool =3D rcu_dereference_protected(zswap_pools[i], + lockdep_is_held(&zswap_pools_lock)); + if (!zswap_slot_is_pool(pool) || + strcmp(pool->tfm_name, compressor)) continue; /* if we can't get it, it's about to be destroyed */ if (!zswap_pool_tryget(pool)) @@ -509,10 +591,14 @@ unsigned long zswap_total_pages(void) { struct zswap_pool *pool; unsigned long total =3D 0; + int i; =20 rcu_read_lock(); - list_for_each_entry_rcu(pool, &zswap_pools, list) - total +=3D zs_get_total_pages(pool->zs_pool); + for (i =3D ZSWAP_FIRST_POOL_SLOT; i < ZSWAP_MAX_POOLS; i++) { + pool =3D rcu_dereference(zswap_pools[i]); + if (zswap_slot_is_pool(pool)) + total +=3D zs_get_total_pages(pool->zs_pool); + } rcu_read_unlock(); =20 return total; @@ -574,7 +660,6 @@ static int zswap_compressor_param_set(const char *val, = const struct kernel_param if (pool) { zswap_pool_debug("using existing", pool); WARN_ON(pool =3D=3D zswap_pool_current()); - list_del_rcu(&pool->list); } =20 spin_unlock_bh(&zswap_pools_lock); @@ -601,16 +686,16 @@ static int zswap_compressor_param_set(const char *val= , const struct kernel_param spin_lock_bh(&zswap_pools_lock); =20 if (!ret) { + /* + * Remember the old current, then publish the new pool into its + * slot before making it current: array walkers must never see + * the current pool as a reserved (not-yet-ready) slot. + */ put_pool =3D zswap_pool_current(); - list_add_rcu(&pool->list, &zswap_pools); + zswap_pool_publish_slot(pool); + rcu_assign_pointer(zswap_current_pool, pool); zswap_has_pool =3D true; } else if (pool) { - /* - * Add the possibly pre-existing pool to the end of the pools - * list; if it's new (and empty) then it'll be removed and - * destroyed by the put after we drop the lock - */ - list_add_tail_rcu(&pool->list, &zswap_pools); put_pool =3D pool; } =20 @@ -1815,7 +1900,10 @@ static int zswap_setup(void) pool =3D __zswap_pool_create_fallback(); if (pool) { pr_info("loaded using pool %s\n", pool->tfm_name); - list_add(&pool->list, &zswap_pools); + spin_lock_bh(&zswap_pools_lock); + zswap_pool_publish_slot(pool); + rcu_assign_pointer(zswap_current_pool, pool); + spin_unlock_bh(&zswap_pools_lock); zswap_has_pool =3D true; static_branch_enable(&zswap_ever_enabled); } else { --=20 2.43.0 From nobody Tue Sep 29 03:55:25 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B63773D9537 for ; Sat, 15 Aug 2026 06:54:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786776874; cv=none; b=O4LN3r/FNT8gSuebLzMxj9mPQ85sy8YVgsmo6xiJdeTYDW6D7uS5GP3BrVJBAJfxR3yZMTNLpQpeCbadTzqNUzcigAyDhpeYeZyeY4D+7sEotFuiOqpPxE1HvQufVC4QLl4LIyG35D7Id3u/w3FFhLMKi7SE9tvFR9g+sWsfyw8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786776874; c=relaxed/simple; bh=A52yWtqdzuqe1T3O/mBSV4fk7lgI/crkjKYJQVY6VRw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=OQPRM1EFQBmmgW3FDm+TcxYsokiIUgsvqmgY299inGE35RluyNTtPwoD/YosADbNScBc+Pzpbp/Kii1OzPX3aZH5XueClxSUTls0lGgh4xwyG+IESs9AV4FvMqOMZYDoiqwLBEypZS1G/lIL9KI9tSHmyDETfDVkxlGpApCkuBI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Xfr808GI; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xfr808GI" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-38d489b6b71so1927075a91.0 for ; Fri, 14 Aug 2026 23:54:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786776864; x=1787381664; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LB66jlKU4CiUMoYa9Xqd6V/xDZ5/WArrBjIBC1saW6k=; b=Xfr808GIgw5igZ5xGz5F470Rlsg99+H1IddEQ3+ugcm8b1IuIt2gPuQ/dzbfwleH6R 4PIGK6+pneeQaVYhSy09qD5oC2gzLumG3y4s/4sjQ81QEeMliaEY6jrAOl02+1FUcbWJ nokXB2CS3XT58xZMGr/CjrylNSaRjzjV1COYV1HrP4LMjpSoBCghPZ9vhTHmsS3G0SBC uMkVTefMlnz3bf3QH4ZwsZOKt2T1knEnK8nzd1DO+xZncTgSUfb+1UM4SQx0woJ4jYIz IaLVEyzougEMHTafgF9ldJg3fOZMMJO6NHtbD/dxuxbA8rrqYXDxTKKwcVGlZPpdHdI9 Nddw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786776864; x=1787381664; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=LB66jlKU4CiUMoYa9Xqd6V/xDZ5/WArrBjIBC1saW6k=; b=W7hVQIw7i8cE62suD99sV5T3cVZjr3cPzWS3L00fSyBWejjVjl5/KEFHTvsnHoA/h8 Ve+/5r/mKfefSSKZawkYfEUFD1xiLxgxgQuYyo0IOukqCRqZla34yPMRCJfYVcsVKp2T EjWV8JzgTcv4Xi2/n/QS9RMHdotrmscjvUrRL3NKoe6LYzwTkKrtC31jbxGu4roaSj0w gcHLQ3m334SUmCdzGNzX8Qf3u/YTFGhq3xqTeI9aKcg11LyEaMzaSPr1r4lLhBe4eD6U sZZXLr1E3NhtMkmRXUtBL2ZhH3LDqgTU4KL9mN/wVJxaiMrN972USi06BGYq1FK62J8u HR0w== X-Forwarded-Encrypted: i=1; AHgh+RqG7xdrc61xZFH8G6wG5NDBxGhdb6bpXojoRs/Dg9AIFEH73efQAUIt+tIhU49m8L2n8Ps2qLP83j+VUso=@vger.kernel.org X-Gm-Message-State: AOJu0YxD8ECIXgVsxJax6s9qOes4a6iWD7HpZ/7uvD2iJD2VrVZZ01GF q5jyXLhg/5BS1Kc9CdVm0waqwZ1pRV+mwgepOk0+aovtnhLdIqr9z3JH X-Gm-Gg: AR+sD125v7RkbKGNldcmnfjHRPg4b/nC6WOZNnXNVDnFDcy/BDMM2h6mTrBoOCxzIAS K12rFZRkeeuKn1OQzo9/gMlGOo2jMkDKDLcCnM0YMT3bSKIyWZla+4nTSjOBgw/tZExG5N2vm6/ mbKaE0ozgxn11JHCxPSrhWtciXgjhzdUqZD0DyrUxFGOHmpYfYPg50dVPg6Ef0dy+wPD+MuOQL2 9DK8ItlmQxKPDX+L2UgXEYUIcX/wOF31bYfOuo+43QAlAZIFwZTf1QAZrr9LHrW3bo3KRI6HDin lqBA8bCt08BLoqrn0VsNuqFPFW7P1UwT7ayXba7aUMzP8yIOCtLXljV41tWTXOjAVck2wdb1Oru lF+YDRswJ2vEeltvwPcTdT9zJavHeSL1EUrFxvajFxk/QC5HDFdUbiNhVErZ/mdyqouKOLBllgp yTecftbQL4urte3GwKTVRiYct7ldEGnxyZ8wvymdRc6pO8Ux8ZUJxb+0hDpD7yUU/SiK/q+6Jib YU0nprSVxJc2QT7HC+Gh7ANLM0+5ph18cAUP1w= X-Received: by 2002:a17:90b:4c8f:b0:38d:dfd1:7c1 with SMTP id 98e67ed59e1d1-3933b77fff8mr11003337a91.2.1786776863659; Fri, 14 Aug 2026 23:54:23 -0700 (PDT) Received: from NV-J4GCB44.localdomain ([103.74.125.162]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-394eb752fe2sm5691360a91.14.2026.08.14.23.54.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Aug 2026 23:54:23 -0700 (PDT) From: Jianyue Wu Date: Sat, 15 Aug 2026 14:54:09 +0800 Subject: [PATCH RFC v3 3/3] mm/zswap: reference the pool by index to shrink struct zswap_entry Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260815-shrink_zswap_entry_0815_v2-v3-3-0171bd86a667@gmail.com> References: <20260815-shrink_zswap_entry_0815_v2-v3-0-0171bd86a667@gmail.com> In-Reply-To: <20260815-shrink_zswap_entry_0815_v2-v3-0-0171bd86a667@gmail.com> To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton Cc: Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Jianyue Wu X-Mailer: b4 0.13.0 struct zswap_entry is allocated once per stored page, so any reduction in its size is multiplied across every entry zswap holds. It currently embeds an 8-byte pool pointer, even though the live pools now sit in a small fixed array indexed by a u8 slot number. Replace the per-entry pool pointer with that u8 slot index and resolve it through a small helper that indexes the pools array. The entry holds a reference to its pool, which keeps the pool alive and its slot occupied, so the lookup is safe without any lock or RCU read-side section. The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on x86_64. This raises objs_per_slab from 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap. Suggested-by: Chris Li Signed-off-by: Jianyue Wu --- mm/zswap.c | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 603fdc418041..3974fca40a4a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -213,7 +213,7 @@ static struct shrinker *zswap_shrinker; * writeback logic. The entry is only reclaimed by the writeb= ack * logic if referenced is unset. See comments in the shrinker * section for context. - * pool - the zswap_pool the entry's data is in + * pool_idx - slot of the zswap_pool that the entry's data is in. * handle - zsmalloc allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. @@ -222,12 +222,24 @@ struct zswap_entry { swp_entry_t swpentry; unsigned int length; bool referenced; - struct zswap_pool *pool; + u8 pool_idx; unsigned long handle; struct obj_cgroup *objcg; struct list_head lru; }; =20 +static struct zswap_pool *zswap_entry_pool(struct zswap_entry *entry) +{ + /* + * The entry holds a reference to its pool, so the slot cannot be + * cleared or reused while the entry is alive: the read is stable + * without the pools lock or an RCU read-side section. Non-zero + * pool_idx (slot 0 is unused) also marks a live entry for lockdep. + */ + return rcu_dereference_protected(zswap_pools[entry->pool_idx], + entry->pool_idx !=3D 0); +} + static struct xarray *zswap_trees[MAX_SWAPFILES]; static unsigned int nr_zswap_trees[MAX_SWAPFILES]; =20 @@ -861,9 +873,13 @@ static void zswap_entry_cache_free(struct zswap_entry = *entry) */ static void zswap_entry_free(struct zswap_entry *entry) { + struct zswap_pool *pool =3D zswap_entry_pool(entry); + zswap_lru_del(entry); - zs_free(entry->pool->zs_pool, entry->handle); - zswap_pool_put(entry->pool); + if (!WARN_ON_ONCE(!pool)) { + zs_free(pool->zs_pool, entry->handle); + zswap_pool_put(pool); + } if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); obj_cgroup_put(entry->objcg); @@ -1020,12 +1036,15 @@ static bool zswap_compress(struct page *page, struc= t zswap_entry *entry, =20 static bool zswap_decompress(struct zswap_entry *entry, struct folio *foli= o) { - struct zswap_pool *pool =3D entry->pool; + struct zswap_pool *pool =3D zswap_entry_pool(entry); struct scatterlist input[2]; /* zsmalloc returns an SG list 1-2 entries */ struct scatterlist output; struct crypto_acomp_ctx *acomp_ctx; int ret =3D 0, dlen; =20 + if (WARN_ON_ONCE(!pool)) + return false; + acomp_ctx =3D raw_cpu_ptr(pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); zs_obj_read_sg_begin(pool->zs_pool, entry->handle, input, entry->length); @@ -1061,7 +1080,7 @@ static bool zswap_decompress(struct zswap_entry *entr= y, struct folio *folio) pr_alert_ratelimited("Decompression error from zswap (%d:%lu %s %u->%d)\n= ", swp_type(entry->swpentry), swp_offset(entry->swpentry), - entry->pool->tfm_name, + pool->tfm_name, entry->length, dlen); return false; } @@ -1564,7 +1583,7 @@ static bool zswap_store_page(struct page *page, * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - entry->pool =3D pool; + entry->pool_idx =3D pool->idx; entry->swpentry =3D page_swpentry; entry->objcg =3D objcg; entry->referenced =3D true; --=20 2.43.0