From nobody Sat Sep 26 21:13:44 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBD0E346ADB for ; Sun, 30 Aug 2026 11:47:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788090471; cv=none; b=ZCT1utCDqHwj03VLM65nkspaLSy0gm/UKi1e88gJPBdS7cMqEXwU2vnJqMmX3os3iWeaX+Jx3lnbMICtoc57Rb6uC+KmDmKM7wiCF5s7OCkFY3vRQoPNDsbinHD0kVoQbbxrLqsdQM2KE1ZHvf+usqPGB8f0H3ZecnQ6im8t0QY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788090471; c=relaxed/simple; bh=8Lf8JSPERRhAveP2ulwaGRXyS4FqNXOTCVEmFATgblg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WHW6JsYFJ+fVnJvmNYXk1STVE8x3YE8CauOGF/qR9WxOsfLWNdlcbr6IfmOg8fbpTJwr0c6cAV8QUKAYp6pGl6VarZgK26NwWux1zIVOMPe4zTuv9MkaZQLr8rzM1tVZUa1/qzo/fYIosv1PUrqnO4VjEoGiTNulSWckRvVdPHE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=L00LwaXe; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="L00LwaXe" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-8518b3ff3e9so2602295b3a.2 for ; Sun, 30 Aug 2026 04:47:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788090469; x=1788695269; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wo0HLGDj3y3Y33U9g+Ukt7jsaJQ99Obk41w4N/1Rf1c=; b=L00LwaXe0I/cvjQZY5BbxVRhCGemizzzEAiO3laCFgwuYmCxzhWPeEaEwInYgT6X7I mHPxQt1/2Uz0bShwV1oyUwrxEnaZK93afAG1jvbNFB3K5FeomlpbmPDB3B7t/Oh3Qq9f QhN6PLDMFW5y7UGXIfKE04dINX0du25AKSgMinheFobz7LJLZtEgovk1nML8ddDNzJ0U kecpn4JWx4sS2+k4Sw70M003wLNF2OJeY6rfLgNQSIaU1bOHeMPm7cBvkUekb3GxVlt9 fRdE4Xd4DOXfKECJZCsahundRZHsgWMde+P9hU5yqrzIZ6++/LFxFCr+ZnHGj2va15FX 3wsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788090469; x=1788695269; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wo0HLGDj3y3Y33U9g+Ukt7jsaJQ99Obk41w4N/1Rf1c=; b=EPUeUYdNG0XlyY7p3bPNPUjt2Z6Muii/RD4NQs/V9/heWqg3+3nGSGTpOLGKJ3aO0o MnJ9WYisGYTefjBszNshGx39aOLQDdv6eIurscgnM2xW1237us5JZH2jWfvDiOL4Ozj0 QFGV2NtAXrEKYpZcnbq/6je+e8TTnU/ngzwX1zzfgkBgbcRZ9kTD3yOdqUOqwB3hrfjR f9++OPBroPRTPHXK4EdrSxHVem7eXOae4FtyZZpfptJDUVYAQ/qOz+tawPk3+fFRr2l+ QL/pjUZcIWJCIxji/j23irzs747mvian7oUOXMgHvd07fVD78v3MP/EobHsjgRtl5jt4 2auw== X-Forwarded-Encrypted: i=1; AHgh+RpZ+11XUzlxPmFgM/IaD1kYR4cnfFCxkpsziV/Q76vVGGif3SoFjMgkL0Mu1BkWnrjXCDJ1vVg4xzoQrP4=@vger.kernel.org X-Gm-Message-State: AFuF++mLAFPBteWjsH4X5rEIzTtYpC2AjQ69uW6N++8LBjdr2HerZamm nJwbbHgTHKgfidi726eJS3sBPfYD6bFRIK/FoHmHPQw9Uy+dwSyi/IoE X-Gm-Gg: AR+sD13H6i19umzZYBlnC+dUGmZ6028fH+DuLSj2K38Ox5O9jenhUDy4c0Jhd0DXgBG xdizL/QQSvgw3VHyL6KiqVDu7wK+JIGPD7oyFe77VDPmzt2F/tbkUhNEXujwSeRapBovafW25bW NkMJS3PnXq50SGEwQuPm8/d4Rw9lnx35dUQnMhCp9+Pkg4fZQTQo+4a1qvkwi+w4FAC9nacMztE s40ia8pQ2nzMXWCnsw7ZrF7OOYjNWqypYgAHArYYL6aFRvHA1clhN7bzDc7YRIQh5TDkSbExxL1 Np29DLbpc+VRSyGY/EQyRmyKksiRUxcpDSfIREFU+FH/c4nUYcn5ebI+U1Cfj6dmsWneBcPHx3m mDBz2jqj+21+WH8SAFzGYO1mT7jpT1JmGiEkNrxenR0PW9XJoeVejTKYPVRLXxxbCY5SzsXegwf MPwgHRmKOvvuzdhqIuaHuYHZ79V6aCQH4cit1RphGRSQpUBz4X1AccHg9hvhac0IgsYRVgPGphD wKEqsFQxo/apnMtSyuHSZ5bNeGs6OYmN6RVtQ== X-Received: by 2002:a05:6a00:14c1:b0:848:8619:c149 with SMTP id d2e1a72fcca58-8562868db75mr29260125b3a.2.1788090469003; Sun, 30 Aug 2026 04:47:49 -0700 (PDT) Received: from NV-J4GCB44.nvidia.com ([2409:8929:a7b:121:d0e3:2a32:8ea7:5ade]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-856a397c0a7sm2395264b3a.49.2026.08.30.04.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Aug 2026 04:47:47 -0700 (PDT) From: Jianyue Wu To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton Cc: Jianyue Wu , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 1/3] mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu() Date: Sun, 30 Aug 2026 19:47:29 +0800 Message-ID: <20260830114731.8322-2-wujianyue000@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260830114731.8322-1-wujianyue000@gmail.com> References: <20260830114731.8322-1-wujianyue000@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When a pool's last reference is dropped, __zswap_pool_empty() removes it from the pool list and schedules __zswap_pool_release(), which calls synchronize_rcu() to wait for readers before tearing the pool down. synchronize_rcu() is a synchronous, potentially long wait. Replace it with queue_rcu_work(): __zswap_pool_empty() hands the pool to queue_rcu_work(), which waits for a grace period asynchronously and then runs __zswap_pool_release() from a worker for the sleepable teardown (__zswap_pool_empty() can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it. Suggested-by: Yosry Ahmed Signed-off-by: Jianyue Wu Acked-by: Yosry Ahmed Reviewed-by: Johannes Weiner --- mm/zswap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 37f34e406c8e..0bb30e58950a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -155,7 +155,7 @@ struct zswap_pool { struct crypto_acomp_ctx __percpu *acomp_ctx; struct percpu_ref ref; struct list_head list; - struct work_struct release_work; + struct rcu_work release_work; struct hlist_node node; char tfm_name[CRYPTO_MAX_ALG_NAME]; }; @@ -379,10 +379,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool) =20 static void __zswap_pool_release(struct work_struct *work) { - struct zswap_pool *pool =3D container_of(work, typeof(*pool), - release_work); - - synchronize_rcu(); + struct zswap_pool *pool =3D container_of(to_rcu_work(work), + typeof(*pool), release_work); =20 /* nobody should have been able to get a ref... */ WARN_ON(!percpu_ref_is_zero(&pool->ref)); @@ -406,8 +404,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref) =20 list_del_rcu(&pool->list); =20 - INIT_WORK(&pool->release_work, __zswap_pool_release); - schedule_work(&pool->release_work); + INIT_RCU_WORK(&pool->release_work, __zswap_pool_release); + queue_rcu_work(system_percpu_wq, &pool->release_work); =20 spin_unlock_bh(&zswap_pools_lock); } --=20 2.43.0 From nobody Sat Sep 26 21:13:44 2026 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD3993B05B6 for ; Sun, 30 Aug 2026 11:47:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788090478; cv=none; b=HG2KqPK78PrlkuWwTWLpjUoZ/zCQg9Z2XwQjJS/3uIVjYtr5XxwHY0TTfTsPLigePkSx/5xnUDTSseiemLnjkAQkhwP4g/1y4OEQl/88dE/NRexq0/hTcERSeVHNuO13Otr7JMuTrMfy+315olxHitt4DT6BCEC7DaZP35/4uaw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788090478; c=relaxed/simple; bh=TG/tnR9sDfGM5yHrG3LWTG8fvKdEeVzEQ53MY5CmnN4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WINyXUhaBukKiXBtnCL/pkoIsnrRqp+EPPM9vVm2zqUOTuvBtVwKNFnvhYs6qa5OcuzS4xdTp+owHUNCemnsTE1xEvh/XMuiJwLuZFUQmbqeSCMUuCYr3/qUwSXNURakq/NZJLtnLXnpvHeiAvj+XsKK7wdLtcgVsHLeUygxdg4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=n7T8+gGK; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="n7T8+gGK" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-8556ec44e9aso2431016b3a.3 for ; Sun, 30 Aug 2026 04:47:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788090476; x=1788695276; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KzS/CUfq1CG/W/aMP1Q7L3GLy6yqyPsMXhDWwjppSfA=; b=n7T8+gGKrQJVRdQBw/QbdbI2rWjgP5nZhASMo75j+C2AQwYbVaukh+dLpi4qmOoFhy gerfc+ODOIRIUewwZkpGrPIiNKsMaUMbTzjXlTlHDMz55oEckwmv6GLKW2Daox57JtCr alQ7TZDSMLkkYhIdSObaYzaIOUYb2FpExjcru4xy5I5YqOlCmZWH3gbzBxhYBFtZNdez sjakXSdi86Ek2l4Km42yNKWFpeCRXJFIoQFqphA7S8I7tJpXP2C6of0Hg8D4piJpyr3c LIiJJj1+zunTo5/6etzRIx44pZwuc06NTBdphKW9dkVZ/F/EvnRVlwSDO5czgsAFko8R ZizA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788090476; x=1788695276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=KzS/CUfq1CG/W/aMP1Q7L3GLy6yqyPsMXhDWwjppSfA=; b=PUivGRWT5C9Q8aT1XDdGbxHhx44IpTyWQDn08F3zQcnEPorne4V7CUP/hpCQDQYnJD ZDXxYu+eUOz0G8zkz+rucMw9K0VIaiCZmJCdddfPc3hGE30s/P9LWt3FpfynS5bnuwaO tZSlYqsCNOIhC36eCRXtPKXORxVPhty0xe2SfO3u0qYZ7yqDJ73lD/++vtdj+5AlONC0 jB4oVRer85t/eDs7hXmJc5MW20LFILqXFGpKb2+a7tuF51R/OkOIY3xeJQrlS7X7/+zY 0nFuOkoP10YcLRmPGsgrYGd/bWEKZg3zejZKZyBWm6l2jPR3qcT/j3CyB138WcGLyYo3 V+dQ== X-Forwarded-Encrypted: i=1; AHgh+RoEFXSQrQ2HG1bzTNnyDgsjyj8y6Lso6I9DkkXJUlOlgHZ0OCNeWpp7iKrJPbc3sX/h/k0+AW4pmjw5V5Q=@vger.kernel.org X-Gm-Message-State: AFuF++mfYuhTQC3WosYbOidZ6Ij/MfpXZzLZucpzDmWzRUxPmuWply5Q XPXkWKO9azYCUpxhLci8o3a/N5HwXF1+jcYs3KxEuI3ZQQoCIV4gUR3g X-Gm-Gg: AR+sD114KSD44pQzILP4f9TNoS/EFqW2UR2wToLbD6oRZs1yU/Xp3YQFmrgfjW0m7mZ UQKi+W+J8WkXcxdi8lNPQsCRx8P3cOY1IJqqYDNZCn4B11u4p9GCat5Ofhr7wkUiyE46WbiN6Os phvNRBfoC1v8TEVU20XYxziJyz+A5Y2Pbd7zsyTGmwuc5zQ4cZyJGw9BWnzj1Wlm6kzB1YL6zcc tet5Rr7eFDjuEiYEd8OsX/qQpolK37jLFxM8Pw63B5mmNxGS8F7TjlrvJrUysyw3FV/cQFTF1W5 P3vNJDeAhkXyZnVjtTmO3Qc5w88g33D/eDBcLRBB5LAiMFB8icI0+MZtFC7YK0Ry5RlfHFlVwyK HkBeVlzxsv3dKXJPij7NsqJcnw+bOz3kkTlK+5t7bFWNGb+NcbkfbL/Ppt83cEfcUS5wz018jjp 6FTSdSdmM0PpQRqpjQhr15kZxzA3u3M0P2RW9J37VMPjGd9hGksZwkKwwq8nMPOwut2pUk3fCMb mTgwp6tibOQRLqZ+/8h5IbxgingHcOcjTqAjFsy3xvsjd8U X-Received: by 2002:a05:6a00:3311:b0:857:72ba:ff0a with SMTP id d2e1a72fcca58-85772baff9cmr19441422b3a.18.1788090475923; Sun, 30 Aug 2026 04:47:55 -0700 (PDT) Received: from NV-J4GCB44.nvidia.com ([2409:8929:a7b:121:d0e3:2a32:8ea7:5ade]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-856a397c0a7sm2395264b3a.49.2026.08.30.04.47.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Aug 2026 04:47:55 -0700 (PDT) From: Jianyue Wu To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton Cc: Jianyue Wu , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 2/3] mm/zswap: replace the zswap_pools list with a fixed pools array Date: Sun, 30 Aug 2026 19:47:30 +0800 Message-ID: <20260830114731.8322-3-wujianyue000@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260830114731.8322-1-wujianyue000@gmail.com> References: <20260830114731.8322-1-wujianyue000@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Originally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches. Hold the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and track the current pool with a separate rcu-protected pointer. Slot 0 is intentionally left unused (always NULL): a zeroed or incorrectly initialized pool index then resolves to NULL and trips a WARN rather than silently aliasing a live pool in another slot. The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. A fully-constructed pool is stored into its slot as the last step of zswap_pool_create(), so array walkers only ever observe a NULL slot or a ready pool. Pool creation is serialized by the module-wide kernel param mutex (all built-in params share one lock) and otherwise only happens during single-threaded init, so no two creators race for a slot. zswap_pools_lock still serializes the store against a retiring pool clearing its slot in __zswap_pool_empty(). Behavior change: the fixed array bounds the number of simultaneously live pools at ZSWAP_MAX_POOLS - 1 (15, since slot 0 is reserved), whereas the old list was unbounded. A pool is only live while it is the current pool or still has stored pages referencing it, and pools are reused across compressor switches, so 15 is far more than any real configuration needs. Once all slots are occupied, creating a pool for a 16th distinct compressor fails: zswap_pool_create() errors and returns NULL, and the compressor switch is rejected with -EINVAL rather than silently succeeding. The cap can be raised by increasing ZSWAP_MAX_POOLS (bounded by the u8 slot index, so up to 256). Suggested-by: Nhat Pham Suggested-by: Yosry Ahmed Signed-off-by: Jianyue Wu Acked-by: Yosry Ahmed --- mm/zswap.c | 97 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 25 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 0bb30e58950a..b3b5e2887c00 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -13,6 +13,7 @@ =20 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt =20 +#include #include #include #include @@ -154,12 +155,27 @@ struct zswap_pool { struct zs_pool *zs_pool; struct crypto_acomp_ctx __percpu *acomp_ctx; struct percpu_ref ref; - struct list_head list; struct rcu_work release_work; struct hlist_node node; + u8 idx; char tfm_name[CRYPTO_MAX_ALG_NAME]; }; =20 +#define ZSWAP_MAX_POOLS 16 +/* + * Slot 0 is intentionally never used: it stays NULL so that a zeroed or + * incorrectly initialized pool->idx resolves to NULL (and trips a WARN) + * instead of silently aliasing a live pool in another slot. + */ +#define ZSWAP_FIRST_POOL_SLOT 1 +static struct zswap_pool __rcu *zswap_pools[ZSWAP_MAX_POOLS]; +static_assert(ZSWAP_MAX_POOLS - 1 <=3D U8_MAX); +/* + * The current pool (NULL if none): an alias of one zswap_pools[] slot. + * It always holds a ref, so a pool is never retired while it is current. + */ +static struct zswap_pool __rcu *zswap_current_pool; + /* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru; =20 @@ -200,9 +216,6 @@ struct zswap_entry { static struct xarray *zswap_trees[MAX_SWAPFILES]; static unsigned int nr_zswap_trees[MAX_SWAPFILES]; =20 -/* RCU-protected iteration */ -static LIST_HEAD(zswap_pools); -/* protects zswap_pools list modification */ static DEFINE_SPINLOCK(zswap_pools_lock); /* pool counter to provide unique names to zsmalloc */ static atomic_t zswap_pools_count =3D ATOMIC_INIT(0); @@ -270,6 +283,31 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *ac= omp_ctx) acomp_ctx->buffer =3D NULL; } =20 +/* + * Publish a fully-constructed pool into a free array slot. Pool creation= is + * serialized by the module-wide kernel param mutex (all built-in params s= hare + * one lock) and only otherwise happens during single-threaded init, so no= two + * creators race for a slot. The pool is complete before it is stored, and + * zswap_pools_lock still serializes this store against a concurrent retir= ing + * pool clearing its slot in __zswap_pool_empty(), so array walkers only e= ver + * observe a NULL slot or a ready pool. + */ +static int zswap_pool_assign_slot(struct zswap_pool *pool) +{ + int i; + + guard(spinlock_bh)(&zswap_pools_lock); + for (i =3D ZSWAP_FIRST_POOL_SLOT; i < ZSWAP_MAX_POOLS; i++) { + if (!rcu_access_pointer(zswap_pools[i])) { + pool->idx =3D i; + rcu_assign_pointer(zswap_pools[i], pool); + return i; + } + } + + return -ENOSPC; +} + static struct zswap_pool *zswap_pool_create(char *compressor) { struct zswap_pool *pool; @@ -313,19 +351,29 @@ static struct zswap_pool *zswap_pool_create(char *com= pressor) if (ret) goto cpuhp_add_fail; =20 - /* being the current pool takes 1 ref; this func expects the - * caller to always add the new pool as the current pool + /* + * The initial ref keeps the pool alive while it is current. Stored + * entries take additional refs so a retired pool remains alive while + * any entries still reference it. */ ret =3D percpu_ref_init(&pool->ref, __zswap_pool_empty, PERCPU_REF_ALLOW_REINIT, GFP_KERNEL); if (ret) goto ref_fail; - INIT_LIST_HEAD(&pool->list); + + ret =3D zswap_pool_assign_slot(pool); + if (ret < 0) { + pr_err("cannot create more than %d pools\n", + ZSWAP_MAX_POOLS - ZSWAP_FIRST_POOL_SLOT); + goto slot_fail; + } =20 zswap_pool_debug("created", pool); =20 return pool; =20 +slot_fail: + percpu_ref_exit(&pool->ref); ref_fail: cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); =20 @@ -386,7 +434,6 @@ static void __zswap_pool_release(struct work_struct *wo= rk) WARN_ON(!percpu_ref_is_zero(&pool->ref)); percpu_ref_exit(&pool->ref); =20 - /* pool is now off zswap_pools list and has no references. */ zswap_pool_destroy(pool); } =20 @@ -402,7 +449,7 @@ static void __zswap_pool_empty(struct percpu_ref *ref) =20 WARN_ON(pool =3D=3D zswap_pool_current()); =20 - list_del_rcu(&pool->list); + rcu_assign_pointer(zswap_pools[pool->idx], NULL); =20 INIT_RCU_WORK(&pool->release_work, __zswap_pool_release); queue_rcu_work(system_percpu_wq, &pool->release_work); @@ -433,7 +480,8 @@ static struct zswap_pool *__zswap_pool_current(void) { struct zswap_pool *pool; =20 - pool =3D list_first_or_null_rcu(&zswap_pools, typeof(*pool), list); + pool =3D rcu_dereference_check(zswap_current_pool, + lockdep_is_held(&zswap_pools_lock)); WARN_ONCE(!pool && zswap_has_pool, "%s: no page storage pool!\n", __func__); =20 @@ -466,11 +514,12 @@ static struct zswap_pool *zswap_pool_current_get(void) static struct zswap_pool *zswap_pool_find_get(char *compressor) { struct zswap_pool *pool; + int i; =20 - assert_spin_locked(&zswap_pools_lock); - - list_for_each_entry_rcu(pool, &zswap_pools, list) { - if (strcmp(pool->tfm_name, compressor)) + for (i =3D ZSWAP_FIRST_POOL_SLOT; i < ZSWAP_MAX_POOLS; i++) { + pool =3D rcu_dereference_protected(zswap_pools[i], + lockdep_is_held(&zswap_pools_lock)); + if (!pool || strcmp(pool->tfm_name, compressor)) continue; /* if we can't get it, it's about to be destroyed */ if (!zswap_pool_tryget(pool)) @@ -495,10 +544,14 @@ unsigned long zswap_total_pages(void) { struct zswap_pool *pool; unsigned long total =3D 0; + int i; =20 rcu_read_lock(); - list_for_each_entry_rcu(pool, &zswap_pools, list) - total +=3D zs_get_total_pages(pool->zs_pool); + for (i =3D ZSWAP_FIRST_POOL_SLOT; i < ZSWAP_MAX_POOLS; i++) { + pool =3D rcu_dereference(zswap_pools[i]); + if (pool) + total +=3D zs_get_total_pages(pool->zs_pool); + } rcu_read_unlock(); =20 return total; @@ -560,7 +613,6 @@ static int zswap_compressor_param_set(const char *val, = const struct kernel_param if (pool) { zswap_pool_debug("using existing", pool); WARN_ON(pool =3D=3D zswap_pool_current()); - list_del_rcu(&pool->list); } =20 spin_unlock_bh(&zswap_pools_lock); @@ -588,15 +640,9 @@ static int zswap_compressor_param_set(const char *val,= const struct kernel_param =20 if (!ret) { put_pool =3D zswap_pool_current(); - list_add_rcu(&pool->list, &zswap_pools); + rcu_assign_pointer(zswap_current_pool, pool); zswap_has_pool =3D true; } else if (pool) { - /* - * Add the possibly pre-existing pool to the end of the pools - * list; if it's new (and empty) then it'll be removed and - * destroyed by the put after we drop the lock - */ - list_add_tail_rcu(&pool->list, &zswap_pools); put_pool =3D pool; } =20 @@ -1801,7 +1847,8 @@ static int zswap_setup(void) pool =3D __zswap_pool_create_fallback(); if (pool) { pr_info("loaded using pool %s\n", pool->tfm_name); - list_add(&pool->list, &zswap_pools); + /* zswap_pool_create() already stored the pool in its array slot. */ + rcu_assign_pointer(zswap_current_pool, pool); zswap_has_pool =3D true; static_branch_enable(&zswap_ever_enabled); } else { --=20 2.43.0 From nobody Sat Sep 26 21:13:44 2026 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F0A23AAF43 for ; Sun, 30 Aug 2026 11:48:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788090486; cv=none; b=RT4imuXGLq9FCQjlW14OG0NXhJFAJoivXkfhAPHi3TWCnaQ7tI2mpI2/AOlqXHIuze1XH16Pf+dQWEKkE27K/z2O+ZCjMFbs+mxV6djEIadJcN2wGYLNgLY4d5871wnEa5uOKXA0chiwAj1ZM3OsBuvqkJmcqvtASxTEjBs47P8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788090486; c=relaxed/simple; bh=FervXIRyw43OBYO14oTZZBDzwZVgUWsR4CRlugmFWvw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ThjP17Yk8ctHkOfZV3JDeYjUb4CGz58BB/QGOPriraAg63Hyqrm7OLqturz18jVw9v2Y0VFxA6I5NIhb8GyoCE490ccWorIUZzW1fL4v29u46uf4Fg/E5DPc5AgtxrLu6lp8ZpmHgB2dSZtQGQ8G+rxd/D+RjtnMiPXP0F6E+3Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JeDWx55n; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JeDWx55n" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-84864086bfeso2584000b3a.1 for ; Sun, 30 Aug 2026 04:48:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788090483; x=1788695283; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wZaMr9PcVum7xXG2P1QWnBBkzHTyXA9Vmtkj6+XCSfc=; b=JeDWx55nZxvoQ4S7DHIPM2EcqOLxvS82iMX9pNOA17vbubtI4C633dcRTm3wZui3sg eyh6J3e/zyMT75m4vAJb2X9mfPvYpRduaHDgb34ha7YblvIdF4DDIHyqxnQrFmAIs4+w aPOCsX7ZIje9Kn6hg9b36ZKVoTMXxXJSeycdrHEXK7L+lNRV4YJ0sUQb39R2+L2go1X9 Re+eBdeL8GK5VVaUPZl93FV8+5n4obck2s3DPzoBpsLbHva1vraYg1UytI/cybUfbrcI +sYzoKi06pJ2H3ygXDUHzBhZ0dvJrDzLkcsZgvbxC5+R6sMcrhxJHFC6ZjFNFXjPgDkZ 6HTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788090483; x=1788695283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wZaMr9PcVum7xXG2P1QWnBBkzHTyXA9Vmtkj6+XCSfc=; b=aYoVcGbxDJAkF0X0yjUvF8cx81ZXLC6tUkQkqhXRgeAQaG8KEpiCDPeTmQPt01Ty9Q eK7zb0ATuLdgSZz6NxcK9KFmJeKh/dQfgSoJdhbVx2ENbpLmsZtYWcJNRd68LQ7P2NVF Lt1D8f593/OtVZDKP7KoyK5L+XZ5QPC+o88wiOON/a7AP3GyXuemtUuPEEgz4AYfOi85 BG535D/i6k2pC+jkbt9mxCXx4/Pcg+CCNsMfvHNE3SiGgB8+VlmtMICPK3SdORHReWhy +KY5DzxmrrzZYRrMFHeefhLgV/lmeMTM45cqoBBdw6uYSHVekqhiLo+/4a3k2j2h9aI2 mE5Q== X-Forwarded-Encrypted: i=1; AHgh+RoCJX4O0ilA6TqcKVLEyalgvmI3K6veHOHAeh8SRVvJ4SaCulf3uDRmOOGEXlAHftFA4OPFSLu3E7pA6vk=@vger.kernel.org X-Gm-Message-State: AFuF++m7EKUhq9aeEAphxlKQv4AvkkIRb8FDHTZSF+csSfl1p1g3PCfj KNWVpxd7djpbLRhqWWcgA+BbRsavuG5hB0Ir+Yl2g7OK6s/ktL9UwZyr X-Gm-Gg: AR+sD12TdP0ZnpdFFWmwC20kDAP2sWyO3lZLi4DeJQ25n5yAa0eM6N9922aOfxyRYre xq75McZNTcOA2wKxobtWFMZwooYNoFEpun6l9ypw3Msm0DlNCtYPnakZ+x9q7pwd9lGBdwRPu9M uk5RekkQYiyACGE1GPONn7NZKizoLORm6MHUc08l+PMQvbE/vawrZPyGle5vUgqbryjAuwVYlE/ jLHYJ9VghNdGIaPOsJjep+UdNCIR0NPNV71J923PWpiRymRuA5GJ4FqrRQODuQ8g2D+ZoGqp/3B t2rznDuQtn6bjqMrAJ1QPukqlnaUqNVhvBXTukNhYWqwkCC6wcUPyS75S1wrx2k18kTOVqZ6CsS DixOk1cFBd8wyBQScp5fASw28h8zx741ru6cgdXfR0dbmsof1A1M6hWRSPz4+IhqtHJRqZTIJsS 3JTefG5z+3NnaSTkDCnv+E+hGP7VzQKQr7xUVHO7VVFmXadSTmWWLTmN90uGpP/+veirZVfATy+ u64Z63VWhUwZdIVhr5f0HgNCIIR2WCkwfxVCg== X-Received: by 2002:a05:6a00:1c9a:b0:84e:9257:d0c2 with SMTP id d2e1a72fcca58-8562a00149emr33552260b3a.9.1788090482772; Sun, 30 Aug 2026 04:48:02 -0700 (PDT) Received: from NV-J4GCB44.nvidia.com ([2409:8929:a7b:121:d0e3:2a32:8ea7:5ade]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-856a397c0a7sm2395264b3a.49.2026.08.30.04.47.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Aug 2026 04:48:02 -0700 (PDT) From: Jianyue Wu To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton Cc: Jianyue Wu , Chris Li , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 3/3] mm/zswap: reference the pool by index to shrink struct zswap_entry Date: Sun, 30 Aug 2026 19:47:31 +0800 Message-ID: <20260830114731.8322-4-wujianyue000@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260830114731.8322-1-wujianyue000@gmail.com> References: <20260830114731.8322-1-wujianyue000@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" struct zswap_entry is one allocation per stored page, so its size is pure overhead. It currently embeds an 8-byte pool pointer, even though the live pools now sit in a small fixed array indexed by a u8 slot number. Replace the per-entry pool pointer with that u8 slot index and resolve it through the fixed pool array. A live entry holds a reference to its pool, so the slot cannot be reused under it. The lookup therefore needs no RCU read-side section or zswap_pools_lock. The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on x86_64. This raises objs_per_slab from 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap. Suggested-by: Chris Li Signed-off-by: Jianyue Wu Acked-by: Yosry Ahmed --- mm/zswap.c | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index b3b5e2887c00..521e0187bcd1 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -198,7 +198,7 @@ static struct shrinker *zswap_shrinker; * writeback logic. The entry is only reclaimed by the writeb= ack * logic if referenced is unset. See comments in the shrinker * section for context. - * pool - the zswap_pool the entry's data is in + * pool_idx - slot of the zswap_pool that the entry's data is in. * handle - zsmalloc allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. @@ -207,12 +207,22 @@ struct zswap_entry { swp_entry_t swpentry; unsigned int length; bool referenced; - struct zswap_pool *pool; + u8 pool_idx; unsigned long handle; struct obj_cgroup *objcg; struct list_head lru; }; =20 +static struct zswap_pool *zswap_entry_pool(struct zswap_entry *entry) +{ + /* + * A live entry holds a pool reference, so the slot stays valid with no + * RCU read-side section. The !=3D 0 check marks access protected by + * the reference. A live entry never uses the reserved slot 0. + */ + return rcu_dereference_check(zswap_pools[entry->pool_idx], entry->pool_id= x !=3D 0); +} + static struct xarray *zswap_trees[MAX_SWAPFILES]; static unsigned int nr_zswap_trees[MAX_SWAPFILES]; =20 @@ -808,9 +818,13 @@ static void zswap_entry_cache_free(struct zswap_entry = *entry) */ static void zswap_entry_free(struct zswap_entry *entry) { + struct zswap_pool *pool =3D zswap_entry_pool(entry); + zswap_lru_del(entry); - zs_free(entry->pool->zs_pool, entry->handle); - zswap_pool_put(entry->pool); + if (!WARN_ON_ONCE(!pool)) { + zs_free(pool->zs_pool, entry->handle); + zswap_pool_put(pool); + } if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); obj_cgroup_put(entry->objcg); @@ -967,12 +981,15 @@ static bool zswap_compress(struct page *page, struct = zswap_entry *entry, =20 static bool zswap_decompress(struct zswap_entry *entry, struct folio *foli= o) { - struct zswap_pool *pool =3D entry->pool; + struct zswap_pool *pool =3D zswap_entry_pool(entry); struct scatterlist input[2]; /* zsmalloc returns an SG list 1-2 entries */ struct scatterlist output; struct crypto_acomp_ctx *acomp_ctx; int ret =3D 0, dlen; =20 + if (WARN_ON_ONCE(!pool)) + return false; + acomp_ctx =3D raw_cpu_ptr(pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); zs_obj_read_sg_begin(pool->zs_pool, entry->handle, input, entry->length); @@ -1008,7 +1025,7 @@ static bool zswap_decompress(struct zswap_entry *entr= y, struct folio *folio) pr_alert_ratelimited("Decompression error from zswap (%d:%lu %s %u->%d)\n= ", swp_type(entry->swpentry), swp_offset(entry->swpentry), - entry->pool->tfm_name, + pool->tfm_name, entry->length, dlen); return false; } @@ -1511,7 +1528,7 @@ static bool zswap_store_page(struct page *page, * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - entry->pool =3D pool; + entry->pool_idx =3D pool->idx; entry->swpentry =3D page_swpentry; entry->objcg =3D objcg; entry->referenced =3D true; --=20 2.43.0