From nobody Sat Jul 25 20:03:47 2026 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E325B3D413F for ; Tue, 14 Jul 2026 08:15:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784016949; cv=none; b=Oh5dI7CAMX5/uSZFU89qgDNoF22I0T9Cae5VfjvixlTDkQ5TNuKmGOqH9m7qmJKRWQGBczQlOIuVVUIVl040Omo6lVHMtgLI2DpcSz6s45zagmZlVIhyKPwcxq9HwBp5D3sWG3bvFCpnHRFbppNZwSai5H/r2Z+7ChmvPQqbuPs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784016949; c=relaxed/simple; bh=RmrTguKVjxq+QIse/ROJuq25wzI7h7cDgXR48/b+tIg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JBaHxGVrmDy13GSmH3BTI2Xc6EalN6cuIpDU3G6AIppfhnzBjzkOO2oZDdGEMC8pcAnBasurOeqFZJS5QqZe+ifhXOv6YhrRvjPrfbM7XTgAQSyucQEANhU96r2S51cpmWcvaKR7auVTygKNKZtaStt1k6e9sVU/jDy/0NKW+yQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aYcg1Nli; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aYcg1Nli" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-3811f512167so4732066a91.3 for ; Tue, 14 Jul 2026 01:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784016945; x=1784621745; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=5f2zwwfkcjVdLU+tnkGLj6NeYVD1qAWbI5zgmwCL7z4=; b=aYcg1NlidWx54BwKndl7c8L4xz4iLrhwunPLNe3R9TeGkbqXGXEavAtd7JnBfl9Dgy vw4aR6g2vk++G2E7mlI4oSnXNUFD4Eh1d+36aTH7zG7E7aI2+fRuGxb9eftQRZhfPyXc uyP6SIvPONuQ+6TWAb/4JD3AINyyh0Re9t5EdGBq5SUZQoE/QeW38PkhrDCeE8ENGolY zHFVz9dXWgVfOcBTrbTEIdu0CYWXkKEqGuwmOEbAwZCjm/mmsBNztRwYMma32kj0IV8T PHNQFpig2soLJ4T0UHQkwyuv+gvDGmQ2iCPxaMpMwweMWSArbtoFzZZ9v8EUn0mDqeP0 bKKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784016945; x=1784621745; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=5f2zwwfkcjVdLU+tnkGLj6NeYVD1qAWbI5zgmwCL7z4=; b=IV3wEjIl2oxXkM2ShIC6n06jkKCZ7l1Esaa2yAa21T0wmYmel9Wasvk51bPyVQFqz4 mYTkd+ng4M/rP7OAN8hn4QKxbRAZKcLAmmpdK7rHviXoWheAyxu+EfpZ9Qn8s3CL0lYJ oAjUIT7FcwJo22yC6+O3gu5frLIen5yu3qYntqQcUiTmoOohpFEFAAqRwL3sjq+kuLzg pKT0i/7DhqWjPQHJfcK8u3uOTsSVuYhP32o88WytR7xIjGF/mXArZDA9W3NUXrNKmj9b A9XftgqllwODEbP5spZ93mi5e9QzoB0jcO715UUuI3hQD6OKMCDWIIZxz4kkqoBtLH2a bjFw== X-Forwarded-Encrypted: i=1; AHgh+RpjnbVxe7hHHo9Dl/mZaQoZM86R/9uz2sV9VMB5Rvm+0GRDZT6bKHXx0dwr2NhKWFLcevrQXR0Hxf/n9ak=@vger.kernel.org X-Gm-Message-State: AOJu0Yxt3lg5GgLZphmlvGCGJHszHqDbvsusLgm9uhXyMLI/t5s7vOyW RP70KGRu+DZ2ZaDvUrArb370WiIza0PlNgyWqcZDwEmlI8QT1bPIa/94 X-Gm-Gg: AfdE7ckRqOXbl9yVUPEcdtnUAWUniySul6zT3wbk00Hjv09tabCrnWZh1u5dTVDO8y3 Vi+WKjFpnw4K0d6KFdH2laZS0vRuTp9TYoUtREPO3wWqkisAYAPxkbUqI6HsdQ643czG22jH2RJ h5tRiQcr0BTtO2aFr3TPzvslnX8FreSrZDy9hxEwkNQmOFGzyIgWZJOdG1fNPhh64hWISpwsibd +OXY7nvOFk2O7e43wk6dC47OF+zRDqaDeKizV3IOhqGbS7rj243D9tM814GzKS9mKAcN0VX3fMy XNOF8cg+zx+5BRLlPitGHgW9V2F+gicaK5Ix7sdaWXdBDv7s6hBTSiWKnNA6sOszOsj0FdRd3O0 i/bx/cWZ46eT3hxI9tqTmPJntKOvv48x+5zjQ3w3DFtrV3AJry7l7hJpDYFypsSqMX3Jl65usWz +2RtsgBGU8q3MMqlWP2MNl+RDtdepPvzwXtUABGOx5FKGKVaVLVkg= X-Received: by 2002:a17:90b:4a90:b0:37f:d262:1e0a with SMTP id 98e67ed59e1d1-38e1ae9fc51mr1541369a91.12.1784016944718; Tue, 14 Jul 2026 01:15:44 -0700 (PDT) Received: from localhost.localdomain ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-38e17443f4fsm1091008a91.11.2026.07.14.01.15.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jul 2026 01:15:44 -0700 (PDT) From: Hao Jia To: akpm@linux-foundation.org, tj@kernel.org, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@kernel.org, yosry@kernel.org, mkoutny@suse.com, nphamcs@gmail.com, chengming.zhou@linux.dev, muchun.song@linux.dev, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Hao Jia , stable@vger.kernel.org Subject: [PATCH 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled Date: Tue, 14 Jul 2026 16:15:09 +0800 Message-Id: <20260714081510.16895-2-jiahao.kernel@gmail.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: <20260714081510.16895-1-jiahao.kernel@gmail.com> References: <20260714081510.16895-1-jiahao.kernel@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hao Jia When memory cgroup is disabled, mem_cgroup_iter() always returns NULL. Therefore, the global shrinker shrink_worker() always takes the !memcg branch. After MAX_RECLAIM_RETRIES empty walks, the worker simply gives up, so it fails to write back anything. Therefore, when memory cgroup is disabled, fall through with the !memcg branch and shrink the root memcg directly. With memcg disabled, shrink_memcg() only returns -ENOENT when the root LRU is empty, which means the total pages are already below thr. The loop then safely bails out via the zswap_total_pages() <=3D thr check. For any other return value from shrink_memcg(), the loop is guaranteed to terminate, either after MAX_RECLAIM_RETRIES failures or once the threshold is met. Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware") Cc: stable@vger.kernel.org Suggested-by: Nhat Pham Acked-by: Nhat Pham Acked-by: Yosry Ahmed Reported-by: Yosry Ahmed Closes: https://lore.kernel.org/all/CAO9r8zPVzMKFbCixxD-qgtRrkFxWVrHiZZeLc= =3DeyTPKPVQgX4g@mail.gmail.com Signed-off-by: Hao Jia --- mm/zswap.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index b5a17ea20237..3d697a1a5365 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1356,11 +1356,12 @@ static void shrink_worker(struct work_struct *w) } while (memcg && !mem_cgroup_tryget_online(memcg)); spin_unlock(&zswap_shrink_lock); =20 - if (!memcg) { - /* - * Continue shrinking without incrementing failures if - * we found candidate memcgs in the last tree walk. - */ + /* + * A NULL memcg ends a full hierarchy pass (except when memcg is + * disabled, where it is always NULL: fall through to the root LRU). + * Count a failure only if the last pass found no candidates. + */ + if (!memcg && !mem_cgroup_disabled()) { if (!attempts && ++failures =3D=3D MAX_RECLAIM_RETRIES) break; =20 --=20 2.34.1 From nobody Sat Jul 25 20:03:47 2026 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61B523F9F5E for ; Tue, 14 Jul 2026 08:15:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784016959; cv=none; b=CZ9UWKcCNcHGSbyEkwicxzn9GlJAPYeqVfiqrbPOfUCG74MmRnH8A0XiRb4IkMFkdIRjsND3HYmhi/iA1iFKduFQgB7avc2jV6maIEYshPswPRGP2Sn285zb8iOUSBBarxIsyau6UwGe1av4aOGMAWst9ybKRClbhfBNdtp36+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784016959; c=relaxed/simple; bh=uUYlBhAZn84mwF8H+rHHQ5q26TeChr1Bxo2Zj9hIRxs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=l1QFYNHRlKTL+mmgb70u3KZ4GtwpFtzG35IIi8JpDRb5sZPCDy3Hq3QzfBuqz/8HOS4JMfPL2NZ0D0DTuDgUyiigSFrFE+VRaSNPMQOjeHPUMTQOw0taCv6U4HyADdFWezHlk1uPuShmZKngQo3MQV3oHASKhW81qyZomhoUEIY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bK2+Zgwr; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bK2+Zgwr" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-38dfe910e9dso1523658a91.3 for ; Tue, 14 Jul 2026 01:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784016955; x=1784621755; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8UXPYiyfSEkWVl+cC8sK2nhliwJgX/ylc+YW6RL4x9c=; b=bK2+ZgwriEcZxcsFZPGVeDHWH/YnTVCMUtFd6ff0Onn15zry6Qs6v6fseWCOJ1iAHL WVBr+3lpCYcscld5d1dI8uYxR0txO7oydcfabcfvlMWAytxh26S3TDBX6+7m34VurCpn ZiBSA2XbzBERk3jXqkA8ZHxLK3By2yZv73hIr1LgjPquhCHoa+lKGFeLD/ZkV2vBnBSi HyifKxTYCOQCroc8Bke03+c8ES+g1R6fx1wL90phx/BiUVroF9EhNf+tbynLkjFLpt2k ARZVJX5pf/sCdXcb6hu06q4gHa9Qq9uUhhgjLor1E8BHlPAgvQ5e5xGFS6XKVj7Jo9hk SvCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784016955; x=1784621755; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=8UXPYiyfSEkWVl+cC8sK2nhliwJgX/ylc+YW6RL4x9c=; b=Ux3UF6/3Xbrn+fKaaMnVXPYKZy30kCYJ6CSiXqQbXoebZcF5DOVJzXWW7g4DxX3jK2 rDoux3TGnKLzRbJzRkbFjYY9Zmzd5bvfg+T5OWOLLbiNv9uV2MGXBv/fmKhOVqticWVV GwM/KhqbJUaqaR0BNbrQ6QbAl5RgFKCdaKXYfHopOwS9VM62cmxo1R4Cj3EaonH8ECrb fDoT5pYscCmGtn1ydi5XLGqVFslBw9Lf0K4NCxAIGQXKjyMX2uTHSiV/U1uyIcUrXeSj 2NZxNwELSnT0wKBEWmLzbHR4Kh4+C941jTobfrH5Bi3FIbx+yZq2bhbZP+QXO/aSPz+e 6vFw== X-Forwarded-Encrypted: i=1; AHgh+Rp65HdTY7FVxVIh7eXgryxb0GE2DRuHgO5tT9mKT6VeDPBovRhrDANR2FbJyysBrlz0VTa+iGeXRRV+a4c=@vger.kernel.org X-Gm-Message-State: AOJu0YyINUxTdsTqiLlmI8wd5uDq6FuKMwJl7kFao6WGYCrOPk9zXBiU aTwt6kHleUoNypkaYqZ0W8wkQwBbFeJmdq0BeWEENTfHLqjSh/UxXsQT X-Gm-Gg: AfdE7ck8pwuaaIEIitySsvk1Bl5M1HgV4CSXfcJQOJTw3puwcud58e4MCkGSJZO885r J93fzDdQMZh0lW3BKyMqaTRVBfWDQZkGbjM4y6gS1RA8C1xj7HCR4v0R8Ue/avHUeXZrf0VdtoI f8sMz8V9QO4oNX2sgmVU5laYVTSkYupIn/Q5zIBnpKyGfeHjhnLZiqsosrcByrj9Tly462j84qy 0F54/kjTuJgBtwOkQALh/nR/EpNuOkpj2N3YsxcmzJrS+SvDUr4j5HxQdWBTlKPvwPKV3zFExtB ZNBcJg2b/dC4kqm46sT0o1XgW+O3RPtBFS2UljSsOCEf90XyjWD9ldABU29Sobyg1yt8SlIfNdw 5z3JM5x6istr2fc9+5FeKdnJISCwMfbWUasQI/FzxbdyOvv3q10/4b3dhCoq9wx/sLNbQvHgUZE 1UoTtY6DRHjIdQrHWbwltZ7n0cv15W+jvZXI5RF0tS X-Received: by 2002:a17:90b:28c4:b0:38d:fda6:4873 with SMTP id 98e67ed59e1d1-38e17db7bd5mr2555852a91.10.1784016954644; Tue, 14 Jul 2026 01:15:54 -0700 (PDT) Received: from localhost.localdomain ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-38e17443f4fsm1091008a91.11.2026.07.14.01.15.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jul 2026 01:15:54 -0700 (PDT) From: Hao Jia To: akpm@linux-foundation.org, tj@kernel.org, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@kernel.org, yosry@kernel.org, mkoutny@suse.com, nphamcs@gmail.com, chengming.zhou@linux.dev, muchun.song@linux.dev, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Hao Jia Subject: [PATCH 2/2] mm/zswap: Support batch writeback in shrink_memcg() Date: Tue, 14 Jul 2026 16:15:10 +0800 Message-Id: <20260714081510.16895-3-jiahao.kernel@gmail.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: <20260714081510.16895-1-jiahao.kernel@gmail.com> References: <20260714081510.16895-1-jiahao.kernel@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hao Jia Currently, shrink_memcg() writes back at most one entry per-node during its traversal. This makes shrink_worker() inefficient, as it must repeatedly re-enter shrink_memcg() to make any substantial progress. To address this, extend shrink_memcg() and rewrite its LRU iteration logic to support batch writeback. Introduce the nr_to_scan parameter to bound how many pages are scanned per call. This enables batch writeback in the shrink_worker() path, while maintaining a low scan budget in the zswap_store() path. Additionally, to prepare for future proactive writeback, update the return value semantics of shrink_memcg(): a positive value now represents the actual number of compressed bytes written back, 0 indicates that candidates existed but no writeback succeeded, and a negative value represents an error code. Test Setup: Total memory: 32 GB. zswap settings: max_pool_percent=3D1, accept_threshold_percent=3D50, shrinker_enabled=3DN. Allocate 512MB of anonymous pages and fill them with random data (to avoid compression), then use cgroup memory.reclaim to force a large amount of anonymous pages into zswap. At an interval of 2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers and the rest are zeros, and then trigger a reclamation of this 4K anonymous page through cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg() will be triggered. The test data after running for 120s is as follows: Baseline Patched shrink_worker wakeups 5363 85 shrink_memcg calls 11,345,012 188,264 written_back 40214 40275 Conclusion: Under the same workload and run duration, the patched kernel shows a significant reduction in both shrink_worker wakeups and shrink_memcg calls. Suggested-by: Yosry Ahmed Signed-off-by: Hao Jia --- mm/zswap.c | 89 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 3d697a1a5365..6d492762957a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -160,6 +160,11 @@ struct zswap_pool { char tfm_name[CRYPTO_MAX_ALG_NAME]; }; =20 +struct zswap_shrink_walk_arg { + unsigned long bytes_written; + bool encountered_page_in_swapcache; +}; + /* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru; =20 @@ -1089,8 +1094,9 @@ static enum lru_status shrink_memcg_cb(struct list_he= ad *item, struct list_lru_o void *arg) { struct zswap_entry *entry =3D container_of(item, struct zswap_entry, lru); - bool *encountered_page_in_swapcache =3D (bool *)arg; + struct zswap_shrink_walk_arg *walk_arg =3D arg; swp_entry_t swpentry; + unsigned int length; enum lru_status ret =3D LRU_REMOVED_RETRY; int writeback_result; =20 @@ -1133,10 +1139,11 @@ static enum lru_status shrink_memcg_cb(struct list_= head *item, struct list_lru_o =20 /* * Once the lru lock is dropped, the entry might get freed. The - * swpentry is copied to the stack, and entry isn't deref'd again - * until the entry is verified to still be alive in the tree. + * needed fields are copied to the stack, and entry isn't deref'd + * again until it is verified to still be alive in the tree. */ swpentry =3D entry->swpentry; + length =3D entry->length; =20 /* * It's safe to drop the lock here because we return either @@ -1155,12 +1162,13 @@ static enum lru_status shrink_memcg_cb(struct list_= head *item, struct list_lru_o * into the warmer region. We should terminate shrinking (if we're in th= e dynamic * shrinker context). */ - if (writeback_result =3D=3D -EEXIST && encountered_page_in_swapcache) { + if (writeback_result =3D=3D -EEXIST) { ret =3D LRU_STOP; - *encountered_page_in_swapcache =3D true; + walk_arg->encountered_page_in_swapcache =3D true; } } else { zswap_written_back_pages++; + walk_arg->bytes_written +=3D length; } =20 return ret; @@ -1169,8 +1177,11 @@ static enum lru_status shrink_memcg_cb(struct list_h= ead *item, struct list_lru_o static unsigned long zswap_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) { + struct zswap_shrink_walk_arg walk_arg =3D { + .bytes_written =3D 0, + .encountered_page_in_swapcache =3D false, + }; unsigned long shrink_ret; - bool encountered_page_in_swapcache =3D false; =20 if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(sc->memcg)) { @@ -1179,9 +1190,9 @@ static unsigned long zswap_shrinker_scan(struct shrin= ker *shrinker, } =20 shrink_ret =3D list_lru_shrink_walk(&zswap_list_lru, sc, &shrink_memcg_cb, - &encountered_page_in_swapcache); + &walk_arg); =20 - if (encountered_page_in_swapcache) + if (walk_arg.encountered_page_in_swapcache) return SHRINK_STOP; =20 return shrink_ret ? shrink_ret : SHRINK_STOP; @@ -1275,9 +1286,31 @@ static struct shrinker *zswap_alloc_shrinker(void) return shrinker; } =20 -static int shrink_memcg(struct mem_cgroup *memcg) +#define NR_ZSWAP_WB_BATCH 64UL + +/* + * Scan up to @nr_to_scan pages across the per-node zswap LRUs of @memcg + * and write back the reclaimable ones. + * + * Since the second-chance algorithm rotates referenced entries to the + * LRU tail, the per-node scan is capped at the current LRU length so + * each entry is scanned at most once per call. It is up to the caller + * to handle retries, deciding whether to scan another memcg to complete + * the full iteration, or to rescan the current memcg to drain its zswap + * entries. + * + * Return: The number of compressed bytes written back (>=3D 0), or -ENOENT + * if @memcg has writeback disabled, is a zombie cgroup, or has empty + * zswap LRUs. + */ +static long shrink_memcg(struct mem_cgroup *memcg, unsigned long nr_to_sca= n) { - int nid, shrunk =3D 0, scanned =3D 0; + struct zswap_shrink_walk_arg walk_arg =3D { + .bytes_written =3D 0, + .encountered_page_in_swapcache =3D false, + }; + unsigned long nr_remaining =3D nr_to_scan; + int nid; =20 if (!mem_cgroup_zswap_writeback_enabled(memcg)) return -ENOENT; @@ -1290,24 +1323,40 @@ static int shrink_memcg(struct mem_cgroup *memcg) return -ENOENT; =20 for_each_node_state(nid, N_NORMAL_MEMORY) { - unsigned long nr_to_walk =3D 1; + unsigned long nr_to_walk; =20 - shrunk +=3D list_lru_walk_one(&zswap_list_lru, nid, memcg, - &shrink_memcg_cb, NULL, &nr_to_walk); - scanned +=3D 1 - nr_to_walk; + /* + * Cap the scan at per-node LRU length so each entry is scanned + * at most once per call. + */ + nr_to_walk =3D min(nr_remaining, + list_lru_count_one(&zswap_list_lru, nid, memcg)); + if (!nr_to_walk) + continue; + + nr_remaining -=3D nr_to_walk; + list_lru_walk_one(&zswap_list_lru, nid, memcg, &shrink_memcg_cb, + &walk_arg, &nr_to_walk); + /* Return the unused share of the budget to the pool. */ + nr_remaining +=3D nr_to_walk; + + if (!nr_remaining) + break; } =20 - if (!scanned) + /* Nothing was scanned: every LRU under @memcg was empty. */ + if (nr_remaining =3D=3D nr_to_scan) return -ENOENT; =20 - return shrunk ? 0 : -EAGAIN; + return walk_arg.bytes_written; } =20 static void shrink_worker(struct work_struct *w) { struct mem_cgroup *memcg; - int ret, failures =3D 0, attempts =3D 0; + int failures =3D 0, attempts =3D 0; unsigned long thr; + long ret; =20 /* Reclaim down to the accept threshold */ thr =3D zswap_accept_thr_pages(); @@ -1369,7 +1418,7 @@ static void shrink_worker(struct work_struct *w) goto resched; } =20 - ret =3D shrink_memcg(memcg); + ret =3D shrink_memcg(memcg, NR_ZSWAP_WB_BATCH); /* drop the extra reference */ mem_cgroup_put(memcg); =20 @@ -1383,7 +1432,7 @@ static void shrink_worker(struct work_struct *w) continue; ++attempts; =20 - if (ret && ++failures =3D=3D MAX_RECLAIM_RETRIES) + if (ret <=3D 0 && ++failures =3D=3D MAX_RECLAIM_RETRIES) break; resched: cond_resched(); @@ -1493,7 +1542,7 @@ bool zswap_store(struct folio *folio) objcg =3D get_obj_cgroup_from_folio(folio); if (objcg && !obj_cgroup_may_zswap(objcg)) { memcg =3D get_mem_cgroup_from_objcg(objcg); - if (shrink_memcg(memcg)) { + if (shrink_memcg(memcg, 1) <=3D 0) { mem_cgroup_put(memcg); goto put_objcg; } --=20 2.34.1