From nobody Sat Jul 25 05:28:43 2026 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 089A733688D for ; Fri, 17 Jul 2026 08:52:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784278336; cv=none; b=A3WimC8iQoBjppyADI7nLF4ZuXIhwDJINL7OscsQCuEOgCbNneAHupx3BNfN3g/mpMv+bcIQC5o4qB2Hq20KWTtFI+R/6gwWLS68KBtGdZhIiyvjUePhJX6SLvley7uktZ6KwxYs28Hq3fjuK1zKRoIZxqMzhY0Xdq1Ql9hMpOE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784278336; c=relaxed/simple; bh=h+3gZPmYltKvxFoFTsqU186dmpWa1sBnf0q8P6lIN2Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XGmZ0MwTkI46mFp5aBVe8x2ktKr9zIYYkO39SfGSCqQwmoP6E5daIWhzZtT7nHRczyLOXOWormEVKrMUNQ5SzaMB4xC5BGTezgvlTerQg9bXDsuR9jRHX3bZCINKEqlSTdQq2f3SvrP1HG7kWKDW5BR+I+SuBdm2nrZhk6tD974= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YZsKJnPD; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YZsKJnPD" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-84867f07d63so8658146b3a.2 for ; Fri, 17 Jul 2026 01:52:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784278333; x=1784883133; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=idUPlycWgvJqE3e2FFLxSJRzfsl5YsmQQbKWafiBiP4=; b=YZsKJnPDx7BRgVQN8tx8LCKvVRiZyVM9hO0csYNTWKFUD2ArbYi8ZD/TcQiLFj/Dyu 2jkOkynUsmbg5gXc4U3IJhIUH2qg7OJXSvTyKeg4cTerK6wTHLdTdJIJR9nP//CZ+1Db WvMDqbUNESP2spw9/TiwtsQpnKrs9kO6aeoxhkTbXsw9KW5p6BSU2T0hYPTfUpUhRCNm r+AtgwD0oXYiKY+0urymQ6K2XGL5Zwa+2teUWSuIh2rUhWLUn4NatV5xxsTH2hea+NXu V+FTgB+kzOXCSPibjRWmK3b6/gOlrftTKPsSaae0Y48W4py0X3N4nYEZ4XfkEpANF4J3 FqYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784278333; x=1784883133; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=idUPlycWgvJqE3e2FFLxSJRzfsl5YsmQQbKWafiBiP4=; b=WSIX/DypyOns9Uwj7qhTE85sHY8Ax0i9cBUO/y7O6bBN54z2c9QqrgcRrJqHeWSJSl 9u5WAvRExczz0/hCN6G0MBva3p/JuZEgkuSghnFq4LxbfX6SbCF8NxMyMN3lJGWV6rKt J3VoB1J0LT46ignrx8RiFn5tqjMezhNVIjXxoeyJHnCVpAAuLG/bI1QI8pxzv/e/MLdG SmUMxRyx+IvrHoQAFrGh+DJx9naYqvjxpoYCGrOmsSyjdFxZekKXCE2lNnEci15FiQ78 ztxIqv2bSitzDvgFd9h9gx/YH19hLlST5DeNhdlFqzxIcOUBT1hPWL2Nc5P892M23w2e 37Sg== X-Forwarded-Encrypted: i=1; AHgh+Rq4WH1/Umiq+IUPHHM2tQ7KUjS4llpbS6+bCgb1p5d9OQPwO6eijMyxk+seW9t+969KbWwxnfPOc1PtuKo=@vger.kernel.org X-Gm-Message-State: AOJu0YxPtAPcuOYTdUngkysicxGQ0P9PNvZavBmCjlEGqhszyN+Bdkqw HCSaMDB1vT+i/F4vkgid/TWjYCvyqOGLTm4XtyXgO4/Nc1lrToCL7vnE X-Gm-Gg: AfdE7cmetBYoEYyUiV+gTYZNqOBkKbki1JBfItm8DRIJClKRMDyAzOqAhQSsu+CSWef EQN0JDDOxSZUJyUcjsrm8z9XhewzPYjiqM1w8eQlZYn0vWGeroxdnSD4Nu1HTwu84T9s/4eB8YW 6NA3M7RRDqGxko0qd2L8OSb4Z5NvJaK5HGWI3xtL3YnUdElvlnMHwWdH1CiVNkSE6HM5RWg1TQH xnd+iNDDNOHGr0QzOXLu7zjS7udonw7Vz1YQIaB63dDqQcVfXmX59gMNw+W9WMqyD4le2Qzvkz8 d5+azQ9McIoXlKrKLc3z4cX7iiHzwWH8HIJkSjdq+20WskUJEkD9xVnS+OD4+ReGuV+1bydTJ8b J+G9UnrxZWMol70LRR/eIjHyrS9MZF69boK6X8MPSZU3mCO6rT3Bvwy+Gr4Fpma6F3Ntx8Hw0cQ BNULt88RNDXBxC5GA1F/DxU2Gi64NPyA== X-Received: by 2002:a05:6a00:21d2:b0:84a:29a7:f650 with SMTP id d2e1a72fcca58-84c29093c61mr1731239b3a.0.1784278333446; Fri, 17 Jul 2026 01:52:13 -0700 (PDT) Received: from localhost.localdomain ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84c2af9f477sm619474b3a.58.2026.07.17.01.52.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 17 Jul 2026 01:52:12 -0700 (PDT) From: Hao Jia To: akpm@linux-foundation.org, tj@kernel.org, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@kernel.org, yosry@kernel.org, mkoutny@suse.com, nphamcs@gmail.com, chengming.zhou@linux.dev, muchun.song@linux.dev, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Hao Jia , stable@vger.kernel.org Subject: [PATCH v2 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled Date: Fri, 17 Jul 2026 16:51:50 +0800 Message-Id: <20260717085151.22822-2-jiahao.kernel@gmail.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: <20260717085151.22822-1-jiahao.kernel@gmail.com> References: <20260717085151.22822-1-jiahao.kernel@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hao Jia Zswap writeback on hitting the pool limit is broken when memory cgroup is disabled, because mem_cgroup_iter() always returns NULL. Therefore, the global shrinker shrink_worker() always takes the !memcg branch. After MAX_RECLAIM_RETRIES empty walks, the worker simply gives up, so it fails to write back anything. Therefore, when memory cgroup is disabled, fall through with the !memcg branch and shrink the root memcg directly. With memcg disabled, shrink_memcg() only returns -ENOENT when the root LRU is empty, which means the total pages are already below thr. In the absence of heavy concurrent zswap stores, the loop then safely bails out via the zswap_total_pages() <=3D thr check; otherwise, it will resume shrinking the memcg after processing the reschedule check. For any other return value from shrink_memcg(), the loop is guaranteed to terminate, either after MAX_RECLAIM_RETRIES failures or once the threshold is met. Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware") Cc: stable@vger.kernel.org Suggested-by: Nhat Pham Acked-by: Nhat Pham Acked-by: Yosry Ahmed Reported-by: Yosry Ahmed Closes: https://lore.kernel.org/all/CAO9r8zPVzMKFbCixxD-qgtRrkFxWVrHiZZeLc= =3DeyTPKPVQgX4g@mail.gmail.com Signed-off-by: Hao Jia Acked-by: Johannes Weiner --- mm/zswap.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index b5a17ea20237..48fc7b575e24 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1356,11 +1356,12 @@ static void shrink_worker(struct work_struct *w) } while (memcg && !mem_cgroup_tryget_online(memcg)); spin_unlock(&zswap_shrink_lock); =20 - if (!memcg) { - /* - * Continue shrinking without incrementing failures if - * we found candidate memcgs in the last tree walk. - */ + /* + * A NULL memcg ends a full hierarchy pass (except when memcg is + * disabled, where it is always NULL: fall through to the root LRU). + * Count a failure only if the last pass found no candidates. + */ + if (!memcg && !mem_cgroup_disabled()) { if (!attempts && ++failures =3D=3D MAX_RECLAIM_RETRIES) break; =20 @@ -1379,7 +1380,7 @@ static void shrink_worker(struct work_struct *w) * and failures. */ if (ret =3D=3D -ENOENT) - continue; + goto resched; ++attempts; =20 if (ret && ++failures =3D=3D MAX_RECLAIM_RETRIES) --=20 2.34.1 From nobody Sat Jul 25 05:28:43 2026 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E51BA3793A5 for ; Fri, 17 Jul 2026 08:52:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784278343; cv=none; b=mqwxJxOCz6zYXUmIidftWb+NO499mjHjhZH3XeSg1+ZEqrNNPPrlza5R8FSUq5eOUFEb+kYnCNxKNygPYO1kmN/8nhE9HwHqawLvIPvlinKbX/YMCyZQt1JttyZooSkg0/ZjXevfQ055nqu/P/6+8VOAqrj8ONrlg837fscM/Gc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784278343; c=relaxed/simple; bh=xaylzQGm3XlQeVpD4W6mIwMTYmCT1luwxC2820fJ9F0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t96UsnVf5QNA0tLqhwKIhXbAqts6xp+MnOaudNrQ9B1TMPmcIFv62s0eu/xW5+uhhYV1TBZyilHqTaCN8d9TLGoS+us0wDZE+E368CkKw5p8b2ZyisLUuyMUKFzrU9TAFUa/r39cBfVJJ4ACt824u2VUUVSBsgQvcHVNfcwi2Yo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=c630a9GX; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c630a9GX" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-8487214ad2bso5871766b3a.1 for ; Fri, 17 Jul 2026 01:52:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784278341; x=1784883141; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=VAaayER3b5hTUVDck3FSxcWjhNxHvBfs9GbGsiNXdFQ=; b=c630a9GXW/NVTncgLhskhN505yrAIadBRmpVTe1ndFYVkUUlLMOjpDwsFkHYSKy56m w2R0JM44kx6tomrxjFbZPrRvzlKVniftlvo3SLFHQaKjWHiqgDLWh7RWMoIckyBZmSjJ nNfEogtw4DcPG1fTzbBlsXPhlBf9Jp41o61wwyAsGLjG0lJOm5DRWloF5iyM9OyAoDAc T4h6ZZaWRaai2N1BMy6alnwGBPGmn7lOgAiV2tWziT//lZh+GVqBe+MpKJow04F7mvhj RFpFkJkVlWr64lMgNNFnCiiPWobSUhlxKG04skdeuXpz8+CjNDReAl+g3MrzKOk0AVtR 56bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784278341; x=1784883141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=VAaayER3b5hTUVDck3FSxcWjhNxHvBfs9GbGsiNXdFQ=; b=p0dbeBVyhB+Upy40xbBIS1QTzJKYr3x3gUMKpZzLDcz71uTC/BWJK1sckV9VRKignU I6lmYYNcEvdWu7dLgAytn9H4Z/GDbnuD5FI618+qGU/f7XfKK3XMQvupq9dAqJ+DU2jF whGJj55E0Is9HTyOkswUloLLwoEMSqHNpQzIb8smwm4VRs+0FkK6/8uhCFxUb7GX5kCO nfEFSwaXT2RNipszZwzZmPMmsK9pS1tY/6ZlhMYODNsSS6fT4pDLPn4GHygqGrAdL5+R qKDK8bH1DfH/uImf8t4GYzI5u8fOpS3fF24m6Kf1K8UD0X+sbsc1HcZulsaBh/h9UX2f P9zQ== X-Forwarded-Encrypted: i=1; AHgh+RqIbvlMByd9K9UNyOFKIt246HCB6l+kbm7FYn5sAIUZr3Nejtahm1EjpqZAbcGsvXj/A6NsxyEfF8t/RDw=@vger.kernel.org X-Gm-Message-State: AOJu0YwzGlo4f2NaoHdRw92n1C1z39liB3zCgwTjeHeyW7nTN885TI66 TJVi1QihXP18LRuoPEk62HGtMSQ7C6zTUf+zzXjeJLLj6LzfhNWhq2CE X-Gm-Gg: AfdE7ckCEOsfSJEfx1C5I0EnrAkjqgDqH3JEh19uB/oimcrzIXMTLXlJf3DvYev6wiQ ITBwGLHGl1x9oAq5Cre5i6JsOSfKspQk3IHFD0VCf4MzDDE+IepAS+k/bUkF8Py53cj506U6qCF in0BmAaNELfIV925y2j6quXICgm5YbLdRhNqlCeA+cEb+MjzbJLuHKsMERLeQBeHjeW8xQGLP3S o9fYGOfpRzduokacLFZuBcghGjSpfqE/ng3GwU9p2eUfNBKV/zl3ryHV9Iao803OJ+2fvbrbGtl 4REXnU0zeAXlmHCV4gYo03KlTM3QN+2qOJ4iXxgEyAuj5/lCWXDDtyCHDeprrMGWPSCRZzLv8JZ EKvXceuQIoN7qq9FKEhHff7QR4Q+tpThwXUVwg0bvOdh6MgMKtJo92eQ+Y1bbcI6Hu8SPB1aMiD n8aXSsYtnev/ryVlCzSeCMhQh1IfsGBQ== X-Received: by 2002:a05:6a00:44c5:b0:845:e08c:a4eb with SMTP id d2e1a72fcca58-84c29519471mr1855968b3a.58.1784278341157; Fri, 17 Jul 2026 01:52:21 -0700 (PDT) Received: from localhost.localdomain ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-84c2af9f477sm619474b3a.58.2026.07.17.01.52.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 17 Jul 2026 01:52:20 -0700 (PDT) From: Hao Jia To: akpm@linux-foundation.org, tj@kernel.org, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@kernel.org, yosry@kernel.org, mkoutny@suse.com, nphamcs@gmail.com, chengming.zhou@linux.dev, muchun.song@linux.dev, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Hao Jia Subject: [PATCH v2 2/2] mm/zswap: Support batch writeback in shrink_memcg() Date: Fri, 17 Jul 2026 16:51:51 +0800 Message-Id: <20260717085151.22822-3-jiahao.kernel@gmail.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: <20260717085151.22822-1-jiahao.kernel@gmail.com> References: <20260717085151.22822-1-jiahao.kernel@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hao Jia Currently, shrink_memcg() writes back at most one entry per-node during its traversal. This makes shrink_worker() inefficient, as it must repeatedly re-enter shrink_memcg() to make any substantial progress. Under high memory pressure, this can cause the writeback speed to be too slow to keep up with refaults, leading to zswap store failures and forcing pages to skip zswap and go directly to disk, which results in an LRU inversion. To address this, extend shrink_memcg() and rewrite its LRU iteration logic to support batch writeback. Introduce the nr_to_scan parameter to bound how many pages are scanned per call. This enables batch writeback in the shrink_worker() path, while maintaining a low scan budget in the zswap_store() path. Test Setup: - Total memory: 32 GB. - zswap settings: max_pool_percent=3D1, accept_threshold_percent=3D50, shrinker_enabled=3DN. Test Case 1: Allocate 512MB of anonymous pages and fill them with random data (to avoid compression), then use cgroup memory.reclaim to force a large amount of anonymous pages into zswap. At an interval of 2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers and the rest are zeros, and then trigger a reclamation of this 4K anonymous page through cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg() will be triggered. The test data after running for 120s is as follows: Baseline Patched shrink_worker wakeups 5,363 85 shrink_memcg calls 11,373,201 180,928 written_back pages 40,212 40,236 zswap_store calls 161,190 168,741 store succeeded (ret=3D1) 102,743 127,644 store rejected (ret=3D0) 58,447 41,097 store reject rate ~36% ~24% pool_limit_hit delta 55,826 14,062 pswpout 98,659 81,333 pswpin 2 1 Test Case 2: To consistently force zswap store failures and trigger shrink_worker(), the following stress-ng command was run for 120 seconds within a cgroup limited to a memory.max of 1G: bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \ exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \ 120s -q' The test data after running for 120s is as follows: Baseline Patched shrink_worker wakeups 5,640 987 shrink_memcg calls 8,481,500 2,504,818 written_back pages 260 768,576 zswap_store calls 2,742,756 2,301,414 store succeeded (ret=3D1) 934,640 1,308,686 store rejected (ret=3D0) 1,808,116 992,728 store reject rate ~66% ~43% pool_limit_hit delta 1,181,310 101,593 pswpout 1,808,376 1,761,304 pswpin 4,288,497 3,902,658 Under identical workloads and runtimes, batching the zswap shrinker exhibits a significant reduction in both shrink_worker wakeups and shrink_memcg calls. Furthermore, the sharp drop in both pool_limit_hit and zswap_store rejections demonstrates that batching the zswap shrinker effectively mitigates zswap_store failures caused by hitting the pool limit. This significantly prevents pages from bypassing zswap and falling back directly to disk, thereby reducing LRU inversion. Suggested-by: Yosry Ahmed Signed-off-by: Hao Jia Acked-by: Nhat Pham Acked-by: Yosry Ahmed --- mm/zswap.c | 47 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 40 insertions(+), 7 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 48fc7b575e24..6a09b9ebfd25 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1275,9 +1275,27 @@ static struct shrinker *zswap_alloc_shrinker(void) return shrinker; } =20 -static int shrink_memcg(struct mem_cgroup *memcg) +#define NR_ZSWAP_WB_BATCH 64UL + +/* + * Scan up to @nr_to_scan pages across the per-node zswap LRUs of @memcg + * and write back the reclaimable ones. + * + * Since the second-chance algorithm rotates referenced entries to the + * LRU tail, the per-node scan is capped at the current LRU length so + * each entry is scanned at most once per call. It is up to the caller + * to handle retries, deciding whether to scan another memcg to complete + * the full iteration, or to rescan the current memcg to drain its zswap + * entries. + * + * Return: 0 if at least one entry was written back, -EAGAIN if entries + * were scanned but none could be written back, or -ENOENT if @memcg has + * writeback disabled, is a zombie cgroup, or has empty zswap LRUs. + */ +static int shrink_memcg(struct mem_cgroup *memcg, unsigned long nr_to_scan) { - int nid, shrunk =3D 0, scanned =3D 0; + unsigned long nr_remaining =3D nr_to_scan; + int nid, shrunk =3D 0; =20 if (!mem_cgroup_zswap_writeback_enabled(memcg)) return -ENOENT; @@ -1290,14 +1308,29 @@ static int shrink_memcg(struct mem_cgroup *memcg) return -ENOENT; =20 for_each_node_state(nid, N_NORMAL_MEMORY) { - unsigned long nr_to_walk =3D 1; + unsigned long nr_to_walk; =20 + /* + * Cap the scan at per-node LRU length so each entry is scanned + * at most once per call. + */ + nr_to_walk =3D min(nr_remaining, + list_lru_count_one(&zswap_list_lru, nid, memcg)); + if (!nr_to_walk) + continue; + + nr_remaining -=3D nr_to_walk; shrunk +=3D list_lru_walk_one(&zswap_list_lru, nid, memcg, &shrink_memcg_cb, NULL, &nr_to_walk); - scanned +=3D 1 - nr_to_walk; + /* Return the unused share of the budget to the pool. */ + nr_remaining +=3D nr_to_walk; + + if (!nr_remaining) + break; } =20 - if (!scanned) + /* Nothing was scanned: every LRU under @memcg was empty. */ + if (nr_remaining =3D=3D nr_to_scan) return -ENOENT; =20 return shrunk ? 0 : -EAGAIN; @@ -1369,7 +1402,7 @@ static void shrink_worker(struct work_struct *w) goto resched; } =20 - ret =3D shrink_memcg(memcg); + ret =3D shrink_memcg(memcg, NR_ZSWAP_WB_BATCH); /* drop the extra reference */ mem_cgroup_put(memcg); =20 @@ -1493,7 +1526,7 @@ bool zswap_store(struct folio *folio) objcg =3D get_obj_cgroup_from_folio(folio); if (objcg && !obj_cgroup_may_zswap(objcg)) { memcg =3D get_mem_cgroup_from_objcg(objcg); - if (shrink_memcg(memcg)) { + if (shrink_memcg(memcg, 1)) { mem_cgroup_put(memcg); goto put_objcg; } --=20 2.34.1