From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC6E884A26 for ; Mon, 25 Mar 2024 23:50:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410624; cv=none; b=PzUptR7EonH+S4b7ro7XSbnW9Vn6OAHl6wPjcMVqgFvbe0RVy6HS1N99y+eOTgD9aZLZP+h/02fzPm6bHgxdG+M/vNPuA9Fr+NhB2XjXcmmE9h5nR4MY9LXoc/JfiM0wTt0MJ82rv/YkJSivojPQWJCj4wvubXReLBC5QimMmpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410624; c=relaxed/simple; bh=6xxUCt1k7C+LwXDbzCekm9e6ZfMrsxua/6M5UyPsHns=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LgFjXUwKO2kp/U0wp5gfR8k99qpHqyRdPWB/mRWzYWlEL+LMZB91JfeFUukYIAYyFJf2Twa90qMNJhxW8pNhr6RFfJ2QHgEsgRUZjX8gmWENa5uKbaQ/H6yEoR+FknwFx0GQBh99WWFezMjdFLPKs0xdIAuBRRINevYk7g1yzZc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kjRJCKjx; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kjRJCKjx" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a03635590so99284487b3.0 for ; Mon, 25 Mar 2024 16:50:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410622; x=1712015422; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u9fvbqW1pLtPErlE8l2YKQeiKLLE5wFNh7avewbFYWs=; b=kjRJCKjxUN2aU1haa67W+vz+lyNDM3VM4OeLHTfKotzkn3jMo3uakiw2baap7rZ0Bw EN4tb8BK9AaV1Kgpzh2/4PWKOdgnOyXu7gHZQOrn8RGOyH9h8k0Kt0VrBBEOi65PskXs +KM3arrq/7cFfwMDFXFS2UgkzxKMTRe/Dyw00T+zcIrQu/DAk7fmFMQTLCPL6ApyOJ/c Acm2aASX7xFDapHAauKyj7477grt++HAYeEp3H759KS55WxqvnK3bxSMwhd5XoHb5jHG 1c3Be2qgoM42YOAtwyPRcCiHLcePuAidbIfPGrvgBOiIJXxsmLb18j6RCfaarjmDab8o 0d2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410622; x=1712015422; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u9fvbqW1pLtPErlE8l2YKQeiKLLE5wFNh7avewbFYWs=; b=tFiM2ZzIqmr1RDPhDJUF0L6CZYjdYWE1icLNADF1MdcY0ewDkJSQ060OZ+OElYvfX2 Db8zU+eSlZF56NsF/0c37Z610IdvbItT9gxYbolLzc2n5u5SywUD0jkQZCr4llkmHdEx gqlnC8AiKsETxH+hOlJ1mE0qHeR/uyO+SVCv/eSdxR4NHZvKJq6yytMbmvETfYNU9TF3 ObihNcuQRF9EYsRQzyQz5sW5BXOUJJJR0O+/PKyIcS37R9gGEP9jae+ZAFhDjPN5xa5z 7nw6GjqIuqf20lnFhsPak6TYISGUahxiJ8rome69WIh21uV3uysWByHrTperuUAbP/7e WTgQ== X-Forwarded-Encrypted: i=1; AJvYcCWt9tYZ/99jf9cxGAk/hMveLz33tk8Tr7YTq07/7M2QHRxNoBysugPv3UZ75WU3mZkTU5ozgBj4aFmO57okx21G1GCVQaN0qTNmKEYE X-Gm-Message-State: AOJu0YyNPivweKkekyZB+VgN7y/b3hYIHF1J8GLqFlJUDLIVxYJCW/rF CAPneKZZM3CqqcXRdw9peoS3+W9QH+twg8wZdRI2F9IpCj5W7dgsVWk8GFzfyMueyHDwaxdGanH PeNNprfbpCaGKIcLn6g== X-Google-Smtp-Source: AGHT+IH+DXWV+vX4OULsfH+Uyq0gzOaLllxrZEK/FYDaAYjBCM9JZ11SOX35Q3oKMnkT3VL0MGvjY9eZ8pU6LA5R X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:2491:b0:dcb:e982:4e40 with SMTP id ds17-20020a056902249100b00dcbe9824e40mr2590953ybb.12.1711410621894; Mon, 25 Mar 2024 16:50:21 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:09 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-2-yosryahmed@google.com> Subject: [RFC PATCH 1/9] mm: zswap: always shrink in zswap_store() if zswap_pool_reached_full From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The cleanup code in zswap_store() is not pretty, particularly the 'shrink' label at the bottom that ends up jumping between cleanup labels. Instead of having a dedicated label to shrink the pool, just use zswap_pool_reached_full directly to figure out if the pool needs shrinking. zswap_pool_reached_full should be true if and only if the pool needs shrinking. The only caveat is that the value of zswap_pool_reached_full may be changed by concurrent zswap_store() calls between checking the limit and testing zswap_pool_reached_full in the cleanup code. This is fine because: - If zswap_pool_reached_full was true during limit checking then became false during the cleanup code, then someone else already took care of shrinking the pool and there is no need to queue the worker. That would be a good change. - If zswap_pool_reached_full was false during limit checking then became true during the cleanup code, then someone else hit the limit meanwhile. In this case, both threads will try to queue the worker, but it never gets queued more than once anyway. Also, calling queue_work() multiple times when the limit is hit could already happen today, so this isn't a significant change in any way. Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Reviewed-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index c4979c76d58e3..1cf3ab4b22e64 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1429,12 +1429,12 @@ bool zswap_store(struct folio *folio) if (cur_pages >=3D max_pages) { zswap_pool_limit_hit++; zswap_pool_reached_full =3D true; - goto shrink; + goto reject; } =20 if (zswap_pool_reached_full) { if (cur_pages > zswap_accept_thr_pages()) - goto shrink; + goto reject; else zswap_pool_reached_full =3D false; } @@ -1540,6 +1540,8 @@ bool zswap_store(struct folio *folio) zswap_entry_cache_free(entry); reject: obj_cgroup_put(objcg); + if (zswap_pool_reached_full) + queue_work(shrink_wq, &zswap_shrink_work); check_old: /* * If the zswap store fails or zswap is disabled, we must invalidate the @@ -1550,10 +1552,6 @@ bool zswap_store(struct folio *folio) if (entry) zswap_entry_free(entry); return false; - -shrink: - queue_work(shrink_wq, &zswap_shrink_work); - goto reject; } =20 bool zswap_load(struct folio *folio) --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29CD284D3F for ; Mon, 25 Mar 2024 23:50:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410626; cv=none; b=GTCVtWoPUrSXK6O2CxFlhWW2EvAAsBNaPf1vigbt1hcaAIA3zd+xAU0UAnqt6V8SZ2L6Y7wqwseVf0uZD5DTRUQwVxiXgtUAHc8BeJ56QnV8XUXj1/lHc8BwNyBAFfV+1cAaaDFI+Gaoh6DP6WZik+fP4qlk4L2goB4bgio3j9M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410626; c=relaxed/simple; bh=N3EhBhoPx4G/jPI5J5rhJXw3XGLAmDgO8lD2yh8kUnA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LLs4WW54tOkyLEuGC46DuEPR2lc7xqi1ZtfAB0xM10pjI2t4IHZ+c2xRm6hiHbbVR/EF8Zu0dBV0ZbzrpbdeG3GOnF1rMKG4JY1dZD7yCGfZTGS62RVNJqdym14ovTnXhxgmBeoTiWRp510DXpm8mYPQnH9Bl5ucNKQQoEFMSI0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Vp16IZDx; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Vp16IZDx" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60cd041665bso94042327b3.0 for ; Mon, 25 Mar 2024 16:50:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410623; x=1712015423; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ltPcwa62HBgtxDU26adJs5Ek9NcA4wCnN/irpaDy45s=; b=Vp16IZDx2OImL9QaaZ0imlCmnw9vXM5OYNlINxsa4YnCWqmJ9xwV6d3k8naBoOR0ip EisHnAx16oZV9zIj8JYHk96Cr+i2H9uHUnTeHMlZ79NN+7hs6gmXpCMiMWaZLZTQqFxu AAtWf7dHc3DvIhfU906cRcwp8hEHXSSFAcGqbrkAsHD4FWOBK17hmkDNMXg0Ydpe/nF2 WBcwRWtOD3iNfOr4gQsHAJ4febC8g06uPAJBFWmphABqpx0puMnnuQUzpvRCiD/p64RH qLGYnuCkQXBaNAG30R1CatkU/CFMawTO9u4ua+QtW9HAXxzdnfRXKy2D/lwh6RqCi/Pu VULQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410623; x=1712015423; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ltPcwa62HBgtxDU26adJs5Ek9NcA4wCnN/irpaDy45s=; b=CyzH3Kt0MghGHSZBd70h1GogOqnh8DrFDlr2BEX7JfSMUwmEglt4TDb3Pbg3exwBB1 GnBSxerQ+JMzLM5YdkafBdBN5iDDg6UBv3LiyR3z8vn0dDd9CFRkozGu8qz+8wAuGcOy ZpdfyK6qJs1RKq1RstMOKam74jmMdrHl36bmJ5yzA1OmunFMeCQgX4dGxZxv4N1eQ8OX 55qik2KAd2dmnnSgb1jstsLPCWlGIYF+5MEoOA1sQenyKu6T6FDU9pB4O1jDk71vCk2D 5DhVOKFWYDUmJeJOgJjm41lI1mUwG087SD/4jvdTvarCQvPBoBTkbYh2WyZL9aCAcD4X BmfA== X-Forwarded-Encrypted: i=1; AJvYcCWcS2QVP9uRZBMgTSl3IpTdhqhBpQTrEhu9pPpOEhGgvdyHIRO1CaHqQ7VxpP7/VO240Ux55XjLKUbCMCcKOnPc50pEk2Q+WfXBdBqf X-Gm-Message-State: AOJu0Yz4rR/n6T4OOh0dgLm1PkCM2g5XrTtxNKA4pbnlFgDEb4wt4Iup MUdcN4T7O9Wb5sxPYXBwhD+JPjAnW4o2xwzc/od/2wy2wWbb3VD8L1A9/NL+JOXosmwvPeMnVOK xoo3dRDDy8nmaAcV/qw== X-Google-Smtp-Source: AGHT+IETdRLWcKvQU7ALPH/FxGMfk6VVMBGbdetkJernCRp2EMtMjfpphTVvqJ9nLnh8UXcxjIFAV/8JU9lCEqEU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a0d:eb04:0:b0:610:d0c9:696b with SMTP id u4-20020a0deb04000000b00610d0c9696bmr2096495ywe.8.1711410623435; Mon, 25 Mar 2024 16:50:23 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:10 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-3-yosryahmed@google.com> Subject: [RFC PATCH 2/9] mm: zswap: refactor storing to the tree out of zswap_store() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor the code that attempts storing to the xarray, handling erros, and freeing stale entries into a helper. This will be reused in a following patch to free other types of tree elements as well. Signed-off-by: Yosry Ahmed --- mm/zswap.c | 42 ++++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 1cf3ab4b22e64..ff1975afb7e3d 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -827,6 +827,30 @@ static void zswap_entry_free(struct zswap_entry *entry) atomic_dec(&zswap_stored_pages); } =20 +/********************************* +* zswap tree functions +**********************************/ +static int zswap_tree_store(struct xarray *tree, pgoff_t offset, void *new) +{ + void *old; + int err; + + old =3D xa_store(tree, offset, new, GFP_KERNEL); + err =3D xa_is_err(old); + if (err) { + WARN_ONCE(err !=3D -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + } else if (old) { + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + zswap_entry_free(old); + } + return err; +} + /********************************* * compressed storage functions **********************************/ @@ -1396,10 +1420,10 @@ bool zswap_store(struct folio *folio) swp_entry_t swp =3D folio->swap; pgoff_t offset =3D swp_offset(swp); struct xarray *tree =3D swap_zswap_tree(swp); - struct zswap_entry *entry, *old; struct obj_cgroup *objcg =3D NULL; struct mem_cgroup *memcg =3D NULL; unsigned long max_pages, cur_pages; + struct zswap_entry *entry; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1485,22 +1509,8 @@ bool zswap_store(struct folio *folio) entry->swpentry =3D swp; entry->objcg =3D objcg; =20 - old =3D xa_store(tree, offset, entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err =3D xa_err(old); - - WARN_ONCE(err !=3D -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; + if (zswap_tree_store(tree, offset, entry)) goto store_failed; - } - - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); =20 if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D19EA8529B for ; Mon, 25 Mar 2024 23:50:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410627; cv=none; b=ne0jXDpCCgUJyv5qGt1IDXr7zmscQ1MzXD4l5cexhPwIsPnRL+25uK+M0dx731FGYtangESIvdeEC7uflixPG8sNeIiPHt5C4rIhj2HmCwsTQT/EO8GpQ/iZ2duwJjH+kXwsL7h7AJXYn/nO5+iXVPGoZDo44Hwz5lnXU0IBSBQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410627; c=relaxed/simple; bh=etnStzb5dr7nTQEyqbOx6RG6Z1pSCvmhvSaa+izLU2U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=H+XG/9sAvxoPykaYXndY42QRlk+ctdnPeFfWPZ20fseks0SvEXPqwAD2NzZyGZd4ohbbI6eNbB8GG5xl2A7IfkeknBj4hUEQcPjkgnARRZ5Rzza1KHL98yETiX12ZbNhtI8n8mkjYVGgizQgXArHHzGhIrKAQxcbrlOIthiqwHY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VxI7pGjc; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VxI7pGjc" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60cd073522cso90178107b3.1 for ; Mon, 25 Mar 2024 16:50:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410625; x=1712015425; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Dd0Yn1LXqAraa72sbpCkcMKeXtgN9BdSqsW0A73YyKg=; b=VxI7pGjcjYhioSrcG2qVE/9DFD6rkzzKii1IIJyxUhqeeDn/zxovXsPm0NSL6LYHxl O3upUhYB9CAj81qc0pPHosgD5xB7VzP0bprB3lF1FG31d6hDgJ6oaFmgDj3uoLxR70hb O4gldPeGSVo9hapDHQbvsH3ulgLki/AwOolj5J8Rrq1u4qm4xH/smhN95jBMXGPyzHPA VGSrOwAf3lLAgf3smZKOCYbEiN0CxNHa2ZV3lh1feRMTUy5E9OHWNYuy3nOi0+H/Cfb9 Lh6xVO5urrBCpWU3VA+0EPlsucF0TNq/ef7Dj22WxPP0nR4elw58voywjZotUh+ynH8h 6W2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410625; x=1712015425; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dd0Yn1LXqAraa72sbpCkcMKeXtgN9BdSqsW0A73YyKg=; b=eaMa5ldOXc6zuDEHiE0M2sM/7lyI8l4XP+6FXUUFBCDRwAqimcdeHP8qU1YFdMqGrn LX+ikQqRCcipFxXTY0/dM2KrDbJ2msZW9ZXPTMkeEQXfjP/pVRgSHdnKs+BvSIR7OvRM rBirIz+JO/xilIyWPdd9ukyJw/Dqk25wX9NattIe9SOdJ8Hp9VxaEsg0NFEEESTWlrYX S7b2gdDUAE4L8DnPaGUNoK4+fWfF8Bm5gVNAigbkXDJqy0/7ueMo0JxvY6q81cfYl5cR UKcIo6v/79O/MTJ154RM0xjCaTNrsNh3I51VJ4Wz8IaZXPMt/+QjrO9DshYmYzWw4ugT XvWQ== X-Forwarded-Encrypted: i=1; AJvYcCXovWg+wVnLrIbrO6Raf5sKIZbpgIIhrseIEDSIan5wnCHVRHjEO4ql4BHadJhxwa5L/FFc9q+1Sn+wmHcTYlGG24wlYiL4vsRZTFiz X-Gm-Message-State: AOJu0YzhyG7xVk12F8EgG53nOwXKMhO4sDtAbdfB0JEO3N6oBJ2Xrrif 2FBffV+sWP1a7KkpOgdlN8yPscR05ESE1IhBvBD4Bwpcf2gcknnDlav5FWG0NAxu61PSGouvgUF vADSkOusIPoDap7mFQw== X-Google-Smtp-Source: AGHT+IHM4PjIMqVasHvxcW/NHELy1n3EZw0GsOSiULCSozqmucZP5yLYhHluyhz301rMxFhkk5EZKcifeTwjvilx X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a0d:fe44:0:b0:611:a84c:fef2 with SMTP id o65-20020a0dfe44000000b00611a84cfef2mr165827ywf.10.1711410624906; Mon, 25 Mar 2024 16:50:24 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:11 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-4-yosryahmed@google.com> Subject: [RFC PATCH 3/9] mm: zswap: refactor limit checking from zswap_store() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor limit and acceptance threshold checking outside of zswap_store(). This code will be moved around in a following patch, so it would be cleaner to move a function call around. Signed-off-by: Yosry Ahmed --- mm/zswap.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index ff1975afb7e3d..6b890c8590ef7 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1415,6 +1415,21 @@ static void zswap_fill_page(void *ptr, unsigned long= value) memset_l(page, value, PAGE_SIZE / sizeof(unsigned long)); } =20 +static bool zswap_check_limit(void) +{ + unsigned long cur_pages =3D zswap_total_pages(); + unsigned long thr =3D zswap_accept_thr_pages(); + unsigned long max_pages =3D zswap_max_pages(); + + if (cur_pages >=3D max_pages) { + zswap_pool_limit_hit++; + zswap_pool_reached_full =3D true; + } else if (zswap_pool_reached_full && cur_pages <=3D thr) { + zswap_pool_reached_full =3D false; + } + return !zswap_pool_reached_full; +} + bool zswap_store(struct folio *folio) { swp_entry_t swp =3D folio->swap; @@ -1422,7 +1437,6 @@ bool zswap_store(struct folio *folio) struct xarray *tree =3D swap_zswap_tree(swp); struct obj_cgroup *objcg =3D NULL; struct mem_cgroup *memcg =3D NULL; - unsigned long max_pages, cur_pages; struct zswap_entry *entry; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1446,22 +1460,8 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } =20 - /* Check global limits */ - cur_pages =3D zswap_total_pages(); - max_pages =3D zswap_max_pages(); - - if (cur_pages >=3D max_pages) { - zswap_pool_limit_hit++; - zswap_pool_reached_full =3D true; + if (!zswap_check_limit()) goto reject; - } - - if (zswap_pool_reached_full) { - if (cur_pages > zswap_accept_thr_pages()) - goto reject; - else - zswap_pool_reached_full =3D false; - } =20 /* allocate entry */ entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65C9286634 for ; Mon, 25 Mar 2024 23:50:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410628; cv=none; b=lRA0is4D++XaephA8RnhlNzRSQIp0/0r9jnelEUgAZluDuSksKGSn0K5to5D3Pnp2zndDLmh30nalkGynIwDJ+fHX/q1/ReVNWykyVptNH7h3zmJ/FFQljqArMBOTJd6SIM552fLcuh2JoBsuEzeJpl05itBSqBUO9Aq3nwL7uA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410628; c=relaxed/simple; bh=wNdfsIJSEt5cPySFHvOGJeIj3izJ/ngdP5B8j4GiEoc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=smXi13iPi3no5/H4cjF4cQmumHramSz5Pl4pd1U7Qghf7zaVxdAFmUNY3lA3Xgzz3Fw7TI31VVs+qDxFd5rBiYZcsdBlcuujJkWEKOzL4i5GwEWKdlmKN6C51hCQHFJt25LPpprQEJP3EO5igI9vphO2RjhnWtXAjjfTO1LkXTA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xRmxL7y9; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xRmxL7y9" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcd94cc48a1so7800325276.3 for ; Mon, 25 Mar 2024 16:50:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410626; x=1712015426; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+3t/0JKEdL4mJ/vvKrn6t1zV3oNIlUMv4pczMFevQsk=; b=xRmxL7y9CJkqpPOM5WwWgGekrwDkSD3dR70jQdPdERU+EKQAO20kzXLXcdUsicisP+ te9VbC4tVwmj5zZFQU/U2vl4mWRrZCx9t4rYxceZu+rxSLCi0meWxNXVH0cTt5vcqel9 zKx1JFMdlqS3szNCpMKpcIfAGZ+vCvMtpfy4q24gHgj2gZblG+A2Vhkw7vGJoVxFEGdh ict3+W650Zra+qi8DAPwytSvXFZJpiHOgd4YSQjXoX86yBAQEGkJtLrU6sDTxbgMAZFA WQvedOwnAeSuzUUIRpecGbdHwCXsj9jMz0afRer5u0xrs4pTyHXE/iIQnZ9WxWum4CdG XI6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410626; x=1712015426; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+3t/0JKEdL4mJ/vvKrn6t1zV3oNIlUMv4pczMFevQsk=; b=iMyMYsaOidsNVCxF69tRE9Jkr1xaGeAFo5BnBQLYUG1PrHwMqo0YD6B/TtLki4pOYs MaRFhl+kvESvImXyyOLZjBjXe/Ix7BMd2MmUXUz8ZU1qGczTxkWwjKfbuXPjPZCbu8FC hmWBY1lmy9lHoXZD4l0GihYalemJjQ95dG2v81//p4eryTd296Sbs07BPi4hi0fWK2Kk g1LEXeZIWK5rho0lVRWOjeq/bwVufklUxJcEbzXmBhu0DnDjB8k5ADfU3dMrlP60s+5P ceKrCixYQIKGge8bqbhWNrboNw3GQOc/OizLhzyrCNnI9uEyHkusi9wHskZlJbHr50ov OVPw== X-Forwarded-Encrypted: i=1; AJvYcCV17+cjAh+gd2VT3Ix0JeqAQcTA74ldJzVpOefQ1yKW9rqjcbr4sDv4tatE7G6343hTLNr9nwTXualbGPDuhCiLCT232W9vC0xsbHRV X-Gm-Message-State: AOJu0YyFcYQdfeX5Y3dVaX8M9E1O7nVv7v4Sd3ZP2YE6Tk5u4mWbTRVR UoMjfXmOh6b6XKY6tMeXypoUPrIqajgK0WsLNKrPhiwtidbv4K95wlT7+hiZeuf7W12L4E9jS4f ki3Ks6p10ijYWGdBymQ== X-Google-Smtp-Source: AGHT+IGK+u8xbKSjmKMyfk167gHIuEXTDEJgFDPVRTLSrDn75a8RPhmRcxr7pMVWHrDhD2ZGkE5Banp7zX8fMMUb X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:18cd:b0:dc6:fec4:1c26 with SMTP id ck13-20020a05690218cd00b00dc6fec41c26mr2628277ybb.1.1711410626525; Mon, 25 Mar 2024 16:50:26 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:12 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-5-yosryahmed@google.com> Subject: [RFC PATCH 4/9] mm: zswap: move more same-filled pages checks outside of zswap_store() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, zswap_store() check zswap_same_filled_pages_enabled, kmaps the folio, then calls zswap_is_page_same_filled() to check the folio contents. Move this logic into zswap_is_page_same_filled() as well (and rename it to use 'folio' while we are at it). This makes zswap_store() cleaner, and makes following changes to that logic contained within the helper. Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 45 ++++++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 6b890c8590ef7..498a6c5839bef 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1385,26 +1385,36 @@ static void shrink_worker(struct work_struct *w) } while (zswap_total_pages() > thr); } =20 -static int zswap_is_page_same_filled(void *ptr, unsigned long *value) +static bool zswap_is_folio_same_filled(struct folio *folio, unsigned long = *value) { unsigned long *page; unsigned long val; unsigned int pos, last_pos =3D PAGE_SIZE / sizeof(*page) - 1; + bool ret; =20 - page =3D (unsigned long *)ptr; + if (!zswap_same_filled_pages_enabled) + return false; + + page =3D kmap_local_folio(folio, 0); val =3D page[0]; =20 - if (val !=3D page[last_pos]) - return 0; + if (val !=3D page[last_pos]) { + ret =3D false; + goto out; + } =20 for (pos =3D 1; pos < last_pos; pos++) { - if (val !=3D page[pos]) - return 0; + if (val !=3D page[pos]) { + ret =3D false; + goto out; + } } =20 *value =3D val; - - return 1; + ret =3D true; +out: + kunmap_local(page); + return ret; } =20 static void zswap_fill_page(void *ptr, unsigned long value) @@ -1438,6 +1448,7 @@ bool zswap_store(struct folio *folio) struct obj_cgroup *objcg =3D NULL; struct mem_cgroup *memcg =3D NULL; struct zswap_entry *entry; + unsigned long value; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1470,19 +1481,11 @@ bool zswap_store(struct folio *folio) goto reject; } =20 - if (zswap_same_filled_pages_enabled) { - unsigned long value; - u8 *src; - - src =3D kmap_local_folio(folio, 0); - if (zswap_is_page_same_filled(src, &value)) { - kunmap_local(src); - entry->length =3D 0; - entry->value =3D value; - atomic_inc(&zswap_same_filled_pages); - goto insert_entry; - } - kunmap_local(src); + if (zswap_is_folio_same_filled(folio, &value)) { + entry->length =3D 0; + entry->value =3D value; + atomic_inc(&zswap_same_filled_pages); + goto insert_entry; } =20 if (!zswap_non_same_filled_pages_enabled) --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47A5284D15 for ; Mon, 25 Mar 2024 23:50:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410630; cv=none; b=kaJ+6KgOyxxQjwJTg1JiHbrlSoiBXU3ZDrcmoKnZfx7e3r/0ZFFZ5kZEtYaI/l+PGaGc9wNJF66Sta2JUcIv4iH3HYtq36PdRv0yY0Sp99Pkh89ER6NJ1jB1wcw8sOINaFzxKXUWPhUXFPJsKT4/frIutnQrGNwBO8hp8hDuHEQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410630; c=relaxed/simple; bh=4PhLDyJKyNrd+eY0Bpr57YddoLSPy5v21IAWqWdH8Io=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SmoVo300jR2+ERraC4pmEyQHewnXT78y62EjwZN0vAKeW9LeZ2meOH9IHiG0CBdi8nWNmyQH+uwFrZ2nEzVa9eRZESCSP1Ln1QJg1SFyBt/VgR7kT3TxhsT/GAa1MQJGm6ky5kZyaxnEJSfDEtg0R2zYRql4+yltXehjmxYOSQ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=t4py96hw; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="t4py96hw" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60a0815e3f9so75220987b3.2 for ; Mon, 25 Mar 2024 16:50:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410628; x=1712015428; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6rS+9vqlodSMLG0LOLi3GYviKuQ0ce9GmEUN+BZASUg=; b=t4py96hwnBcj231C7LOLMDhDlXIwVLiBAj/vTVUrgybSbjxxQ1ng3xZ7zeIS2F5CJK WD5iW6fiDK5HwWwpCi9rBSE+tMoQvsIUJjno4GFYZDxiJerFSmq3EI1SRbL09LbrC56Y bNzsCpjzOfEDfluDHWmJSzdTHcCjo9xPWgaPRAgrP1Vnao1hSZBxCgw9oKQQ3Iq4SMkn 0wRg8t8K/BJQAXcYa8a03hOO8btO0GJijd5YRpDbq629KPwiRW8C4cAH2ux4KtA8KdrF NZ9o7LndH8Fl1Ke8HgcoWzsTAIJf8eMkNSb/fxg+Gowr18qROXjvhegO3z+6qczw0wq2 rT7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410628; x=1712015428; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6rS+9vqlodSMLG0LOLi3GYviKuQ0ce9GmEUN+BZASUg=; b=m4O5Tq+g6HEeX8mdq5N6hXp1m1S3WotxwPXWcMB1Uhio1sgRDoXjf/v1uK9BIPlQ3W PxY0ls/V3U8NW/Wrvmd7Acw9BtsjND1Tn+cpvf6ErD54Ib8OFMGATuq6cqaCZtg6kLty +x221H2EM0oDRTSZAIcCXv6WX8RZSiAY7Dmy0zjdZIAqK30WEMBx520SQHJ86m9B0fsW FhkWqHwejcS7L134Li8GHxf198VmZqjEQFVoBmsWDRA40XaRrEZKnkVPjLwLh9KgeqmQ 4vtKkiW87zQXEvr685W2sMV/eODrlG7ygMbt7AJBt7OBqAN5UPK+B8R7LOQmeLb9mdW4 I2Hg== X-Forwarded-Encrypted: i=1; AJvYcCV29AQMrN9QucxFhgTtf89VFdQEvSgOTXwefJifR6xr98nga7EPR3Qk8rQhck5qM6NrBk4+Mb2oINLbJyQN6TPsls2FLIJYRlVRsAm/ X-Gm-Message-State: AOJu0Yw1qqM648WII/augGjCnFPmFVH5JazHMC5Qr9DHFp62pqGQ8/bY tNmHmR2a4FAfv8QuMgefnSRcCJEGLT5kRDW5CNLT6aRa76dINb6dknVQtt+WHASMxtY84xGKG8X gLHg+IlNuImelYWeGNg== X-Google-Smtp-Source: AGHT+IGKv5EFMnzQrdwPt+dy51PJVNwrFBzsXHDzxlAniqnpd5xrXPYCvCMKdfAD9j2nV5K8TiJsS4lrUK/HiNL3 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:e82:b0:dc6:dfd9:d423 with SMTP id dg2-20020a0569020e8200b00dc6dfd9d423mr371033ybb.3.1711410628291; Mon, 25 Mar 2024 16:50:28 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:13 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-6-yosryahmed@google.com> Subject: [RFC PATCH 5/9] mm: zswap: remove zswap_same_filled_pages_enabled From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no logical reason to refuse storing same-filled pages more efficiently and opt for compression. Remove the userspace knob. Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Reviewed-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 498a6c5839bef..0fc27ae950c74 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -123,14 +123,6 @@ static unsigned int zswap_accept_thr_percent =3D 90; /= * of max pool size */ module_param_named(accept_threshold_percent, zswap_accept_thr_percent, uint, 0644); =20 -/* - * Enable/disable handling same-value filled pages (enabled by default). - * If disabled every page is considered non-same-value filled. - */ -static bool zswap_same_filled_pages_enabled =3D true; -module_param_named(same_filled_pages_enabled, zswap_same_filled_pages_enab= led, - bool, 0644); - /* Enable/disable handling non-same-value filled pages (enabled by default= ) */ static bool zswap_non_same_filled_pages_enabled =3D true; module_param_named(non_same_filled_pages_enabled, zswap_non_same_filled_pa= ges_enabled, @@ -1392,9 +1384,6 @@ static bool zswap_is_folio_same_filled(struct folio *= folio, unsigned long *value unsigned int pos, last_pos =3D PAGE_SIZE / sizeof(*page) - 1; bool ret; =20 - if (!zswap_same_filled_pages_enabled) - return false; - page =3D kmap_local_folio(folio, 0); val =3D page[0]; =20 --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75425127B62 for ; Mon, 25 Mar 2024 23:50:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410633; cv=none; b=biClwxuSCXCBmTbUKwTfMLgA7l3+X8iMwdIHsAKGaF9m9rs3Er8pIUJDoSQLZ3Bp1QDKAmrSMG4OgEtfebGJc3t3J6MCTxOS7DPbwWBFrBNoRqMud8vWkJhhHnaxoaR0dg+Aq/XKZI/svoTxc6eqa1YbHHm2oLmFnVRLbxfEyL4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410633; c=relaxed/simple; bh=ms64h5+7StXPRldy0PEcIOWgPm2l+gGBLc2kLYJLR1M=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LE6+FqSd1ZdrmgLAK2MC/TI8StlLRqilk7y6jBgl05pcymrTGhQb2KBmnCIei3Sq0mvODCFj2dkMK2Auc9znpecH1MZD6jpcDUO6ULd1NY4Z6bigN8O58J/Kk4gBk6kXkawMaLpwIGYY+QlhddNdpmt7FxtvpqDVkd5XJSlDtq8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XbE/R+nq; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XbE/R+nq" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcc0bcf9256so6169641276.3 for ; Mon, 25 Mar 2024 16:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410630; x=1712015430; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=XbE/R+nqRlBZWK8wzY0OUcAleromwQSEwlCj8re6CMPOKtNVk5b5DMNqTIZajbwmma sZGqGXoKgfHkzsg+dYVY2dQBnxEHijBx/Yt9gMHfPPOh57JOGMEvabBYxyfc0z1gIXe1 tkJyW4SKfm8lkxUBATx7BhjpLRPgBL0v4bk2V7o7/vxWYKDoiaNuntan9sItk8jc3HxO kMHXxTaeI8QZJiSryeh12g11TNoj7VUcRvN3UfE3bRt0NOQ6u5fb9UJTn+HK1IUUrAv2 htwYa7eeDqoRwUromp207jyFMQpPkxGVMpxlH6V1kekA8GTiXycK2nEnoEr575tbeEbg Gb9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410630; x=1712015430; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=YdXu1LDA5jMzjQHU5WvEJjP0fas9lGoXzu3483u7VSz8ctTKe5GFWnOjhbUZxHZ36X r3rkqLq3Nv2wu0BUHS4k58Cy39m5/dVrkFa8vWmfkYaBHUjVjgc1EGa/pPbOYubd0WpT h+Jml22fiZxpQFxroB7peZHux5sk/uVdiOXHDEFWl+E+Tlho+Emta9EYUaUoqaIFRBV2 xTdOqXYwEWrJSlUeooJc5bJTUA0H01XVzv+FK5oE7ls9OF/Btjq9Iu09p6Ebi24FaHao Yt0yoBRX2bbfktTJ7cz3FK+sWW/8fRcfltk3GeXDOYK3IUfS5nMmOeWniCNVZ/xsL+nz avoA== X-Forwarded-Encrypted: i=1; AJvYcCVY38t5dhXAQLuQ47A1ELoexp62kZllmYFtRMq3Hv83g5mFo+yIfL1hx+xgXOCtSGUfcmw6cvpQJxKsA6zvJXldr0nh7b76jOjye2c8 X-Gm-Message-State: AOJu0Yy+Oi63J5DQNIQXLAuBV3a3dJbwR4Xh8etyUJpV3hgwxF2RBGEJ vQR+Z98N7ZxgfYqGIFMzT+uBDItmbeWJjTlXiKsmDVyu+jifMlpYyWmmPsZ8e8SWJBRhLYuDrSF VOU7qwrbq+AHAqed5JQ== X-Google-Smtp-Source: AGHT+IEOvQVqxv2Y0ouP0PCwiv8cpiRAswb4b3lmfWQXPAhEkV469xAOUnqSRda9DyCx6d0T4Sypd2qE7f/9L3CU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:705:b0:dcd:59a5:7545 with SMTP id k5-20020a056902070500b00dcd59a57545mr451342ybt.10.1711410629980; Mon, 25 Mar 2024 16:50:29 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:14 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-7-yosryahmed@google.com> Subject: [RFC PATCH 6/9] mm: zswap: drop support for non-zero same-filled pages handling From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current same-filled pages handling supports pages filled with any repeated word-sized pattern. However, in practice, most of these should be zero pages anyway. Other patterns should be nearly as common. Drop the support for non-zero same-filled pages, but keep the names of knobs exposed to userspace as "same_filled", which isn't entirely inaccurate. This yields some nice code simplification and enables a following patch that eliminates the need to allocate struct zswap_entry for those pages completely. There is also a very small performance improvement observed over 50 runs of kernel build test (kernbench) comparing the mean build time on a skylake machine when building the kernel in a cgroup v1 container with a 3G limit: base patched % diff real 70.167 69.915 -0.359% user 2953.068 2956.147 +0.104% sys 2612.811 2594.718 -0.692% This probably comes from more optimized operations like memchr_inv() and clear_highpage(). Note that the percentage of zero-filled pages during this test was only around 1.5% on average, and was not affected by this patch. Practical workloads could have a larger proportion of such pages (e.g. Johannes observed around 10% [1]), so the performance improvement should be larger. [1]https://lore.kernel.org/linux-mm/20240320210716.GH294822@cmpxchg.org/ Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 76 ++++++++++++++---------------------------------------- 1 file changed, 20 insertions(+), 56 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 0fc27ae950c74..413d9242cf500 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -44,8 +44,8 @@ **********************************/ /* The number of compressed pages currently stored in zswap */ atomic_t zswap_stored_pages =3D ATOMIC_INIT(0); -/* The number of same-value filled pages currently stored in zswap */ -static atomic_t zswap_same_filled_pages =3D ATOMIC_INIT(0); +/* The number of zero-filled pages currently stored in zswap */ +static atomic_t zswap_zero_filled_pages =3D ATOMIC_INIT(0); =20 /* * The statistics below are not protected from concurrent access for @@ -123,9 +123,9 @@ static unsigned int zswap_accept_thr_percent =3D 90; /*= of max pool size */ module_param_named(accept_threshold_percent, zswap_accept_thr_percent, uint, 0644); =20 -/* Enable/disable handling non-same-value filled pages (enabled by default= ) */ -static bool zswap_non_same_filled_pages_enabled =3D true; -module_param_named(non_same_filled_pages_enabled, zswap_non_same_filled_pa= ges_enabled, +/* Enable/disable handling non-zero-filled pages (enabled by default) */ +static bool zswap_non_zero_filled_pages_enabled =3D true; +module_param_named(non_same_filled_pages_enabled, zswap_non_zero_filled_pa= ges_enabled, bool, 0644); =20 /* Number of zpools in zswap_pool (empirically determined for scalability)= */ @@ -187,11 +187,10 @@ static struct shrinker *zswap_shrinker; * * swpentry - associated swap entry, the offset indexes into the red-black= tree * length - the length in bytes of the compressed page data. Needed during - * decompression. For a same value filled page length is 0, and b= oth + * decompression. For a zero-filled page length is 0, and both * pool and lru are invalid and must be ignored. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data - * value - value of the same-value filled pages which have same content * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. */ @@ -199,10 +198,7 @@ struct zswap_entry { swp_entry_t swpentry; unsigned int length; struct zswap_pool *pool; - union { - unsigned long handle; - unsigned long value; - }; + unsigned long handle; struct obj_cgroup *objcg; struct list_head lru; }; @@ -805,7 +801,7 @@ static struct zpool *zswap_find_zpool(struct zswap_entr= y *entry) static void zswap_entry_free(struct zswap_entry *entry) { if (!entry->length) - atomic_dec(&zswap_same_filled_pages); + atomic_dec(&zswap_zero_filled_pages); else { zswap_lru_del(&zswap_list_lru, entry); zpool_free(zswap_find_zpool(entry), entry->handle); @@ -1377,43 +1373,17 @@ static void shrink_worker(struct work_struct *w) } while (zswap_total_pages() > thr); } =20 -static bool zswap_is_folio_same_filled(struct folio *folio, unsigned long = *value) +static bool zswap_is_folio_zero_filled(struct folio *folio) { - unsigned long *page; - unsigned long val; - unsigned int pos, last_pos =3D PAGE_SIZE / sizeof(*page) - 1; + unsigned long *kaddr; bool ret; =20 - page =3D kmap_local_folio(folio, 0); - val =3D page[0]; - - if (val !=3D page[last_pos]) { - ret =3D false; - goto out; - } - - for (pos =3D 1; pos < last_pos; pos++) { - if (val !=3D page[pos]) { - ret =3D false; - goto out; - } - } - - *value =3D val; - ret =3D true; -out: - kunmap_local(page); + kaddr =3D kmap_local_folio(folio, 0); + ret =3D !memchr_inv(kaddr, 0, PAGE_SIZE); + kunmap_local(kaddr); return ret; } =20 -static void zswap_fill_page(void *ptr, unsigned long value) -{ - unsigned long *page; - - page =3D (unsigned long *)ptr; - memset_l(page, value, PAGE_SIZE / sizeof(unsigned long)); -} - static bool zswap_check_limit(void) { unsigned long cur_pages =3D zswap_total_pages(); @@ -1437,7 +1407,6 @@ bool zswap_store(struct folio *folio) struct obj_cgroup *objcg =3D NULL; struct mem_cgroup *memcg =3D NULL; struct zswap_entry *entry; - unsigned long value; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1470,14 +1439,13 @@ bool zswap_store(struct folio *folio) goto reject; } =20 - if (zswap_is_folio_same_filled(folio, &value)) { + if (zswap_is_folio_zero_filled(folio)) { entry->length =3D 0; - entry->value =3D value; - atomic_inc(&zswap_same_filled_pages); + atomic_inc(&zswap_zero_filled_pages); goto insert_entry; } =20 - if (!zswap_non_same_filled_pages_enabled) + if (!zswap_non_zero_filled_pages_enabled) goto freepage; =20 /* if entry is successfully added, it keeps the reference */ @@ -1532,7 +1500,7 @@ bool zswap_store(struct folio *folio) =20 store_failed: if (!entry->length) - atomic_dec(&zswap_same_filled_pages); + atomic_dec(&zswap_zero_filled_pages); else { zpool_free(zswap_find_zpool(entry), entry->handle); put_pool: @@ -1563,7 +1531,6 @@ bool zswap_load(struct folio *folio) struct page *page =3D &folio->page; struct xarray *tree =3D swap_zswap_tree(swp); struct zswap_entry *entry; - u8 *dst; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); =20 @@ -1573,11 +1540,8 @@ bool zswap_load(struct folio *folio) =20 if (entry->length) zswap_decompress(entry, page); - else { - dst =3D kmap_local_page(page); - zswap_fill_page(dst, entry->value); - kunmap_local(dst); - } + else + clear_highpage(page); =20 count_vm_event(ZSWPIN); if (entry->objcg) @@ -1679,7 +1643,7 @@ static int zswap_debugfs_init(void) debugfs_create_atomic_t("stored_pages", 0444, zswap_debugfs_root, &zswap_stored_pages); debugfs_create_atomic_t("same_filled_pages", 0444, - zswap_debugfs_root, &zswap_same_filled_pages); + zswap_debugfs_root, &zswap_zero_filled_pages); =20 return 0; } --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81BE484D22 for ; Mon, 25 Mar 2024 23:50:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410634; cv=none; b=J9VCBHcbsh8m14n5wH3Ffp/kyrn+cHf9dNVZyGKjt+ZqReWPDtY8sA4YlG7nJHiaIr7DwUea4JwEACA/SeQDJH3QGdvEL1XX1QhBWseoeXwqIJWuT4l0Xv5BphnRFvq1ZPMoYqwKItsgQTtWSrrMZ1p9OGCEBYzz24v135Nflxc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410634; c=relaxed/simple; bh=fxQJhfVXwAFATlTYI8RkyDsQTt5NtB3avUf74PP922A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WxZ3T7jzt9krMGlDI5aA6IVs1iYtHvX5s2lA/ixnNcwokLx3BMvv7SgwL8glWKvWHuQ49jp3XTeb6Ok7xOkQydCi29YId6Oggu6YiuY2sr5Relu6eIR5DskXBr0yF0rP3cCPcnPzeu9jW2mOh+HwSWBIF5BeVIfBlp6tHm36dlw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MSvnx+/Y; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MSvnx+/Y" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609fb151752so88889077b3.2 for ; Mon, 25 Mar 2024 16:50:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410632; x=1712015432; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UL7FjpWvNZHo/XDshUAVHcqtRsHSd5BUf04PTKRvwLE=; b=MSvnx+/YRfpIh73y9ZqhTd/g08l4Nan1jtS/Emh+L9Hl6EJlYmOxDbqVbHp89YooDf UkFDiRhiAHCp1Ag48TDtbk98KriXTEv/63M/C6ofPnnVEfQ+gyG5i0GnclMzPpxoi809 ypNAy6yDn6fcIwbdT7hxURgzTY4ECJOx9m56TsL1BpYzjs8ByVnWD+wVGJX6h1bL+D8A aenPjESx+ndrhStLQ+WP39WcjLFHX43ohNoYvMp9n+QRlZODScefKOWuUsvE6DOu8ns3 k4CFYjNrBnvgTu1b32cP7Zpz5+SE1aF9llYoTHBJi1Hp3A2yiX5HkGbRIeuX/pttVpAQ 7QTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410632; x=1712015432; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UL7FjpWvNZHo/XDshUAVHcqtRsHSd5BUf04PTKRvwLE=; b=Ve4zs3zdvNMafU6vwAv5UeU5Z17ld9JJ5Muz+mB0C4ZnBmwJ4Npl1Wi/FqkCgl2oZb FwgEYL/jEJPOtVEvllYIjOqsdOuohh1XH9Z9h8lCm6fXesJ/F3QBtCXaqvmdgMOADAA1 7PIztAaeETZBuSNEdCveVlIe+GwG0uNkUX58UKwoplpMU4qNJt2MYRczLwcP/Bu00ULn BgT6UKwuR+voHxfn3iz5ONoGNDg5lu6lkFuIqoZSNKWCGurUeWRhB6cOaaKC3XcYFL+G sSMo56Wv+XFECCxnGjEMC97Ol0f8iDm+W1u7Lnb52cHwKUpVoDO2vHXE9T9VHgpNq3bQ YbMA== X-Forwarded-Encrypted: i=1; AJvYcCWTsvu6pFbHaSWzTO01f+cjhAmXrEJ7H2M5HCeAhEoa4rZIy6WMugxdlxwYFvBp89wO+/oNiaYOL5Rp2fs7RNU+n8FO6tXnqoPiV56D X-Gm-Message-State: AOJu0Yxwc7nz1N/bunNFOC6IBOrvQkOaTEIPfj1xp3057AF3SISiXqL+ WG5AggomCCo12XyAC3UGBPdABPZAwl0GaLfWH7vozxRcn5GC/eOU/5aiBR0pMQ/rHV1uVeu/WHi ByfXnA3Nqe3M+uofiTw== X-Google-Smtp-Source: AGHT+IG+TLfwzbh3JCRDIgWmOJOjvnZ/+V3sJUz+/QQZSrlaneyk9M9ZmZQb1PVEN8h1tDdQ5SE3NK8iK22/6mCW X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1507:b0:dc6:e8a7:fdba with SMTP id q7-20020a056902150700b00dc6e8a7fdbamr2612541ybu.4.1711410631787; Mon, 25 Mar 2024 16:50:31 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:15 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-8-yosryahmed@google.com> Subject: [RFC PATCH 7/9] mm: zswap: store zero-filled pages without a zswap_entry From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" After the rbtree to xarray conversion, and dropping zswap_entry.refcount and zswap_entry.value, the only members of zswap_entry utilized by zero-filled pages are zswap_entry.length (always 0) and zswap_entry.objcg. Store the objcg pointer directly in the xarray as a tagged pointer and avoid allocating a zswap_entry completely for zero-filled pages. This simplifies the code as we no longer need to special case zero-length cases. We are also able to further separate the zero-filled pages handling logic and completely isolate them within store/load helpers. Handling tagged xarray pointers is handled in these two helpers, as well as the newly introduced helper for freeing tree elements, zswap_tree_free_element(). There is also a small performance improvement observed over 50 runs of kernel build test (kernbench) comparing the mean build time on a skylake machine when building the kernel in a cgroup v1 container with a 3G limit. This is on top of the improvement from dropping support for non-zero same-filled pages: base patched % diff real 69.915 69.757 -0.229% user 2956.147 2955.244 -0.031% sys 2594.718 2575.747 -0.731% This probably comes from avoiding the zswap_entry allocation and cleanup/freeing for zero-filled pages. Note that the percentage of zero-filled pages during this test was only around 1.5% on average. Practical workloads could have a larger proportion of such pages (e.g. Johannes observed around 10% [1]), so the performance improvement should be larger. This change also saves a small amount of memory due to less allocated zswap_entry's. In the kernel build test above, we save around 2M of slab usage when we swap out 3G to zswap. [1]https://lore.kernel.org/linux-mm/20240320210716.GH294822@cmpxchg.org/ Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou --- mm/zswap.c | 137 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 78 insertions(+), 59 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 413d9242cf500..efc323bab2f22 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -183,12 +183,11 @@ static struct shrinker *zswap_shrinker; * struct zswap_entry * * This structure contains the metadata for tracking a single compressed - * page within zswap. + * page within zswap, it does not track zero-filled pages. * * swpentry - associated swap entry, the offset indexes into the red-black= tree * length - the length in bytes of the compressed page data. Needed during - * decompression. For a zero-filled page length is 0, and both - * pool and lru are invalid and must be ignored. + * decompression. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to @@ -794,30 +793,35 @@ static struct zpool *zswap_find_zpool(struct zswap_en= try *entry) return entry->pool->zpools[hash_ptr(entry, ilog2(ZSWAP_NR_ZPOOLS))]; } =20 -/* - * Carries out the common pattern of freeing and entry's zpool allocation, - * freeing the entry itself, and decrementing the number of stored pages. - */ static void zswap_entry_free(struct zswap_entry *entry) { - if (!entry->length) - atomic_dec(&zswap_zero_filled_pages); - else { - zswap_lru_del(&zswap_list_lru, entry); - zpool_free(zswap_find_zpool(entry), entry->handle); - zswap_pool_put(entry->pool); - } + zswap_lru_del(&zswap_list_lru, entry); + zpool_free(zswap_find_zpool(entry), entry->handle); + zswap_pool_put(entry->pool); if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); obj_cgroup_put(entry->objcg); } zswap_entry_cache_free(entry); - atomic_dec(&zswap_stored_pages); } =20 /********************************* * zswap tree functions **********************************/ +static void zswap_tree_free_element(void *elem) +{ + if (!elem) + return; + + if (xa_pointer_tag(elem)) { + obj_cgroup_put(xa_untag_pointer(elem)); + atomic_dec(&zswap_zero_filled_pages); + } else { + zswap_entry_free((struct zswap_entry *)elem); + } + atomic_dec(&zswap_stored_pages); +} + static int zswap_tree_store(struct xarray *tree, pgoff_t offset, void *new) { void *old; @@ -834,7 +838,7 @@ static int zswap_tree_store(struct xarray *tree, pgoff_= t offset, void *new) * the folio was redirtied and now the new version is being * swapped out. Get rid of the old. */ - zswap_entry_free(old); + zswap_tree_free_element(old); } return err; } @@ -1089,7 +1093,7 @@ static int zswap_writeback_entry(struct zswap_entry *= entry, if (entry->objcg) count_objcg_event(entry->objcg, ZSWPWB); =20 - zswap_entry_free(entry); + zswap_tree_free_element(entry); =20 /* folio is up to date */ folio_mark_uptodate(folio); @@ -1373,6 +1377,33 @@ static void shrink_worker(struct work_struct *w) } while (zswap_total_pages() > thr); } =20 +/********************************* +* zero-filled functions +**********************************/ +#define ZSWAP_ZERO_FILLED_TAG 1UL + +static int zswap_store_zero_filled(struct xarray *tree, pgoff_t offset, + struct obj_cgroup *objcg) +{ + int err =3D zswap_tree_store(tree, offset, + xa_tag_pointer(objcg, ZSWAP_ZERO_FILLED_TAG)); + + if (!err) + atomic_inc(&zswap_zero_filled_pages); + return err; +} + +static bool zswap_load_zero_filled(void *elem, struct page *page, + struct obj_cgroup **objcg) +{ + if (!xa_pointer_tag(elem)) + return false; + + clear_highpage(page); + *objcg =3D xa_untag_pointer(elem); + return true; +} + static bool zswap_is_folio_zero_filled(struct folio *folio) { unsigned long *kaddr; @@ -1432,22 +1463,21 @@ bool zswap_store(struct folio *folio) if (!zswap_check_limit()) goto reject; =20 - /* allocate entry */ + if (zswap_is_folio_zero_filled(folio)) { + if (zswap_store_zero_filled(tree, offset, objcg)) + goto reject; + goto stored; + } + + if (!zswap_non_zero_filled_pages_enabled) + goto reject; + entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); if (!entry) { zswap_reject_kmemcache_fail++; goto reject; } =20 - if (zswap_is_folio_zero_filled(folio)) { - entry->length =3D 0; - atomic_inc(&zswap_zero_filled_pages); - goto insert_entry; - } - - if (!zswap_non_zero_filled_pages_enabled) - goto freepage; - /* if entry is successfully added, it keeps the reference */ entry->pool =3D zswap_pool_current_get(); if (!entry->pool) @@ -1465,17 +1495,14 @@ bool zswap_store(struct folio *folio) if (!zswap_compress(folio, entry)) goto put_pool; =20 -insert_entry: entry->swpentry =3D swp; entry->objcg =3D objcg; =20 if (zswap_tree_store(tree, offset, entry)) goto store_failed; =20 - if (objcg) { + if (objcg) obj_cgroup_charge_zswap(objcg, entry->length); - count_objcg_event(objcg, ZSWPOUT); - } =20 /* * We finish initializing the entry while it's already in xarray. @@ -1487,25 +1514,21 @@ bool zswap_store(struct folio *folio) * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - if (entry->length) { - INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); - } + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&zswap_list_lru, entry); =20 - /* update stats */ +stored: + if (objcg) + count_objcg_event(objcg, ZSWPOUT); atomic_inc(&zswap_stored_pages); count_vm_event(ZSWPOUT); =20 return true; =20 store_failed: - if (!entry->length) - atomic_dec(&zswap_zero_filled_pages); - else { - zpool_free(zswap_find_zpool(entry), entry->handle); + zpool_free(zswap_find_zpool(entry), entry->handle); put_pool: - zswap_pool_put(entry->pool); - } + zswap_pool_put(entry->pool); freepage: zswap_entry_cache_free(entry); reject: @@ -1518,9 +1541,7 @@ bool zswap_store(struct folio *folio) * possibly stale entry which was previously stored at this offset. * Otherwise, writeback could overwrite the new data in the swapfile. */ - entry =3D xa_erase(tree, offset); - if (entry) - zswap_entry_free(entry); + zswap_tree_free_element(xa_erase(tree, offset)); return false; } =20 @@ -1531,26 +1552,27 @@ bool zswap_load(struct folio *folio) struct page *page =3D &folio->page; struct xarray *tree =3D swap_zswap_tree(swp); struct zswap_entry *entry; + struct obj_cgroup *objcg; + void *elem; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); =20 - entry =3D xa_erase(tree, offset); - if (!entry) + elem =3D xa_erase(tree, offset); + if (!elem) return false; =20 - if (entry->length) + if (!zswap_load_zero_filled(elem, page, &objcg)) { + entry =3D elem; + objcg =3D entry->objcg; zswap_decompress(entry, page); - else - clear_highpage(page); + } =20 count_vm_event(ZSWPIN); - if (entry->objcg) - count_objcg_event(entry->objcg, ZSWPIN); - - zswap_entry_free(entry); + if (objcg) + count_objcg_event(objcg, ZSWPIN); =20 + zswap_tree_free_element(elem); folio_mark_dirty(folio); - return true; } =20 @@ -1558,11 +1580,8 @@ void zswap_invalidate(swp_entry_t swp) { pgoff_t offset =3D swp_offset(swp); struct xarray *tree =3D swap_zswap_tree(swp); - struct zswap_entry *entry; =20 - entry =3D xa_erase(tree, offset); - if (entry) - zswap_entry_free(entry); + zswap_tree_free_element(xa_erase(tree, offset)); } =20 int zswap_swapon(int type, unsigned long nr_pages) --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A44D684D2D for ; Mon, 25 Mar 2024 23:50:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410636; cv=none; b=GSJhdTCMevmm7h6jgValq2fhxpqESSTuwAwqc848G2IDz9eZHEdKav5aHcds2bL2AAD7+mWWXgAK6nYYPLHhQbw9PppWRWXeM8obwlfeDmxgG40OBGmYKpezqIGvKX/BUKi0OA562AazR/A3sbBfJUM7pxr9a+o0lXV6cFSCzvw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410636; c=relaxed/simple; bh=1ETm0Kb/hFLFDlFEAvOSDyN5Z9LPHRKS7vvJJ/LpulI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ozyijLqEOU8THTiz0tOc5++894vIRDIrYMcisz6PF0cuyTkJNqxsSyV8HMx1kOkKLLY/QDw02YTDlxtEOv+h7kAp71Buf4sF2p812685Payjigw8Lbl+V0xqxzJ6RzLLKWP/Dkb+RAtCaV5pijMGbpLbQzo3kyzMAsatC0+9i+Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BdUFuZPc; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BdUFuZPc" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dd944567b6cso6712983276.2 for ; Mon, 25 Mar 2024 16:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410633; x=1712015433; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KO2q76LvMLcDayey/RN5wyIAuuRCIUh2utVXFGvhTms=; b=BdUFuZPc/DzYe3O+g2m/cdEFg97bSr+3WUmHdMInMEFTF8s49+PqB6fxqH3rfUOwEF ZoDRpD5Dyb2OhvP098bfaKsAHqkfV+blDb0utrkKE5WbpU9WEh3p8VU9WGpUQuLFX5lx WfbF1Xlg7WOTiSu4fJiCSBds6s7TG6ur0FIFEwhQeyJyI1eKTXKlcsJwZzoVygW6Jss2 vNlpuVIyKL+6/1FrtwIu9rcHaQDmxFfNWkbkCYhJqFNJWwNllnAR0gNLWMwDsXdIitOd mqT/G4Zz0f/ijuJ090ZRO2hz7HgZnEkXj+2CNwiPqHh22IW5bNDSokEb61WQGBuS8Se0 2bHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410633; x=1712015433; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KO2q76LvMLcDayey/RN5wyIAuuRCIUh2utVXFGvhTms=; b=bYqKB522o1NnI3QRzJil2k/Ldp+qylGCSXMiCTsfEQTJdGLqXQJN4iBCLofVECMu0F fDXTadTxQvTPlfrHemtB6krYg2IbWEnCKxuwbNqeLIgd7BJUNnBKotrCr4wnJ5XDcrdV +NIZwDBL7BxgRxRK8iLr+usm5ZK8wy2+38fV6tGq02rkrd+ENl10xB/L4DllEQRP7Gkd GSNR85oFtMnCCluEWfa3mN0FjZ02DKHTy1jUzIaLnjux7UxjnYR8tUDl/jHaescjvEDn Yfz7Ca/E7f9BDx5FqyvFgYBZ9xUMhrcNrEd/ZawPEO8opQqqdsK7LmKlaWdgQlJy8npE 9IDg== X-Forwarded-Encrypted: i=1; AJvYcCUKxsqI9qz1cwnCIy3V4UFZBuO/qKgpjORYr+3wPfEw3K2JwZOoGp5V1r/1PDoV+1s/D1Gz/j2mTpw7t/Ft6xYXMaZZEgOSz4s3US8J X-Gm-Message-State: AOJu0YxD/2GiI/NVDusHxvlbYiI6w+4hafbJkNUwW1A0y6myqCFtJQK4 s37oKjsEK1zXlYx6iXquF7iQzI4ysnr9NpYa1E2/nbsexv3uTzPxsepqi8jFxCLHzLwFLHfE5j+ tAC6CvTD3gPQYZO4q2w== X-Google-Smtp-Source: AGHT+IGsngKICRKq/pDsaMChol7XL6fK76guiT9EmRwqzuuG62psC+Y/KoJk23mBwa/MlcuS5UwOswLoFZ1CnMT3 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1241:b0:dcc:4785:b51e with SMTP id t1-20020a056902124100b00dcc4785b51emr370308ybu.12.1711410633551; Mon, 25 Mar 2024 16:50:33 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:16 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-9-yosryahmed@google.com> Subject: [RFC PATCH 8/9] mm: zswap: do not check the global limit for zero-filled pages From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When storing zero-filled pages, there is no point of checking the global zswap limit. These pages do not consume any memory that contributes toward the limit. Move the limit checking after zero-filled pages are handled. This avoids having zero-filled pages skip zswap and go to disk swap if the limit is hit. It also avoids queueing the shrink worker, which may end up being unnecessary if the zswap usage goes down on its own before another store is attempted. Ignoring the memcg limits as well for zero-filled pages is more controversial. Those limits are more a matter of per-workload policy. Some workloads disable zswap completely by setting memory.zswap.max =3D 0, and those workloads could start observing some zswap activity even after disabling zswap. Although harmless, this could cause confusion to userspace. Remain conservative and keep respecting those limits. Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou --- mm/zswap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index efc323bab2f22..9357328d940af 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1460,9 +1460,6 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } =20 - if (!zswap_check_limit()) - goto reject; - if (zswap_is_folio_zero_filled(folio)) { if (zswap_store_zero_filled(tree, offset, objcg)) goto reject; @@ -1472,6 +1469,9 @@ bool zswap_store(struct folio *folio) if (!zswap_non_zero_filled_pages_enabled) goto reject; =20 + if (!zswap_check_limit()) + goto reject; + entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); if (!entry) { zswap_reject_kmemcache_fail++; --=20 2.44.0.396.g6e790dbe36-goog From nobody Sun Feb 8 12:15:02 2026 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBB07128385 for ; Mon, 25 Mar 2024 23:50:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410637; cv=none; b=oepVklFrY9RLZg7hYKU2mMWJlHYq4Dnk7H+QbDVs6VacK1r9SR7NT0i5/+LgdwOsNZ/jWIR9UZSXDqG7/6yEukOIDYdWLaifw7TLLYjyqSty+5eH3MB294Kgv3kWseDfYp3XTsVM7Id+K1FvHp9nE2ssohYD9PnfHUP4FnMpaxk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711410637; c=relaxed/simple; bh=HaCYxlLo5Ti9rOUslx2RmaiaM++bpDdpyJNIYANfmM8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=iJb9ST65An1/Fe0Q5nlvcIPQNq4/SIeSpx6V3w40sTGtZuufGMuuuDjCOjKFJThR9A+7SOWnC5eBnHbTKDSHC6Y0Mag/HTwhpny72qSZoilSD4Z/MGS5evWUBbwT+/c+4N8OeZ1RB/23F25xipinCblO/sw/942QMFPkzsAGYrE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TudUSqTT; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--yosryahmed.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TudUSqTT" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dced704f17cso7910231276.1 for ; Mon, 25 Mar 2024 16:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410635; x=1712015435; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=TudUSqTTCYWiq6Hp4pPry0jpH57iwRaupU7wQ7+OXsX54smOoGhEoRLnVB3gsYmuZZ g9Er7znFYlP8kpMuV3LU9BgsLFTQKpeFrn74y4dVnh3HyGy+5Du+LSZkPtEhvv+yuBgu zuhZT9rg3iWvSLj7448OFTVjwCHV8BnH85qN8TA/XfXizFbQnjW70HgwW/icFJU4mCZn j7fc6pPSCmPT/ZklBm3SHTudfqL7thT+4VqYzeEaX3yDhieXv1DsUUgkaPzGyUYOeG80 HRtXtDTu80mkp2jLBUhE0PxByQzBEWQKgqn/lwTSr56mJofP3v/qivjqme0zBS9AC/5Q Gccw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410635; x=1712015435; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=TQj0a/Xj6eRWThA7JSfRDIPFHt9zqHJoR+t6n4RmO5k6+0m4thKYnvqJn9yONmLr1P 9LcKBOUOPB5lhrRI9m9zGDpCuf91aZsulcZRnkerr1BN0voZMHYcg1cDM36WlR2jBr2E C3ftUdb84+UE0HzIpIh4idUvoxsUyFcS4RP3rTK81MXJ9qcB0uGaWWwUZGwFQRnrC5fe bgYfNPYkXpWZrbvKLufLxW/5eTyaTbVYgO7d83lH2fEAUAIRO2SRKfzaHFioVHc8+fRR 2kHYyAs9BpfcpRGAaX1tTwgCs2MRqBcqJ4S4zdAixEUzHKM7nmqSFDV0cKSAsWu2eRON +WqA== X-Forwarded-Encrypted: i=1; AJvYcCWigM1u17HI8Z4jbUFMRQv1jwq4dnDSy0dv18XivrZaLaA55Jk3P2+oYKDBx179dyZPjeEhejWhQBfu3wmHddwlhW/9I9PvBw0kamTl X-Gm-Message-State: AOJu0Yw3V9BD1huodnxgAQ+7/DpnbC/o5OGsPBZcCoQF9fwthDUcu+oV B+9hZrFm+GsOqVppuYTUpIvmN3ckiIWP4Osx/WNVg29ftBWfrqCn9IiseCy/NyaR177sX1IERFh 0u7tkr60BfInG44VGdA== X-Google-Smtp-Source: AGHT+IHhwuSMeZ8KbkpdVbgoG6k+kjBoalv5THVzyluQgGq+kifTjQF/ACTCwAHFlLZp6s7t/niB6q7ZIp/Hk5oK X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:218d:b0:dda:c565:1025 with SMTP id dl13-20020a056902218d00b00ddac5651025mr317614ybb.2.1711410635103; Mon, 25 Mar 2024 16:50:35 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:17 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-10-yosryahmed@google.com> Subject: [RFC PATCH 9/9] mm: zswap: use zswap_entry_free() for partially initialized entries From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zswap_entry_free() performs four types of cleanups before freeing a zswap_entry: - Deletes the entry from the LRU. - Frees compressed memory. - Puts the pool reference. - Uncharges the compressed memory and puts the objcg. zswap_entry_free() always expects a fully initialized entry. Allow zswap_entry_free() to handle partially initialized entries by making it possible to identify what cleanups are needed as follows: - Allocate entries with __GFP_ZERO and initialize zswap_entry.lru when the entry is allocated. Points are NULL and length is zero upon initialization. - Use zswap_entry.length to identify if there is compressed memory to free. This is possible now that zero-filled pages are handled separately, so a length of zero means we did not successfully compress the page. - Only initialize entry->objcg after the memory is charged in zswap_store(). With this in place, use zswap_entry_free() in the failure path of zswap_store() to cleanup partially initialized entries. This simplifies the cleanup code in zswap_store(). While we are at it, rename the remaining cleanup labels to more meaningful names. Signed-off-by: Yosry Ahmed --- mm/zswap.c | 62 ++++++++++++++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 9357328d940af..c50f9df230ca3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -774,12 +774,13 @@ void zswap_memcg_offline_cleanup(struct mem_cgroup *m= emcg) **********************************/ static struct kmem_cache *zswap_entry_cache; =20 -static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid) +static struct zswap_entry *zswap_entry_cache_alloc(int nid) { struct zswap_entry *entry; - entry =3D kmem_cache_alloc_node(zswap_entry_cache, gfp, nid); - if (!entry) - return NULL; + entry =3D kmem_cache_alloc_node(zswap_entry_cache, + GFP_KERNEL | __GFP_ZERO, nid); + if (entry) + INIT_LIST_HEAD(&entry->lru); return entry; } =20 @@ -795,9 +796,12 @@ static struct zpool *zswap_find_zpool(struct zswap_ent= ry *entry) =20 static void zswap_entry_free(struct zswap_entry *entry) { - zswap_lru_del(&zswap_list_lru, entry); - zpool_free(zswap_find_zpool(entry), entry->handle); - zswap_pool_put(entry->pool); + if (!list_empty(&entry->lru)) + zswap_lru_del(&zswap_list_lru, entry); + if (entry->length) + zpool_free(zswap_find_zpool(entry), entry->handle); + if (entry->pool) + zswap_pool_put(entry->pool); if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); obj_cgroup_put(entry->objcg); @@ -1447,7 +1451,7 @@ bool zswap_store(struct folio *folio) return false; =20 if (!zswap_enabled) - goto check_old; + goto erase_old; =20 /* Check cgroup limits */ objcg =3D get_obj_cgroup_from_folio(folio); @@ -1455,54 +1459,52 @@ bool zswap_store(struct folio *folio) memcg =3D get_mem_cgroup_from_objcg(objcg); if (shrink_memcg(memcg)) { mem_cgroup_put(memcg); - goto reject; + goto put_objcg; } mem_cgroup_put(memcg); } =20 if (zswap_is_folio_zero_filled(folio)) { if (zswap_store_zero_filled(tree, offset, objcg)) - goto reject; + goto put_objcg; goto stored; } =20 if (!zswap_non_zero_filled_pages_enabled) - goto reject; + goto put_objcg; =20 if (!zswap_check_limit()) - goto reject; + goto put_objcg; =20 - entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); + entry =3D zswap_entry_cache_alloc(folio_nid(folio)); if (!entry) { zswap_reject_kmemcache_fail++; - goto reject; + goto put_objcg; } =20 - /* if entry is successfully added, it keeps the reference */ entry->pool =3D zswap_pool_current_get(); if (!entry->pool) - goto freepage; + goto free_entry; =20 if (objcg) { memcg =3D get_mem_cgroup_from_objcg(objcg); if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { mem_cgroup_put(memcg); - goto put_pool; + goto free_entry; } mem_cgroup_put(memcg); } =20 if (!zswap_compress(folio, entry)) - goto put_pool; - - entry->swpentry =3D swp; - entry->objcg =3D objcg; + goto free_entry; =20 if (zswap_tree_store(tree, offset, entry)) - goto store_failed; + goto free_entry; =20 - if (objcg) + if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); + entry->objcg =3D objcg; + } =20 /* * We finish initializing the entry while it's already in xarray. @@ -1514,7 +1516,7 @@ bool zswap_store(struct folio *folio) * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - INIT_LIST_HEAD(&entry->lru); + entry->swpentry =3D swp; zswap_lru_add(&zswap_list_lru, entry); =20 stored: @@ -1525,17 +1527,13 @@ bool zswap_store(struct folio *folio) =20 return true; =20 -store_failed: - zpool_free(zswap_find_zpool(entry), entry->handle); -put_pool: - zswap_pool_put(entry->pool); -freepage: - zswap_entry_cache_free(entry); -reject: +free_entry: + zswap_entry_free(entry); +put_objcg: obj_cgroup_put(objcg); if (zswap_pool_reached_full) queue_work(shrink_wq, &zswap_shrink_work); -check_old: +erase_old: /* * If the zswap store fails or zswap is disabled, we must invalidate the * possibly stale entry which was previously stored at this offset. --=20 2.44.0.396.g6e790dbe36-goog