From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42BF5486E71 for ; Tue, 25 Aug 2026 15:32:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671974; cv=none; b=uQR1aSlT+U1eGuFfyOGazf89/mnIFOmrkURZASKIujr5eQjIMq15LIp6SLIw2V7FFYAItmQbLkIMlFMFiOeFsAI6B0Y+tiqkLU7unuEOykuE16gs/vU54ZhCGp+VK5fyjWiPIZrHsznv9CR3Kz7AQ3H74CFrKu+GwgF1X8cuUzY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671974; c=relaxed/simple; bh=HuZwf1BE1MsdoaDjT7NmYFFyTWvkuI/EdGlALduVFpA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WNgBT5UompfJ5vpliQMIqpur8b0hL7JwWR+SIoZ+rheOUy1jVHDcCkS37ptB+VqiZVZRMa1fw37UgCHtc3B/+F71SSga63es0yOttkSSag9S5DesFvpEnOt/2hJ2kNMlZg+J+u51qi8YgM5IXxbpz2R7WlHKRAh+znVcyqY4tK4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XM8B3wDK; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XM8B3wDK" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-4b28df1f600so3830756b6e.2 for ; Tue, 25 Aug 2026 08:32:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671963; x=1788276763; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xMGygkVCQST0F/zwz/Ue1uFqPHUtxjmJAwo4+G9sBQo=; b=XM8B3wDK9y3bM/VUFzqaqJhkNRVPAskh0u1BqIur4V3VPcJscW4IhgMfiNDSxrv4yW sdvceit0AQOFbIb2c4Y7FGK5mumZerVwe7CZZiZWf3+FneidhpFEWf0/nt0RkLsk7ygl zHBbzAF/Ltco14/C6vksBqIwW0XbunoALd3c1bEdMfD5g4if18+vOJgujxzxaGT6fUQd 10z3NbmVb5OXoFRDr3JehJkGCDDDN8z/gWYMgjgC+X4Pj8r3SHKmaOIYVNVT3yxfO6ep zWbYSlk/LHQHj/FeorNWakN0WqcmRwuaxsx92e5/BDeEBiy82bzqMURyU9QSwgkOO7w0 xUxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671963; x=1788276763; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=xMGygkVCQST0F/zwz/Ue1uFqPHUtxjmJAwo4+G9sBQo=; b=cHLRuXwfkTeY8zJ7A58Gk9DzKAF0BC4glGnOBh7sYhjYCze8pwyQn+AEvMcUqv5ecz 4Rm1lcJQEHZDBIiBAi5MYC6sQW5zC2IJVZWfT+DqYY8WlvEqX/9kIfyHr3rkrZpz1887 ZXsYTxG33tpRoFZIjobLer4RKXAtCIGeOVZfnC3H1WXm+JHVnj9+iLe9uB0uq9WUU27L G1PNpPGdqsA53GJ78so/7GYpFra6KJWH8hxplJzwBAeU6a0xHoWza1Ld0eaCc376jf3Q B/zZ0fKi9in0xmt5FXR7C5IC62q0TbSMKEGuh2KX/0Wn3erovQJAqtssu0P15aaGUAFI S7Og== X-Forwarded-Encrypted: i=1; AHgh+Roa0Jk6aGNF6vxXE04T7NVCcksK7bvp38zvKbf6z/E8XlEvasP0m0xRv/niBmhfMyI26L/37bHrtPB/MH4=@vger.kernel.org X-Gm-Message-State: AFuF++klBKsD7MqsC8decNcn/pirEBUfEGE5rF7JzIvXD2z9/nNmHKvc Qhs/DSKK9cA8Zy79Fxw4HI+iyXfBemOZQmCoGppEel5hQLY20vy1aOs7 X-Gm-Gg: AR+sD11tG/Kb7WClyz3GqdMYsCPQZQUZQ4/vHAya2zCIdXGwcUQC7P5H8mk1W6M7yxK jubHC8lVNqDCXbX/gu8+xjUCDpgVSfxZTvnDffBi1wh7N/FBl8wogN8XqjzJWtwQLYw+Ow9SQGh UiBvOKB9karMrgTz/bz+y+vX+X4860Bm+n6BOx9jrmw6aws1rh+tWktVmGDJ29LlVTUT1wGNHom YGPBbCNVg4hKOi63RGnjGiqumOTiasm/TmIBUeENzYQ+PrMgO3N4ZKdC7jkCxadCHcsBLr2nsZe mlXEZpM88pitlqWO8S3HSIAlRhvMsLf3Iw5fyKi3txwHZ2gSJ5in9X+q8i0tHpx5OrNQ8l7JTgi WrowlfYn/BqEJ+zJI0SrY4UATw86txGSLO/U1TDWrsHzJo6M034iLMLPNuX0YtvuHqP5NrO1e4s eQKKg/kOPNgmTyyA3J1Sl9vx8FjVqn5xl/EhGKS7/1i45klHBLlyYEf1YG2xTlf9pd2scseK/bE rhpMRlymRF3uNUo X-Received: by 2002:a05:6820:150d:b0:6b0:e6d1:d5cf with SMTP id 006d021491bc7-6b19044726fmr6124417eaf.19.1787671962240; Tue, 25 Aug 2026 08:32:42 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:41::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7f48fc8623esm7136078a34.22.2026.08.25.08.32.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:40 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 01/11] mm, swap: add virtual swap device infrastructure Date: Tue, 25 Aug 2026 08:32:27 -0700 Message-ID: <20260825153238.2695446-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create a virtual swap device (16 TB with 4 KB pages), along with the dynamic cluster infrastructure that the rest of the vswap layer is built on. swap_cluster_info_dynamic keeps per-cluster info in an xarray, so a device can be sized without a static cluster_info[] array. For now, vswap requires a 64-bit architecture. The dynamic-cluster allocator is wired in, but nothing reaches it yet. vswap_si is kept off the swap device lists, and no allocation path can select it. Backends (zswap, zero, physical disk) and the vswap-aware swap-out / swap-in / writeback paths arrive in subsequent patches. Routing is controlled by the "vswap=3D" kernel parameter, defaulting to CONFIG_VSWAP_DEFAULT_ON. When off, no device exists, every vswap path is skipped, and swap behavior is unchanged. When on, vswap_init() creates the device and enables a static key only after it is fully published, so callers never observe a half-built device. The device lives for the lifetime of the kernel and cannot be swapon'd or swapoff'd. Suggested-by: Kairui Song Signed-off-by: Nhat Pham --- .../admin-guide/kernel-parameters.txt | 7 + MAINTAINERS | 1 + include/linux/swap.h | 9 + mm/Kconfig | 20 ++ mm/page_io.c | 14 + mm/swap.h | 67 ++++- mm/swap_state.c | 37 ++- mm/swap_table.h | 4 + mm/swapfile.c | 271 ++++++++++++++++-- mm/vswap.h | 34 +++ mm/zswap.c | 6 + 11 files changed, 437 insertions(+), 33 deletions(-) create mode 100644 mm/vswap.h diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 1af62cd16c9d..6612b5e0a055 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -8364,6 +8364,13 @@ Kernel parameters force - force vulnerability detection even on unaffected processors =20 + vswap=3D [MM,EARLY] + Route swapouts through the virtual swap layer, which + allows zswap and zero-filled pages to be used without + a physical swap device. 64-bit only. + Format: { on | off } + Default: on if CONFIG_VSWAP_DEFAULT_ON=3Dy, else off. + vsyscall=3D [X86-64,EARLY] Controls the behavior of vsyscalls (i.e. calls to fixed addresses of 0xffffffffff600x00 from legacy diff --git a/MAINTAINERS b/MAINTAINERS index 29236523cefb..8a5827d27177 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17251,6 +17251,7 @@ F: mm/swap.h F: mm/swap_table.h F: mm/swap_state.c F: mm/swapfile.c +F: mm/vswap.h =20 MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE) M: Andrew Morton diff --git a/include/linux/swap.h b/include/linux/swap.h index 5658a1634b85..5339323486d5 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -207,6 +207,7 @@ enum { SWP_STABLE_WRITES =3D (1 << 11), /* no overwrite PG_writeback pages */ SWP_SYNCHRONOUS_IO =3D (1 << 12), /* synchronous IO is efficient */ SWP_HIBERNATION =3D (1 << 13), /* pinned for hibernation */ + SWP_VSWAP =3D (1 << 14), /* virtual swap device */ /* add others here before... */ }; =20 @@ -276,8 +277,14 @@ struct swap_info_struct { struct list_head discard_clusters; /* discard clusters list */ struct plist_node avail_list; /* entry in swap_avail_head */ const struct swap_ops *ops; + struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info= */ }; =20 +static inline bool swap_is_vswap(struct swap_info_struct *si) +{ + return si->flags & SWP_VSWAP; +} + static inline swp_entry_t page_swap_entry(struct page *page) { struct folio *folio =3D page_folio(page); @@ -408,6 +415,8 @@ void swap_free_hibernation_slot(swp_entry_t entry); =20 static inline void put_swap_device(struct swap_info_struct *si) { + if (swap_is_vswap(si)) + return; percpu_ref_put(&si->users); } =20 diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..08fdc7502c1d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -19,6 +19,26 @@ menuconfig SWAP used to provide more virtual memory than the actual RAM present in your computer. If unsure say Y. =20 +config VSWAP_DEFAULT_ON + bool "Route swapouts through virtual swap by default" + depends on SWAP && 64BIT + default n + help + Virtual swap allows zswap and zero-filled pages to be used + without swapping on a physical device first, and lets a page + move between zswap and a swapfile without invalidating the page + table entries that refer to it. + + Swap entries are handed out by a virtual swap device instead of + naming a slot on a real one, so the backing can be chosen and + changed after the entry exists. + + Say Y to make "vswap=3Don" the default, routing swapouts through + the virtual swap layer from boot. + + Say N (default) to leave vswap off unless "vswap=3Don" is passed + on the kernel command line. + config ZSWAP bool "Compressed cache for swap pages" depends on SWAP diff --git a/mm/page_io.c b/mm/page_io.c index 88962571cb93..3d0e78c17090 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -28,6 +28,7 @@ #include #include "swap.h" #include "swap_table.h" +#include "vswap.h" =20 int generic_swapfile_activate(struct swap_info_struct *sis, struct file *swap_file, @@ -248,6 +249,14 @@ int swap_writeout(struct swap_io_ctx *ctx, struct foli= o *folio) } rcu_read_unlock(); =20 + /* + * A vswap folio has no physical slot to write to, so keep it dirty. + */ + if (is_vswap_entry(folio->swap)) { + folio_mark_dirty(folio); + return AOP_WRITEPAGE_ACTIVATE; + } + __swap_writepage(ctx, folio); return 0; out_unlock: @@ -480,6 +489,11 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct f= olio *folio) if (zswap_load(folio) !=3D -ENOENT) goto finish; =20 + if (unlikely(swap_is_vswap(sis))) { + folio_unlock(folio); + goto finish; + } + /* We have to read from slower devices. Increase zswap protection. */ zswap_folio_swapin(folio); swap_add_folio(ctx, folio, READ); diff --git a/mm/swap.h b/mm/swap.h index 90a551a88df6..f18385dc9c6e 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -67,6 +67,12 @@ struct swap_cluster_info { struct list_head list; }; =20 +struct swap_cluster_info_dynamic { + struct swap_cluster_info ci; + unsigned int index; /* for cluster_index() */ + struct rcu_head rcu; +}; + /* All on-list cluster must have a non-zero flag. */ enum swap_cluster_flags { CLUSTER_FLAG_NONE =3D 0, /* For temporary off-list cluster */ @@ -77,6 +83,7 @@ enum swap_cluster_flags { CLUSTER_FLAG_USABLE =3D CLUSTER_FLAG_FRAG, CLUSTER_FLAG_FULL, CLUSTER_FLAG_DISCARD, + CLUSTER_FLAG_DEAD, /* Vswap dynamic cluster pending kfree_rcu */ CLUSTER_FLAG_MAX, }; =20 @@ -119,12 +126,33 @@ static inline struct swap_info_struct *__swap_entry_t= o_info(swp_entry_t entry) return __swap_type_to_info(swp_type(entry)); } =20 +/** + * __swap_offset_to_cluster - look up the cluster holding a swap offset + * @si: the swap device + * @offset: the swap entry offset + * + * Context: A vswap cluster is freed by kfree_rcu(). Callers must hold the + * RCU read lock, or know the cluster is pinned by an in-use entry. + * + * Return: the cluster, or NULL if @si is a vswap device with no cluster + * allocated at @offset. + */ static inline struct swap_cluster_info *__swap_offset_to_cluster( struct swap_info_struct *si, pgoff_t offset) { + unsigned int cluster_idx =3D offset / SWAPFILE_CLUSTER; + VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */ VM_WARN_ON_ONCE(offset >=3D roundup(si->max, SWAPFILE_CLUSTER)); - return &si->cluster_info[offset / SWAPFILE_CLUSTER]; + + if (swap_is_vswap(si)) { + struct swap_cluster_info_dynamic *ci_dyn; + + ci_dyn =3D xa_load(&si->cluster_info_pool, cluster_idx); + return ci_dyn ? &ci_dyn->ci : NULL; + } + + return &si->cluster_info[cluster_idx]; } =20 static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_= t entry) @@ -133,10 +161,36 @@ static inline struct swap_cluster_info *__swap_entry_= to_cluster(swp_entry_t entr swp_offset(entry)); } =20 +static inline struct swap_cluster_info *__vswap_cluster_lock( + struct swap_info_struct *si, unsigned long offset, bool irq) +{ + struct swap_cluster_info *ci; + + rcu_read_lock(); + ci =3D __swap_offset_to_cluster(si, offset); + if (ci) { + if (irq) + spin_lock_irq(&ci->lock); + else + spin_lock(&ci->lock); + + /* The cluster can be torn down while we wait for the lock. */ + if (ci->flags =3D=3D CLUSTER_FLAG_DEAD) { + if (irq) + spin_unlock_irq(&ci->lock); + else + spin_unlock(&ci->lock); + ci =3D NULL; + } + } + rcu_read_unlock(); + return ci; +} + static __always_inline struct swap_cluster_info *__swap_cluster_lock( struct swap_info_struct *si, unsigned long offset, bool irq) { - struct swap_cluster_info *ci =3D __swap_offset_to_cluster(si, offset); + struct swap_cluster_info *ci; =20 /* * Nothing modifies swap cache in an IRQ context. All access to @@ -149,6 +203,11 @@ static __always_inline struct swap_cluster_info *__swa= p_cluster_lock( */ VM_WARN_ON_ONCE(!in_task()); VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */ + + if (swap_is_vswap(si)) + return __vswap_cluster_lock(si, offset, irq); + + ci =3D __swap_offset_to_cluster(si, offset); if (irq) spin_lock_irq(&ci->lock); else @@ -159,10 +218,12 @@ static __always_inline struct swap_cluster_info *__sw= ap_cluster_lock( /** * swap_cluster_lock - Lock and return the swap cluster of given offset. * @si: swap device the cluster belongs to. - * @offset: the swap entry offset, pointing to a valid slot. + * @offset: the swap entry offset. * * Context: The caller must ensure the offset is in the valid range and * protect the swap device with reference count or locks. + * Return: the locked cluster, or NULL if it is gone. Only a vswap device + * can return NULL, as its clusters are allocated and freed on demand. */ static inline struct swap_cluster_info *swap_cluster_lock( struct swap_info_struct *si, unsigned long offset) diff --git a/mm/swap_state.c b/mm/swap_state.c index b76eb3d876fd..a800abebba38 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -96,8 +96,10 @@ struct folio *swap_cache_get_folio(swp_entry_t entry) struct folio *folio; =20 for (;;) { + rcu_read_lock(); swp_tb =3D swap_table_get(__swap_entry_to_cluster(entry), swp_cluster_offset(entry)); + rcu_read_unlock(); if (!swp_tb_is_folio(swp_tb)) return NULL; folio =3D swp_tb_to_folio(swp_tb); @@ -119,8 +121,10 @@ bool swap_cache_has_folio(swp_entry_t entry) { unsigned long swp_tb; =20 + rcu_read_lock(); swp_tb =3D swap_table_get(__swap_entry_to_cluster(entry), swp_cluster_offset(entry)); + rcu_read_unlock(); return swp_tb_is_folio(swp_tb); } =20 @@ -136,8 +140,10 @@ void *swap_cache_get_shadow(swp_entry_t entry) { unsigned long swp_tb; =20 + rcu_read_lock(); swp_tb =3D swap_table_get(__swap_entry_to_cluster(entry), swp_cluster_offset(entry)); + rcu_read_unlock(); if (swp_tb_is_shadow(swp_tb)) return swp_tb_to_shadow(swp_tb); return NULL; @@ -406,14 +412,16 @@ void __swap_cache_replace_folio(struct swap_cluster_i= nfo *ci, * -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the call= er * should abort or try to use the cached folio instead */ -static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci, - swp_entry_t targ_entry, gfp_t gfp, +static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp, unsigned int order, struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx) { int err; swp_entry_t entry; struct folio *folio; + struct swap_cluster_info *ci; + struct swap_info_struct *si =3D __swap_entry_to_info(targ_entry); + unsigned long offset =3D swp_offset(targ_entry); void *shadow =3D NULL; unsigned short memcg_id; unsigned long address, nr_pages =3D 1UL << order; @@ -423,9 +431,12 @@ static struct folio *__swap_cache_alloc(struct swap_cl= uster_info *ci, entry.val =3D round_down(targ_entry.val, nr_pages); =20 /* Check if the slot and range are available, skip allocation if not */ - spin_lock(&ci->lock); - err =3D __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL); - spin_unlock(&ci->lock); + err =3D -ENOENT; + ci =3D swap_cluster_lock(si, offset); + if (ci) { + err =3D __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL); + swap_cluster_unlock(ci); + } if (unlikely(err)) return ERR_PTR(err); =20 @@ -446,10 +457,13 @@ static struct folio *__swap_cache_alloc(struct swap_c= luster_info *ci, return ERR_PTR(-ENOMEM); =20 /* Double check the range is still not in conflict */ - spin_lock(&ci->lock); - err =3D __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_= id); + err =3D -ENOENT; + ci =3D swap_cluster_lock(si, offset); + if (ci) + err =3D __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg= _id); if (unlikely(err)) { - spin_unlock(&ci->lock); + if (ci) + swap_cluster_unlock(ci); folio_put(folio); return ERR_PTR(err); } @@ -457,10 +471,11 @@ static struct folio *__swap_cache_alloc(struct swap_c= luster_info *ci, __folio_set_locked(folio); __folio_set_swapbacked(folio); __swap_cache_do_add_folio(ci, folio, entry); - spin_unlock(&ci->lock); + swap_cluster_unlock(ci); =20 if (mem_cgroup_swapin_charge_folio(folio, memcg_id, vmf ? vmf->vma->vm_mm : NULL, gfp)) { + /* The folio pins the cluster */ spin_lock(&ci->lock); __swap_cache_do_del_folio(ci, folio, entry, shadow); spin_unlock(&ci->lock); @@ -517,9 +532,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_e= ntry, gfp_t gfp, { int order, err; struct folio *ret; - struct swap_cluster_info *ci; =20 - ci =3D __swap_entry_to_cluster(targ_entry); order =3D highest_order(orders); =20 /* orders must be non-zero, and must not exceed cluster size. */ @@ -527,7 +540,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_e= ntry, gfp_t gfp, return ERR_PTR(-EINVAL); =20 do { - ret =3D __swap_cache_alloc(ci, targ_entry, gfp, order, + ret =3D __swap_cache_alloc(targ_entry, gfp, order, vmf, mpol, ilx); if (!IS_ERR(ret)) break; diff --git a/mm/swap_table.h b/mm/swap_table.h index e6613e62f8d0..868aae6c820f 100644 --- a/mm/swap_table.h +++ b/mm/swap_table.h @@ -6,6 +6,8 @@ #include #include "swap.h" =20 +extern struct swap_info_struct *vswap_si; + /* A typical flat array in each cluster as swap table */ struct swap_table { atomic_long_t entries[SWAPFILE_CLUSTER]; @@ -255,6 +257,8 @@ static inline unsigned long swap_table_get(struct swap_= cluster_info *ci, unsigned long swp_tb; =20 VM_WARN_ON_ONCE(off >=3D SWAPFILE_CLUSTER); + if (!ci) + return SWP_TB_NULL; =20 rcu_read_lock(); table =3D rcu_dereference(ci->table); diff --git a/mm/swapfile.c b/mm/swapfile.c index 53bf01d5f7f1..ae88b91c92a2 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -46,6 +47,7 @@ #include #include #include "swap_table.h" +#include "vswap.h" #include "internal.h" #include "swap.h" =20 @@ -401,6 +403,8 @@ static inline bool cluster_is_usable(struct swap_cluste= r_info *ci, int order) static inline unsigned int cluster_index(struct swap_info_struct *si, struct swap_cluster_info *ci) { + if (swap_is_vswap(si)) + return container_of(ci, struct swap_cluster_info_dynamic, ci)->index; return ci - si->cluster_info; } =20 @@ -586,10 +590,15 @@ static void move_cluster(struct swap_info_struct *si, lockdep_assert_held(&ci->lock); =20 spin_lock(&si->lock); - if (ci->flags =3D=3D CLUSTER_FLAG_NONE) + if (!list) { + /* Going away. An isolated cluster is already off its list. */ + if (ci->flags !=3D CLUSTER_FLAG_NONE) + list_del(&ci->list); + } else if (ci->flags =3D=3D CLUSTER_FLAG_NONE) { list_add_tail(&ci->list, list); - else + } else { list_move_tail(&ci->list, list); + } spin_unlock(&si->lock); ci->flags =3D new_flags; } @@ -607,6 +616,18 @@ static void __free_cluster(struct swap_info_struct *si= , struct swap_cluster_info { swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, false); swap_cluster_free_table(ci); + + if (swap_is_vswap(si)) { + struct swap_cluster_info_dynamic *ci_dyn; + + /* vswap clusters are destroyed, not returned to free_clusters. */ + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + xa_erase(&si->cluster_info_pool, ci_dyn->index); + move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD); + kfree_rcu(ci_dyn, rcu); + return; + } + move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE); ci->order =3D 0; } @@ -843,6 +864,8 @@ static bool cluster_reclaim_range(struct swap_info_stru= ct *si, unsigned long offset =3D start, end =3D start + nr_pages; unsigned long swp_tb; =20 + VM_WARN_ON_ONCE(swap_is_vswap(si)); + spin_unlock(&ci->lock); do { swp_tb =3D swap_table_get(ci, offset % SWAPFILE_CLUSTER); @@ -1034,6 +1057,44 @@ static unsigned int alloc_swap_scan_list(struct swap= _info_struct *si, return found; } =20 +static unsigned int vswap_alloc_cluster(struct swap_info_struct *si, + struct folio *folio) +{ + struct swap_cluster_info_dynamic *ci_dyn; + struct swap_cluster_info *ci; + unsigned long offset; + + VM_WARN_ON(!swap_is_vswap(si)); + + ci_dyn =3D kzalloc_obj(*ci_dyn, GFP_ATOMIC); + if (!ci_dyn) + return SWAP_ENTRY_INVALID; + + spin_lock_init(&ci_dyn->ci.lock); + INIT_LIST_HEAD(&ci_dyn->ci.list); + + if (swap_cluster_alloc_table(&ci_dyn->ci, GFP_ATOMIC)) { + kfree(ci_dyn); + return SWAP_ENTRY_INVALID; + } + + /* Lock before publishing: xa_alloc makes the cluster findable by offset.= */ + ci =3D &ci_dyn->ci; + spin_lock(&ci->lock); + + if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn, + XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1), + GFP_ATOMIC)) { + spin_unlock(&ci->lock); + swap_cluster_free_table(&ci_dyn->ci); + kfree(ci_dyn); + return SWAP_ENTRY_INVALID; + } + + offset =3D cluster_offset(si, ci); + return alloc_swap_scan_cluster(si, ci, folio, offset); +} + static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool f= orce) { long to_scan =3D 1; @@ -1056,7 +1117,9 @@ static void swap_reclaim_full_clusters(struct swap_in= fo_struct *si, bool force) spin_unlock(&ci->lock); nr_reclaim =3D __try_to_reclaim_swap(si, offset, TTRS_ANYWAY); - spin_lock(&ci->lock); + ci =3D swap_cluster_lock(si, offset); + if (!ci) + goto next; if (nr_reclaim) { offset +=3D abs(nr_reclaim); continue; @@ -1070,6 +1133,7 @@ static void swap_reclaim_full_clusters(struct swap_in= fo_struct *si, bool force) relocate_cluster(si, ci); =20 swap_cluster_unlock(ci); +next: if (to_scan <=3D 0) break; =20 @@ -1146,6 +1210,12 @@ static unsigned long cluster_alloc_swap_entry(struct= swap_info_struct *si, goto done; } =20 + if (swap_is_vswap(si)) { + found =3D vswap_alloc_cluster(si, folio); + if (found) + goto done; + } + if (!(si->flags & SWP_PAGE_DISCARD)) { found =3D alloc_swap_scan_list(si, &si->free_clusters, folio, false); if (found) @@ -1282,8 +1352,10 @@ static bool swap_usage_add(struct swap_info_struct *= si, unsigned int nr_entries) /* * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set, * remove it from the plist. + * + * Vswap is never on the avail list, so skip it. */ - if (unlikely(val =3D=3D si->pages)) { + if (unlikely(val =3D=3D si->pages) && !swap_is_vswap(si)) { del_from_avail_list(si, false); return true; } @@ -1298,8 +1370,10 @@ static void swap_usage_sub(struct swap_info_struct *= si, unsigned int nr_entries) /* * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set, * add it to the plist. + * + * Vswap is never on the avail list, so skip it. */ - if (unlikely(val & SWAP_USAGE_OFFLIST_BIT)) + if (unlikely(val & SWAP_USAGE_OFFLIST_BIT) && !swap_is_vswap(si)) add_to_avail_list(si, false); } =20 @@ -1346,6 +1420,10 @@ static void swap_range_free(struct swap_info_struct = *si, unsigned long offset, =20 static bool get_swap_device_info(struct swap_info_struct *si) { + /* The vswap device is always alive, so it needs no refcount. */ + if (swap_is_vswap(si)) + return true; + if (!percpu_ref_tryget_live(&si->users)) return false; /* @@ -1381,11 +1459,11 @@ static bool swap_alloc_fast(struct folio *folio) return false; =20 ci =3D swap_cluster_lock(si, offset); - if (cluster_is_usable(ci, order)) { + if (ci && cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset =3D cluster_offset(si, ci); alloc_swap_scan_cluster(si, ci, folio, offset); - } else { + } else if (ci) { swap_cluster_unlock(ci); } =20 @@ -1507,6 +1585,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t g= fp) if (!si) return 0; =20 + /* The source PTE pins the entry, so its cluster is alive. */ ci =3D __swap_offset_to_cluster(si, offset); ret =3D swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp); =20 @@ -1904,7 +1983,7 @@ struct swap_info_struct *get_swap_device(swp_entry_t = entry) return NULL; put_out: pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_offset, entry.val); - percpu_ref_put(&si->users); + put_swap_device(si); return NULL; } =20 @@ -2036,6 +2115,7 @@ static bool folio_maybe_swapped(struct folio *folio) VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio); =20 + /* Folio is locked and in swap cache, so ci->count > 0: cluster is alive.= */ ci =3D __swap_entry_to_cluster(entry); ci_off =3D swp_cluster_offset(entry); ci_end =3D ci_off + folio_nr_pages(folio); @@ -2230,6 +2310,9 @@ static int __find_hibernation_swap_type(dev_t device,= sector_t offset) =20 if (!(sis->flags & SWP_WRITEOK)) continue; + /* vswap has no bdev, so it is never a hibernation target. */ + if (swap_is_vswap(sis)) + continue; =20 if (device =3D=3D sis->bdev->bd_dev) { struct swap_extent *se =3D first_se(sis); @@ -2356,6 +2439,9 @@ int find_first_swap(dev_t *device) =20 if (!(sis->flags & SWP_WRITEOK)) continue; + /* vswap has no bdev, so it is never a hibernation target. */ + if (swap_is_vswap(sis)) + continue; *device =3D sis->bdev->bd_dev; spin_unlock(&swap_lock); return type; @@ -2572,8 +2658,10 @@ static int unuse_pte_range(struct vm_area_struct *vm= a, pmd_t *pmd, &vmf); } if (!folio) { + rcu_read_lock(); swp_tb =3D swap_table_get(__swap_entry_to_cluster(entry), swp_cluster_offset(entry)); + rcu_read_unlock(); if (swp_tb_get_count(swp_tb) <=3D 0) continue; return -ENOMEM; @@ -2719,8 +2807,10 @@ static unsigned int find_next_to_unuse(struct swap_i= nfo_struct *si, * allocations from this area (while holding swap_lock). */ for (i =3D prev + 1; i < si->max; i++) { + rcu_read_lock(); swp_tb =3D swap_table_get(__swap_offset_to_cluster(si, i), i % SWAPFILE_CLUSTER); + rcu_read_unlock(); if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb)) break; if ((i % LATENCY_LIMIT) =3D=3D 0) @@ -2959,6 +3049,11 @@ static int setup_swap_extents(struct swap_info_struc= t *sis, struct inode *inode =3D mapping->host; int ret; =20 + if (swap_is_vswap(sis)) { + *span =3D 0; + return 0; + } + ret =3D sio_pool_init(); if (ret) return ret; @@ -2984,15 +3079,24 @@ static int setup_swap_extents(struct swap_info_stru= ct *sis, =20 static void _enable_swap_info(struct swap_info_struct *si) { - atomic_long_add(si->pages, &nr_swap_pages); - total_swap_pages +=3D si->pages; + if (!swap_is_vswap(si)) { + atomic_long_add(si->pages, &nr_swap_pages); + total_swap_pages +=3D si->pages; + } =20 assert_spin_locked(&swap_lock); =20 - plist_add(&si->list, &swap_active_head); + /* + * Vswap has no backing file and no swapoff support, so keep it + * off swap_active_head (used by swapoff filename lookup and + * swap_sync_discard) and swap_avail_head (physical allocator). + */ + if (!swap_is_vswap(si)) { + plist_add(&si->list, &swap_active_head); =20 - /* Add back to available list */ - add_to_avail_list(si, true); + /* Add back to available list */ + add_to_avail_list(si, true); + } } =20 /* @@ -3036,12 +3140,31 @@ static void wait_for_allocation(struct swap_info_st= ruct *si) } } =20 -static void free_swap_cluster_info(struct swap_cluster_info *cluster_info, +static void free_swap_cluster_info(struct swap_info_struct *si, + struct swap_cluster_info *cluster_info, unsigned long maxpages) { + struct swap_cluster_info_dynamic *ci_dyn; struct swap_cluster_info *ci; + unsigned long idx; int i, nr_clusters =3D DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER); =20 + if (swap_is_vswap(si)) { + xa_for_each(&si->cluster_info_pool, idx, ci_dyn) { + ci =3D &ci_dyn->ci; + spin_lock(&ci->lock); + if (cluster_table_is_alloced(ci)) { + swap_cluster_assert_empty(ci, 0, + SWAPFILE_CLUSTER, true); + swap_cluster_free_table(ci); + } + spin_unlock(&ci->lock); + kfree(ci_dyn); + } + xa_destroy(&si->cluster_info_pool); + return; + } + if (!cluster_info) return; for (i =3D 0; i < nr_clusters; i++) { @@ -3188,7 +3311,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, special= file) mutex_unlock(&swapon_mutex); kfree(p->global_cluster); p->global_cluster =3D NULL; - free_swap_cluster_info(cluster_info, maxpages); + free_swap_cluster_info(p, cluster_info, maxpages); =20 inode =3D mapping->host; =20 @@ -3535,10 +3658,39 @@ static int setup_swap_clusters_info(struct swap_inf= o_struct *si, unsigned long maxpages) { unsigned long nr_clusters =3D DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER); - struct swap_cluster_info *cluster_info; + struct swap_cluster_info *cluster_info =3D NULL; + struct swap_cluster_info_dynamic *ci_dyn =3D NULL; int err =3D -ENOMEM; unsigned long i; =20 + /* A vswap device uses an xarray pool instead of a static array. */ + if (swap_is_vswap(si)) { + nr_clusters =3D 0; + xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC); + + /* + * Pre-allocate cluster 0 and mark slot 0 (header page) + * as bad so the allocator never hands out page offset 0. + */ + ci_dyn =3D kzalloc_obj(*ci_dyn, GFP_KERNEL); + if (!ci_dyn) + goto err; + spin_lock_init(&ci_dyn->ci.lock); + INIT_LIST_HEAD(&ci_dyn->ci.list); + + err =3D xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL); + if (err) { + kfree(ci_dyn); + goto err; + } + + err =3D swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false); + if (err) + goto err; + + goto setup_cluster_info; + } + cluster_info =3D kvzalloc_objs(*cluster_info, nr_clusters); if (!cluster_info) goto err; @@ -3582,6 +3734,7 @@ static int setup_swap_clusters_info(struct swap_info_= struct *si, goto err; } =20 +setup_cluster_info: INIT_LIST_HEAD(&si->free_clusters); INIT_LIST_HEAD(&si->full_clusters); INIT_LIST_HEAD(&si->discard_clusters); @@ -3603,10 +3756,16 @@ static int setup_swap_clusters_info(struct swap_inf= o_struct *si, } } =20 + /* Slot 0 is bad, so cluster 0 never empties. The rest of it is usable. */ + if (swap_is_vswap(si)) { + ci_dyn->ci.flags =3D CLUSTER_FLAG_NONFULL; + list_add_tail(&ci_dyn->ci.list, &si->nonfull_clusters[0]); + } + si->cluster_info =3D cluster_info; return 0; err: - free_swap_cluster_info(cluster_info, maxpages); + free_swap_cluster_info(si, cluster_info, maxpages); return err; } =20 @@ -3825,7 +3984,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialf= ile, int, swap_flags) si->global_cluster =3D NULL; inode =3D NULL; destroy_swap_extents(si, swap_file); - free_swap_cluster_info(si->cluster_info, si->max); + free_swap_cluster_info(si, si->cluster_info, si->max); si->cluster_info =3D NULL; /* * Clear the SWP_USED flag after all resources are freed so @@ -3956,3 +4115,79 @@ static int __init swapfile_init(void) return 0; } subsys_initcall(swapfile_init); + +struct swap_info_struct *vswap_si; +DEFINE_STATIC_KEY_FALSE(vswap_key); + +static bool vswap_enabled_early __initdata =3D IS_ENABLED(CONFIG_VSWAP_DEF= AULT_ON); + +static int __init early_vswap(char *buf) +{ + return kstrtobool(buf, &vswap_enabled_early); +} +early_param("vswap", early_vswap); + +/* vswap does no IO on its own. */ +static const struct swap_ops vswap_ops =3D { }; + +static int __init vswap_init(void) +{ + struct swap_info_struct *si; + unsigned long maxpages; + int err; + + if (!IS_ENABLED(CONFIG_64BIT)) { + if (vswap_enabled_early) + pr_warn("vswap: requires 64-bit architecture; vswap disabled, swapout f= alls back to direct physical swap\n"); + return 0; + } + + if (!vswap_enabled_early) + return 0; + + si =3D alloc_swap_info(); + if (IS_ERR(si)) { + pr_warn("vswap: alloc_swap_info failed (%ld); vswap disabled, swapout fa= lls back to direct physical swap\n", + PTR_ERR(si)); + return 0; + } + + maxpages =3D min(swapfile_maximum_size, + ALIGN_DOWN((unsigned long)UINT_MAX, SWAPFILE_CLUSTER)); + /* + * SWP_WRITEOK enables slot allocation. SWP_SOLIDSTATE selects + * per-CPU cluster allocation; vswap has no si->global_cluster. + */ + si->flags |=3D SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK; + si->ops =3D &vswap_ops; + si->bdev =3D NULL; + si->max =3D maxpages; + si->pages =3D maxpages - 1; + si->prio =3D SHRT_MAX; + si->list.prio =3D -si->prio; + si->avail_list.prio =3D -si->prio; + + err =3D setup_swap_clusters_info(si, NULL, maxpages); + if (err) + goto fail; + + mutex_lock(&swapon_mutex); + enable_swap_info(si); + mutex_unlock(&swapon_mutex); + + vswap_si =3D si; + pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages); + + /* Last: everything above must be visible before routing starts. */ + static_branch_enable(&vswap_key); + return 0; + +fail: + pr_warn("vswap: setup_swap_clusters_info failed (%d); vswap disabled, swa= pout falls back to direct physical swap\n", + err); + spin_lock(&swap_lock); + si->flags =3D 0; + spin_unlock(&swap_lock); + return 0; +} +late_initcall(vswap_init); diff --git a/mm/vswap.h b/mm/vswap.h new file mode 100644 index 000000000000..16395f357955 --- /dev/null +++ b/mm/vswap.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Virtual swap space + * + * Copyright (C) 2026 Nhat Pham + */ +#ifndef _MM_VSWAP_H +#define _MM_VSWAP_H + +#include +#include +#include "swap.h" + +#ifdef CONFIG_SWAP + +DECLARE_STATIC_KEY_FALSE(vswap_key); + +/* + * Only true once vswap_init() has published vswap_si, so callers never + * see the device half built. + */ +static inline bool vswap_is_enabled(void) +{ + return static_branch_unlikely(&vswap_key); +} + +static inline bool is_vswap_entry(swp_entry_t entry) +{ + return swap_is_vswap(__swap_entry_to_info(entry)); +} + +#endif /* CONFIG_SWAP */ + +#endif /* _MM_VSWAP_H */ diff --git a/mm/zswap.c b/mm/zswap.c index 37f34e406c8e..11643c52ea21 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1000,6 +1000,12 @@ static int zswap_writeback_entry(struct zswap_entry = *entry, if (!si) return -EEXIST; =20 + /* Vswap entries have no physical backing to write to. */ + if (swap_is_vswap(si)) { + put_swap_device(si); + return -EINVAL; + } + mpol =3D get_task_policy(current); folio =3D swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol, NO_INTERLEAVE_INDEX); --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oa1-f43.google.com (mail-oa1-f43.google.com [209.85.160.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E75E348986B for ; Tue, 25 Aug 2026 15:32:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671973; cv=none; b=n3rqrAY8nyKlOrjGF8ekqSKCFOZTb1m/I+FtikEOzvm5wLvTwJtzMkxsGwW0t6SlBcvq5NRLOitkSM8XBRDEFI3JUwH9yhUU+Hwd5TMWF7RQ3MmDNsQKaIHikr77q7hiiFbCfleL6/JaD+y6j2aeE4UeqNeXskWAWc1VQcbqAqc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671973; c=relaxed/simple; bh=a6xHskWFZwEmiH7h36JGISBtn5MXdYRqgyS+ZK0oruw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AB3BpBxmQPls2ZGJ0KZKH/850e1dvf6CgjGpmEnPe8ZD2WFRHfiE2ccJcscsy3CuNVH7fHNUnY3sAC0wu0aMlkPSQLJNUdq8mnHrkvy0gJdZTxeGEeANUSMKIZ6+rrIGmMRZp5VFN4vC6jdMSbLUrDxnniR5n4zUEr8H7dRl0zw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cThW4RsJ; arc=none smtp.client-ip=209.85.160.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cThW4RsJ" Received: by mail-oa1-f43.google.com with SMTP id 586e51a60fabf-448b0ff4a57so6741346fac.2 for ; Tue, 25 Aug 2026 08:32:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671964; x=1788276764; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=M4r8emXyXW5noklgLMUJWgd3OAUbwbCYwVs/Zcel7s4=; b=cThW4RsJEyvcHw/TShwR2eG4oAxQ+G+7/tzH4knCgRVyDmf9ifcmmIG3x4RS0132cp 2R5tiZd3VxXrI2woPCY+7rxQGY+a9os71YhY3FCq07Vq+yqMiQ5R58vxG1V+YjA00KNc 97myoWLd+LmlMvVrkK2W28oVxIsCDWE9GUMkdYjhY81QHmjYoIWid+uNJJx6kPCCptcy WQadqJ/6wGE0DhmKxG8iWL3NynJxJ8IvMevKLYwOfHs7W4V2RbuBNyP5/iWu/0wXgpJ7 DHk4jg3rikLMjgUCgfToVSTd/0hlnU4uXbHmS9GA0GX7sh0Zqv3NkMJ5De83S9oT9ZUo s30Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671964; x=1788276764; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=M4r8emXyXW5noklgLMUJWgd3OAUbwbCYwVs/Zcel7s4=; b=DQONISIxw7YhdSruBptToxYkylL7Pi9f+frDw+tf5Z+Hj2SsCx0A8nlo2YJM5CwXWk IO+7hWnzCKJpqYkzU6j8ru42pig1du6cGyM94Pyt4UurnDML1zzxLaVzhGxkVjjyXyZJ Z5XXmV+QFGbfo5kbJtrXVaMf/bwVqZ7s1mHQTFLr4bCj/Ugm1Rz1LtwNOdmK/7h6lzKt HZ42UmbjKUkYeNC6njui8XKchmx1MeF9iXZ8+bUfvVd9M2V7alXtEooIRCa7eS48nC5t 792M3uJSRURfhm+BpbWxAkM0MovC7iuNtmXEbe/ysoo4/fzZB8Z2ymFIXpEcIuaUzLiB 6gqA== X-Forwarded-Encrypted: i=1; AHgh+RpMWc7h0wjNHg8SjFDlMp445VNVOBzIRuekmW+aHKNlPrcf2au4Dw35meca/vSCaX+QUg6hw/DEFi23jqs=@vger.kernel.org X-Gm-Message-State: AFuF++nAVlYPS5O+XGotO8MIFXOj0+XVQN+EoppcXjEPCAxw56DbnIod 1ciGTAifvZz5BAE8UsGjHVAXmh2Z56rKsbkwRsqgj81G2MCwkOFJNBC7 X-Gm-Gg: AR+sD11N8pNkQPVmbSRSkLLrwKHAHOlHPHJz76VLIFwN14AcD7H8Jp617REGxvaz/4d PuB9bxqQKaLsuY5dEt4p3h0PPKVvUtG4UJCBF96KL8NCbbPAyFqlZ5uZXsICJMb22RMsD3Z22iI wwdHa7m9IYUldh1PMUo6uIpKTFhWhbLUybFm/pMjV1vNc3GKJXS/jeRx9JyIwZCV8YXNF19WYXq Zighs38xw6s8Gk8V8O7gSRC1+6B/4KQTZCewnuqUReoTftatxGI3ysQEC8LvOxBfPvzCNUgNyMA kgf8d8VkrWBejichLsGBqmqzHRUH/UCKxISVCFe+wptnQGDRsTEm6Nj56y+EHnEme0WvUb94cPp 4+7GrjK94tbG/Z/H1dCL3LkNFDa2ksTRtqQXF3cyTJAjFNQ4gTbAflGQwRepgMO7fzNBDSacNip f8IhnQ0Hfs9+SvwtOdFv8ZlPFWosT6HDMiw+71OdQtOCtCRziQilJ+G26TPRuM1ZQXB7jcNwthO mNHSWPbPDc= X-Received: by 2002:a05:6871:809:b0:44a:f310:407c with SMTP id 586e51a60fabf-4644a96fc52mr7150416fac.15.1787671963827; Tue, 25 Aug 2026 08:32:43 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:55::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-46383539b74sm7349640fac.14.2026.08.25.08.32.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:43 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 02/11] mm, swap: support zswap and zero-filled swap pages as vswap backends Date: Tue, 25 Aug 2026 08:32:28 -0700 Message-ID: <20260825153238.2695446-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Build the virtual swap layer on top of the swap-table infrastructure. Virtual swap entries decouple PTE swap entries from physical backing, allowing pages to be compressed by zswap (or detected as zero-filled) without pre-allocating a physical swap slot. This patch only supports zswap and zero-page backends. If zswap_store fails, the page stays dirty in the swap cache. Physical disk backing arrives in later patches. Zswap writeback of vswap-backed entries is also disabled: they have no physical slot to write back to yet, so the zswap shrinker (both the dynamic count path and the pool-full worker path) is skipped while vswap is enabled. Physical backing and real writeback come in later patches. THP swapin is disabled for vswap entries for now. vswap_alloc() only routes a swapout through vswap when vswap is enabled, either by CONFIG_VSWAP_DEFAULT_ON or by "vswap=3Don" on the kernel command line. It also declines the folio when zswap is off, or when the folio's cgroup is already over its zswap limit. In both cases writeout would have to find a physical slot anyway, so the indirection would buy nothing. Suggested-by: Kairui Song Signed-off-by: Nhat Pham --- include/linux/zswap.h | 3 + mm/memcontrol.c | 8 ++ mm/memory.c | 11 ++- mm/page_io.c | 12 ++- mm/shmem.c | 4 +- mm/swap.h | 1 + mm/swap_state.c | 8 ++ mm/swapfile.c | 178 ++++++++++++++++++++++++++++++++++++++-- mm/vmscan.c | 9 +- mm/vswap.h | 185 ++++++++++++++++++++++++++++++++++++++++++ mm/zswap.c | 65 ++++++++++++--- 11 files changed, 458 insertions(+), 26 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 30c193a1207e..4b4f211f3301 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -6,6 +6,7 @@ #include =20 struct lruvec; +struct zswap_entry; =20 extern atomic_long_t zswap_stored_pages; =20 @@ -28,6 +29,7 @@ unsigned long zswap_total_pages(void); bool zswap_store(struct folio *folio); int zswap_load(struct folio *folio); void zswap_invalidate(swp_entry_t swp); +void zswap_entry_free(struct zswap_entry *entry); int zswap_swapon(int type, unsigned long nr_pages); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); @@ -50,6 +52,7 @@ static inline int zswap_load(struct folio *folio) } =20 static inline void zswap_invalidate(swp_entry_t swp) {} +static inline void zswap_entry_free(struct zswap_entry *entry) {} static inline int zswap_swapon(int type, unsigned long nr_pages) { return 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 11b85f4b6828..8508fc7e2dfd 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -65,6 +65,7 @@ #include "internal.h" #include "swap.h" #include "swap_table.h" +#include "vswap.h" #include #include #include "slab.h" @@ -5801,6 +5802,13 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup = *memcg) { long nr_swap_pages =3D get_nr_swap_pages(); =20 + /* + * vswap zswap-backed swapout needs no physical slot, so gate anon + * reclaim on the swap.max headroom instead of the physical free count. + */ + if (vswap_is_enabled() && zswap_is_enabled()) + nr_swap_pages =3D PAGE_COUNTER_MAX; + if (mem_cgroup_disabled() || do_memsw_account()) return nr_swap_pages; for (; !mem_cgroup_is_root(memcg); memcg =3D parent_mem_cgroup(memcg)) diff --git a/mm/memory.c b/mm/memory.c index c54943302553..181a90fa2211 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -89,6 +89,7 @@ #include "pgalloc-track.h" #include "internal.h" #include "swap.h" +#include "vswap.h" =20 #if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) && !defined(CONFIG_COMPILE_TEST) #warning Unfortunate NUMA and NUMA Balancing config, growing page-frame fo= r last_cpupid. @@ -4661,6 +4662,9 @@ static inline bool should_try_to_free_swap(struct swa= p_info_struct *si, */ if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) return true; + /* A vswap entry holds no physical slot, so keeping it saves no IO. */ + if (is_vswap_entry(folio->swap)) + return true; if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || folio_test_mlocked(folio)) return true; @@ -4809,15 +4813,18 @@ static unsigned long thp_swapin_suitable_orders(str= uct vm_fault *vmf) if (unlikely(userfaultfd_armed(vma))) return 0; =20 + entry =3D softleaf_from_pte(vmf->orig_pte); + /* * A large swapped out folio could be partially or fully in zswap. We * lack handling for such cases, so fallback to swapping in order-0 * folio. + * + * THP swapin for vswap is not supported yet either. */ - if (!zswap_never_enabled()) + if (is_vswap_entry(entry) || !zswap_never_enabled()) return 0; =20 - entry =3D softleaf_from_pte(vmf->orig_pte); /* * Get a list of all the (large) orders below PMD_ORDER that are enabled * and suitable for swapping THP. diff --git a/mm/page_io.c b/mm/page_io.c index 3d0e78c17090..3bc69c2dbe4d 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -161,14 +161,19 @@ static void swap_zeromap_folio_set(struct folio *foli= o) struct obj_cgroup *objcg =3D get_obj_cgroup_from_folio(folio); int nr_pages =3D folio_nr_pages(folio); struct swap_cluster_info *ci; + unsigned int voff, i; swp_entry_t entry; - unsigned int i; =20 VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio); VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); =20 ci =3D swap_cluster_get_and_lock(folio); - for (i =3D 0; i < folio_nr_pages(folio); i++) { + if (is_vswap_entry(folio->swap)) { + /* Free any prior backing (e.g. ZSWAP entry from earlier swapout) */ + voff =3D swp_cluster_offset(folio->swap); + __vswap_release_backing(ci, voff, nr_pages); + } + for (i =3D 0; i < nr_pages; i++) { entry =3D page_swap_entry(folio_page(folio, i)); __swap_table_set_zero(ci, swp_cluster_offset(entry)); } @@ -236,6 +241,9 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio= *folio) */ swap_zeromap_folio_clear(folio); =20 + if (is_vswap_entry(folio->swap)) + folio_release_vswap_backing(folio); + if (zswap_store(folio)) { count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT); goto out_unlock; diff --git a/mm/shmem.c b/mm/shmem.c index 599665a3d6e7..d58f30b06255 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -86,6 +86,7 @@ static struct vfsmount *shm_mnt __ro_after_init; #include =20 #include "internal.h" +#include "vswap.h" =20 #define VM_ACCT(size) (PAGE_ALIGN(size) >> PAGE_SHIFT) =20 @@ -1618,7 +1619,8 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct fo= lio *folio, if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap) goto redirty; =20 - if (!total_swap_pages) + /* vswap doesn't contribute to total_swap_pages */ + if (!total_swap_pages && !(vswap_is_enabled() && zswap_is_enabled())) goto redirty; =20 /* diff --git a/mm/swap.h b/mm/swap.h index f18385dc9c6e..1cc9f3f5105e 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -71,6 +71,7 @@ struct swap_cluster_info_dynamic { struct swap_cluster_info ci; unsigned int index; /* for cluster_index() */ struct rcu_head rcu; + atomic_long_t *virtual_table; /* Backing pointers for vswap slots */ }; =20 /* All on-list cluster must have a non-zero flag. */ diff --git a/mm/swap_state.c b/mm/swap_state.c index a800abebba38..0385cc50ff93 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -27,6 +27,7 @@ #include "internal.h" #include "swap_table.h" #include "swap.h" +#include "vswap.h" =20 /* Swap readahead cluster size, as a power of 2 pages. */ static int page_cluster; @@ -197,6 +198,13 @@ static int __swap_cache_add_check(struct swap_cluster_= info *ci, if (nr =3D=3D 1) return 0; =20 + /* + * Reject a vswap batch so swap_cache_alloc_folio falls back to + * order 0. + */ + if (is_vswap_entry(targ_entry)) + return -EBUSY; + is_zero =3D __swap_table_test_zero(ci, ci_off); ci_off =3D round_down(ci_off, nr); ci_end =3D ci_off + nr; diff --git a/mm/swapfile.c b/mm/swapfile.c index ae88b91c92a2..e0b603a247c0 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -131,6 +131,18 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, perc= pu_swap_cluster) =3D { .lock =3D INIT_LOCAL_LOCK(), }; =20 +struct percpu_vswap_cluster { + unsigned long offset[SWAP_NR_ORDERS]; + local_lock_t lock; +}; + +static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = =3D { + .offset =3D { [0 ... SWAP_NR_ORDERS - 1] =3D SWAP_ENTRY_INVALID }, + .lock =3D INIT_LOCAL_LOCK(), +}; + +static bool vswap_alloc(struct folio *folio); + /* May return NULL on invalid type, caller must check for NULL return */ static struct swap_info_struct *swap_type_to_info(int type) { @@ -236,7 +248,8 @@ static int __try_to_reclaim_swap(struct swap_info_struc= t *si, =20 need_reclaim =3D ((flags & TTRS_ANYWAY) || ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) || - ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio))); + ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) && + !is_vswap_entry(folio->swap))); if (!need_reclaim || !folio_swapcache_freeable(folio)) goto out_unlock; =20 @@ -536,7 +549,9 @@ swap_cluster_populate(struct swap_info_struct *si, /* * Only cluster isolation from the allocator does table allocation. * Swap allocator uses percpu clusters and holds the local lock. + * vswap clusters are destroyed rather than freed to si->free_clusters. */ + VM_WARN_ON_ONCE(swap_is_vswap(si)); lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock); if (!(si->flags & SWP_SOLIDSTATE)) lockdep_assert_held(&si->global_cluster_lock); @@ -624,6 +639,7 @@ static void __free_cluster(struct swap_info_struct *si,= struct swap_cluster_info ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); xa_erase(&si->cluster_info_pool, ci_dyn->index); move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD); + vswap_cluster_free_vtable(ci); kfree_rcu(ci_dyn, rcu); return; } @@ -918,7 +934,8 @@ static bool cluster_scan_range(struct swap_info_struct = *si, if (swp_tb_is_null(swp_tb)) continue; if (swp_tb_is_folio(swp_tb) && !__swp_tb_get_count(swp_tb)) { - if (!vm_swap_full()) + /* vswap slots are abundant; never reclaim to reuse one */ + if (swap_is_vswap(si) || !vm_swap_full()) return false; *need_reclaim =3D true; continue; @@ -1026,6 +1043,10 @@ static unsigned int alloc_swap_scan_cluster(struct s= wap_info_struct *si, out: relocate_cluster(si, ci); swap_cluster_unlock(ci); + if (swap_is_vswap(si)) { + this_cpu_write(percpu_vswap_cluster.offset[order], next); + return found; + } if (si->flags & SWP_SOLIDSTATE) { this_cpu_write(percpu_swap_cluster.offset[order], next); this_cpu_write(percpu_swap_cluster.si[order], si); @@ -1078,6 +1099,12 @@ static unsigned int vswap_alloc_cluster(struct swap_= info_struct *si, return SWAP_ENTRY_INVALID; } =20 + if (vswap_cluster_alloc_vtable(ci_dyn, GFP_ATOMIC)) { + swap_cluster_free_table(&ci_dyn->ci); + kfree(ci_dyn); + return SWAP_ENTRY_INVALID; + } + /* Lock before publishing: xa_alloc makes the cluster findable by offset.= */ ci =3D &ci_dyn->ci; spin_lock(&ci->lock); @@ -1087,6 +1114,7 @@ static unsigned int vswap_alloc_cluster(struct swap_i= nfo_struct *si, GFP_ATOMIC)) { spin_unlock(&ci->lock); swap_cluster_free_table(&ci_dyn->ci); + vswap_cluster_free_vtable(&ci_dyn->ci); kfree(ci_dyn); return SWAP_ENTRY_INVALID; } @@ -1170,7 +1198,7 @@ static unsigned long cluster_alloc_swap_entry(struct = swap_info_struct *si, * Swapfile is not block device so unable * to allocate large entries. */ - if (order && !(si->flags & SWP_BLKDEV)) + if (order && !(si->flags & SWP_BLKDEV) && !swap_is_vswap(si)) return 0; =20 if (!(si->flags & SWP_SOLIDSTATE)) { @@ -1223,7 +1251,7 @@ static unsigned long cluster_alloc_swap_entry(struct = swap_info_struct *si, } =20 /* Try reclaim full clusters if free and nonfull lists are drained */ - if (vm_swap_full()) + if (!swap_is_vswap(si) && vm_swap_full()) swap_reclaim_full_clusters(si, false); =20 if (order < PMD_ORDER) { @@ -1384,7 +1412,8 @@ static void swap_range_alloc(struct swap_info_struct = *si, if (vm_swap_full()) schedule_work(&si->reclaim_work); } - atomic_long_sub(nr_entries, &nr_swap_pages); + if (!swap_is_vswap(si)) + atomic_long_sub(nr_entries, &nr_swap_pages); } =20 static void swap_range_free(struct swap_info_struct *si, unsigned long off= set, @@ -1394,8 +1423,10 @@ static void swap_range_free(struct swap_info_struct = *si, unsigned long offset, void (*swap_slot_free_notify)(struct block_device *, unsigned long); unsigned int i; =20 - for (i =3D 0; i < nr_entries; i++) - zswap_invalidate(swp_entry(si->type, offset + i)); + if (!swap_is_vswap(si)) { + for (i =3D 0; i < nr_entries; i++) + zswap_invalidate(swp_entry(si->type, offset + i)); + } =20 if (si->flags & SWP_BLKDEV) swap_slot_free_notify =3D @@ -1414,7 +1445,8 @@ static void swap_range_free(struct swap_info_struct *= si, unsigned long offset, * only after the above cleanups are done. */ smp_wmb(); - atomic_long_add(nr_entries, &nr_swap_pages); + if (!swap_is_vswap(si)) + atomic_long_add(nr_entries, &nr_swap_pages); swap_usage_sub(si, nr_entries); } =20 @@ -1806,6 +1838,57 @@ static int swap_dup_entries_cluster(struct swap_info= _struct *si, return err; } =20 +static bool vswap_alloc(struct folio *folio) +{ + unsigned int order =3D folio_order(folio); + struct swap_cluster_info *ci; + struct obj_cgroup *objcg; + unsigned long offset; + bool may_zswap; + + if (!vswap_is_enabled() || !zswap_is_enabled()) + return false; + + /* + * If zswap will not take the folio, writeout has to find a physical + * slot anyway. We are just incurring indirection overhead + * unnecessarily. + */ + objcg =3D get_obj_cgroup_from_folio(folio); + may_zswap =3D !objcg || obj_cgroup_may_zswap(objcg); + if (objcg) + obj_cgroup_put(objcg); + if (!may_zswap) + return false; + + local_lock(&percpu_vswap_cluster.lock); + offset =3D this_cpu_read(percpu_vswap_cluster.offset[order]); + + if (offset !=3D SWAP_ENTRY_INVALID) { + ci =3D swap_cluster_lock(vswap_si, offset); + if (ci && cluster_is_usable(ci, order)) { + if (cluster_is_empty(ci)) + offset =3D cluster_offset(vswap_si, ci); + alloc_swap_scan_cluster(vswap_si, ci, folio, offset); + } else if (ci) { + swap_cluster_unlock(ci); + } + } + + if (!folio_test_swapcache(folio)) + cluster_alloc_swap_entry(vswap_si, folio); + + if (folio_test_swapcache(folio)) { + /* alloc_swap_scan_cluster updated percpu offset already */ + local_unlock(&percpu_vswap_cluster.lock); + return true; + } + + this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID); + local_unlock(&percpu_vswap_cluster.lock); + return false; +} + /** * folio_alloc_swap - allocate swap space for a folio * @folio: folio we want to move to swap @@ -1842,12 +1925,16 @@ int folio_alloc_swap(struct folio *folio) } } =20 + if (vswap_alloc(folio)) + goto done; + again: local_lock(&percpu_swap_cluster.lock); if (!swap_alloc_fast(folio)) swap_alloc_slow(folio); local_unlock(&percpu_swap_cluster.lock); =20 +done: if (!order && unlikely(!folio_test_swapcache(folio))) { if (swap_sync_discard()) goto again; @@ -1863,6 +1950,73 @@ int folio_alloc_swap(struct folio *folio) return 0; } =20 +/** + * __vswap_release_backing - release the backing of a range of vtable slots + * @ci: the locked vswap cluster + * @ci_start: first slot offset within @ci + * @nr: number of slots + * + * Releases the backing of each slot in [@ci_start, @ci_start + @nr). + * Clears the zero marks if set. + * + * Context: caller must hold @ci->lock. + */ +void __vswap_release_backing(struct swap_cluster_info *ci, + unsigned int ci_start, unsigned int nr) +{ + struct swap_cluster_info_dynamic *ci_dyn; + unsigned int ci_off; + unsigned long vt; + + lockdep_assert_held(&ci->lock); + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + + for (ci_off =3D ci_start; ci_off < ci_start + nr; ci_off++) { + vt =3D __vtable_get(ci_dyn, ci_off); + + switch (vtable_type(vt)) { + case VSWAP_ZSWAP: + zswap_entry_free(vtable_to_zswap(vt)); + break; + case VSWAP_NONE: + break; + default: + /* VSWAP_ZERO/VSWAP_FOLIO are return-only, not vtable tags */ + break; + } + + __vtable_set(ci_dyn, ci_off, VSWAP_NONE); + /* Zero-backed state lives in swap_table; clear it too. */ + if (__swap_table_test_zero(ci, ci_off)) + __swap_table_clear_zero(ci, ci_off); + } +} + +/** + * folio_release_vswap_backing() - Drop all backing for a folio's vswap en= try. + * @folio: the folio, occupying a virtual swap entry. + * + * Release whatever backing the folio's virtual swap slots currently hold = and + * reset them to empty, so a fresh backing can be installed. Used when a + * folio's swap backend is replaced. + * + * Context: Caller must hold the folio lock; @folio must be in the swap ca= che + * and occupy a virtual swap entry. + */ +void folio_release_vswap_backing(struct folio *folio) +{ + struct swap_cluster_info *ci; + int nr =3D folio_nr_pages(folio); + unsigned int voff; + + ci =3D __swap_entry_to_cluster(folio->swap); + voff =3D swp_cluster_offset(folio->swap); + + spin_lock(&ci->lock); + __vswap_release_backing(ci, voff, nr); + spin_unlock(&ci->lock); +} + /** * folio_dup_swap() - Increase swap count of swap entries of a folio. * @folio: folio with swap entries bounded. @@ -2003,6 +2157,9 @@ void __swap_cluster_free_entries(struct swap_info_str= uct *si, =20 VM_WARN_ON(ci->count < nr_pages); =20 + if (swap_is_vswap(si)) + __vswap_release_backing(ci, ci_start, nr_pages); + ci->count -=3D nr_pages; do { old_tb =3D __swap_table_get(ci, ci_off); @@ -3159,6 +3316,7 @@ static void free_swap_cluster_info(struct swap_info_s= truct *si, swap_cluster_free_table(ci); } spin_unlock(&ci->lock); + vswap_cluster_free_vtable(ci); kfree(ci_dyn); } xa_destroy(&si->cluster_info_pool); @@ -3688,6 +3846,10 @@ static int setup_swap_clusters_info(struct swap_info= _struct *si, if (err) goto err; =20 + err =3D vswap_cluster_alloc_vtable(ci_dyn, GFP_KERNEL); + if (err) + goto err; + goto setup_cluster_info; } =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index c1404a59523d..5cc40db37f6b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -69,6 +69,7 @@ #include "internal.h" #include "page_alloc.h" #include "swap.h" +#include "vswap.h" =20 #define CREATE_TRACE_POINTS #include @@ -353,6 +354,9 @@ static inline bool can_reclaim_anon_pages(struct mem_cg= roup *memcg, */ if (get_nr_swap_pages() > 0) return true; + /* vswap doesn't contribute to nr_swap_pages */ + if (vswap_is_enabled() && zswap_is_enabled()) + return true; } else { /* Is the memcg below its swap limit? */ if (mem_cgroup_get_nr_swap_pages(memcg) > 0) @@ -1524,7 +1528,8 @@ static unsigned int shrink_folio_list(struct list_hea= d *folio_list, activate_locked: /* Not a candidate for swapping, so reclaim swap space. */ if (folio_test_swapcache(folio) && - (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio))) + ((mem_cgroup_swap_full(folio) && !is_vswap_entry(folio->swap)) || + folio_test_mlocked(folio))) folio_free_swap(folio); VM_BUG_ON_FOLIO(folio_test_active(folio), folio); if (!folio_test_mlocked(folio)) { @@ -2681,7 +2686,7 @@ static bool can_age_anon_pages(struct lruvec *lruvec, struct scan_control *sc) { /* Aging the anon LRU is valuable if swap is present: */ - if (total_swap_pages > 0) + if (total_swap_pages > 0 || (vswap_is_enabled() && zswap_is_enabled())) return true; =20 /* Also valuable if anon pages can be demoted: */ diff --git a/mm/vswap.h b/mm/vswap.h index 16395f357955..5334c77b6b84 100644 --- a/mm/vswap.h +++ b/mm/vswap.h @@ -11,8 +11,22 @@ #include #include "swap.h" =20 +struct zswap_entry; + +/* + * VSWAP_ZERO and VSWAP_FOLIO are return-only values synthesized from + * swap_table state; the rest are stored in the vtable per slot. + */ +enum vswap_backing_type { + VSWAP_NONE =3D 0, + VSWAP_ZSWAP =3D 1, + VSWAP_ZERO, + VSWAP_FOLIO, +}; + #ifdef CONFIG_SWAP =20 +#include "swap_table.h" DECLARE_STATIC_KEY_FALSE(vswap_key); =20 /* @@ -29,6 +43,177 @@ static inline bool is_vswap_entry(swp_entry_t entry) return swap_is_vswap(__swap_entry_to_info(entry)); } =20 +/* + * Virtual table entry encoding for vswap clusters. + * + * Each entry in ci_dyn->virtual_table stores the backing type and + * pointer for a virtual swap slot. Tag in low 3 bits, payload in + * upper 61 bits. + * + * NONE: |----- 0000 ------|000| - no separate backend pointer + * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bi= ts) + * + * Pointer payloads (ZSWAP) are stored directly with the tag OR'd into the + * low bits (kernel pointers are >=3D 8-byte aligned, same approach as xar= ray). + * + * vtable[i] =3D NONE does not by itself mean "free". The swap_table entry + * and the per-slot zero flag carry the rest of the state. The full + * per-slot state table is: + * + * vtable[i] | swap_table[i] | zero | meaning + * ----------+---------------+-------+-------------------------------- + * NONE | NULL | clear | truly free / unbacked + * NONE | PFN | clear | folio cached, no backing + * NONE | shadow | clear | evicted, no backing: data lost + * NONE | * | set | zero-backed; cached if PFN set + * ZSWAP | PFN | clear | folio cached + zswap entry + * ZSWAP | shadow / NULL | clear | evicted, only in zswap + * + * Locking: a slot's vtable entry (the vswap entry's backend) is only + * stable while the caller owns and holds the lock on that entry's swap + * cache folio. The cluster lock (ci_dyn->ci.lock) only makes an individual + * vtable read atomic, and by itself does not give the caller the right to + * change the backend. A backend read without the folio lock is + * best-effort and must be re-validated under the folio lock before + * being acted on. + * + * Zero-backed slots use the swap_table per-slot zero flag (same as + * direct-mapped physical swap), via __swap_table_test_zero() and friends, + * which fall back to ci->zero_bitmap where the flag does not fit. Cached + * folios are read out of the swap_table PFN entry; there is no separate F= OLIO + * vtable type because the folio pointer would duplicate that PFN and + * would go stale on folio migration / split. + */ + +#define VTABLE_TAG_BITS 3 +#define VTABLE_TAG_MASK ((1UL << VTABLE_TAG_BITS) - 1) + +static inline enum vswap_backing_type vtable_type(unsigned long vt) +{ + return vt & VTABLE_TAG_MASK; +} + +static inline struct zswap_entry *vtable_to_zswap(unsigned long vt) +{ + VM_WARN_ON(vtable_type(vt) !=3D VSWAP_ZSWAP); + return (struct zswap_entry *)(vt & ~VTABLE_TAG_MASK); +} + +/* Virtual table accessors */ + +static inline unsigned long __vtable_get(struct swap_cluster_info_dynamic = *ci_dyn, + unsigned int off) +{ + VM_WARN_ON_ONCE(off >=3D SWAPFILE_CLUSTER); + return atomic_long_read(&ci_dyn->virtual_table[off]); +} + +static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn, + unsigned int off, unsigned long vt) +{ + VM_WARN_ON_ONCE(off >=3D SWAPFILE_CLUSTER); + atomic_long_set(&ci_dyn->virtual_table[off], vt); +} + +/** + * vswap_lock_cluster - look up and lock the vswap cluster for an entry + * @entry: the virtual swap entry + * @voff: out param, receives @entry's slot offset within the cluster + * + * Return: the locked vswap cluster, or NULL if @entry has no live cluster. + */ +static inline struct swap_cluster_info_dynamic * +vswap_lock_cluster(swp_entry_t entry, unsigned int *voff) +{ + struct swap_cluster_info *ci; + + ci =3D swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry)); + if (!ci) + return NULL; + *voff =3D swp_cluster_offset(entry); + return container_of(ci, struct swap_cluster_info_dynamic, ci); +} + +void __vswap_release_backing(struct swap_cluster_info *ci, + unsigned int ci_start, unsigned int nr); + +/** + * vswap_zswap_store - record a zswap entry as the backing for a vswap ent= ry. + * @entry: the vswap entry + * @ze: the zswap entry now holding @entry's compressed data + * + * Releases @entry's previous backing, and sets the zswap entry @ze as the= new + * backing. + * + * Context: takes and drops the vswap cluster lock internally. + */ +static inline void vswap_zswap_store(swp_entry_t entry, + struct zswap_entry *ze) +{ + struct swap_cluster_info_dynamic *ci_dyn; + unsigned int voff; + + ci_dyn =3D vswap_lock_cluster(entry, &voff); + __vswap_release_backing(&ci_dyn->ci, voff, 1); + __vtable_set(ci_dyn, voff, (unsigned long)ze | VSWAP_ZSWAP); + swap_cluster_unlock(&ci_dyn->ci); +} + +/** + * vswap_zswap_load - return the zswap entry backing a vswap entry + * @entry: the virtual swap entry + * + * Context: takes and drops the vswap cluster lock internally. + * Return: the backing zswap entry, or NULL if @entry is not zswap-backed. + */ +static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry) +{ + struct swap_cluster_info_dynamic *ci_dyn; + unsigned int voff; + unsigned long vt; + + ci_dyn =3D vswap_lock_cluster(entry, &voff); + if (!ci_dyn) + return NULL; + vt =3D __vtable_get(ci_dyn, voff); + swap_cluster_unlock(&ci_dyn->ci); + + if (vtable_type(vt) !=3D VSWAP_ZSWAP) + return NULL; + return vtable_to_zswap(vt); +} + +void folio_release_vswap_backing(struct folio *folio); + +static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dyna= mic *ci_dyn, + gfp_t gfp) +{ + ci_dyn->virtual_table =3D kcalloc(SWAPFILE_CLUSTER, + sizeof(*ci_dyn->virtual_table), gfp); + return ci_dyn->virtual_table ? 0 : -ENOMEM; +} + +static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci) +{ + struct swap_cluster_info_dynamic *ci_dyn; + + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + kfree(ci_dyn->virtual_table); + ci_dyn->virtual_table =3D NULL; +} + +#else /* !CONFIG_SWAP */ + +static inline bool vswap_is_enabled(void) +{ + return false; +} + +static inline bool is_vswap_entry(swp_entry_t entry) +{ + return false; +} + #endif /* CONFIG_SWAP */ =20 #endif /* _MM_VSWAP_H */ diff --git a/mm/zswap.c b/mm/zswap.c index 11643c52ea21..e5a9922b951a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -38,6 +38,7 @@ #include =20 #include "swap.h" +#include "vswap.h" #include "internal.h" =20 /********************************* @@ -234,6 +235,25 @@ static inline struct xarray *swap_zswap_tree(swp_entry= _t swp) >> ZSWAP_ADDRESS_SPACE_SHIFT]; } =20 +static struct zswap_entry *zswap_entry_load(swp_entry_t swp) +{ + if (is_vswap_entry(swp)) + return vswap_zswap_load(swp); + return xa_load(swap_zswap_tree(swp), swp_offset(swp)); +} + +static struct zswap_entry *zswap_entry_store(swp_entry_t swp, + struct zswap_entry *entry) +{ + if (is_vswap_entry(swp)) { + vswap_zswap_store(swp, entry); + return NULL; + } + + return xa_store(swap_zswap_tree(swp), swp_offset(swp), entry, + GFP_KERNEL); +} + #define zswap_pool_debug(msg, p) \ pr_debug("%s pool %s\n", msg, (p)->tfm_name) =20 @@ -762,7 +782,7 @@ static void zswap_entry_cache_free(struct zswap_entry *= entry) * Carries out the common pattern of freeing an entry's zsmalloc allocatio= n, * freeing the entry itself, and decrementing the number of stored pages. */ -static void zswap_entry_free(struct zswap_entry *entry) +void zswap_entry_free(struct zswap_entry *entry) { zswap_lru_del(entry); zs_free(entry->pool->zs_pool, entry->handle); @@ -1208,6 +1228,9 @@ static unsigned long zswap_shrinker_count(struct shri= nker *shrinker, if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg)) return 0; =20 + if (vswap_is_enabled()) + return 0; + /* * The shrinker resumes swap writeback, which will enter block * and may enter fs. XXX: Harmonize with vmscan.c __GFP_FS @@ -1290,6 +1313,8 @@ static struct shrinker *zswap_alloc_shrinker(void) * Return: 0 if at least one entry was written back, -EAGAIN if entries * were scanned but none could be written back, or -ENOENT if @memcg has * writeback disabled, is a zombie cgroup, or has empty zswap LRUs. + * + * Also returns -ENOENT when vswap is enabled. */ static int shrink_memcg(struct mem_cgroup *memcg) { @@ -1298,6 +1323,9 @@ static int shrink_memcg(struct mem_cgroup *memcg) if (!mem_cgroup_zswap_writeback_enabled(memcg)) return -ENOENT; =20 + if (vswap_is_enabled()) + return -ENOENT; + /* * Skip zombies because their LRUs are reparented and we would be * reclaiming from the parent instead of the dead memcg. @@ -1326,6 +1354,13 @@ static void shrink_worker(struct work_struct *w) int ret, failures =3D 0, attempts =3D 0; unsigned long thr; =20 + /* + * When vswap is enabled, zswap entries are almost all vswap backed, + * with no slot to write back to. + */ + if (vswap_is_enabled()) + return; + /* Reclaim down to the accept threshold */ thr =3D zswap_accept_thr_pages(); =20 @@ -1428,9 +1463,7 @@ static bool zswap_store_page(struct page *page, if (!zswap_compress(page, entry, pool)) goto compress_failed; =20 - old =3D xa_store(swap_zswap_tree(page_swpentry), - swp_offset(page_swpentry), - entry, GFP_KERNEL); + old =3D zswap_entry_store(page_swpentry, entry); if (xa_is_err(old)) { int err =3D xa_err(old); =20 @@ -1499,7 +1532,7 @@ bool zswap_store(struct folio *folio) struct mem_cgroup *memcg =3D NULL; struct zswap_pool *pool; bool ret =3D false; - long index; + long index =3D 0; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1554,13 +1587,19 @@ bool zswap_store(struct folio *folio) if (!ret && zswap_pool_reached_full) queue_work(shrink_wq, &zswap_shrink_work); check_old: + if (ret) + return ret; + /* * If the zswap store fails or zswap is disabled, we must invalidate * the possibly stale entries which were previously stored at the * offsets corresponding to each page of the folio. Otherwise, * writeback could overwrite the new data in the swapfile. */ - if (!ret) { + if (is_vswap_entry(swp)) { + if (index > 0) + folio_release_vswap_backing(folio); + } else { unsigned type =3D swp_type(swp); pgoff_t offset =3D swp_offset(swp); struct zswap_entry *entry; @@ -1600,8 +1639,7 @@ bool zswap_store(struct folio *folio) int zswap_load(struct folio *folio) { swp_entry_t swp =3D folio->swap; - pgoff_t offset =3D swp_offset(swp); - struct xarray *tree =3D swap_zswap_tree(swp); + struct swap_info_struct *si =3D __swap_entry_to_info(swp); struct zswap_entry *entry; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1620,7 +1658,7 @@ int zswap_load(struct folio *folio) return -EINVAL; } =20 - entry =3D xa_load(tree, offset); + entry =3D zswap_entry_load(swp); if (!entry) return -ENOENT; =20 @@ -1643,8 +1681,13 @@ int zswap_load(struct folio *folio) * compression work. */ folio_mark_dirty(folio); - xa_erase(tree, offset); - zswap_entry_free(entry); + + if (swap_is_vswap(si)) { + folio_release_vswap_backing(folio); + } else { + xa_erase(swap_zswap_tree(swp), swp_offset(swp)); + zswap_entry_free(entry); + } =20 folio_unlock(folio); return 0; --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5DC5489882 for ; Tue, 25 Aug 2026 15:32:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671973; cv=none; b=f7eF1h+R5aW2DV50sQOEvusCdGZ9VejsGVGjRI5Zwyaw7fmPyIJjM9mctf/h76btKCQpeoE6Z1galFVi53yP7/tklmUkmJ5YF3KExffSMWGmTJmXESHxwaFLWK4bpDNPPucHZW43Z7XUYXaNSHgE+MesgrNHnXhBN3pYsk72lbg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671973; c=relaxed/simple; bh=I0qBM9lAgRNZAWyiQ/VCRWTAdZZsBgP8b49S427QfMc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t9cbYbwSeTMrjaRjlX4llDxkmVQqMzxCQ6DQnP0KlQhjanLUK8hm2JGw7/PXMrta3kgpyMa4Zl6ZP3IZ4eDTK0M0ArurkskYl+v+uapjrx0GDrZbQdQzKC8rwXBt647vNXEndAxvHyXajlJF3G9kNucbeOxbvHfxR9JlLEcNyPg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SG7QeVPm; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SG7QeVPm" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-495c49f8eccso4342392b6e.3 for ; Tue, 25 Aug 2026 08:32:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671966; x=1788276766; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=QhSwWX2E/V88OSmZiCSHWiUU/JQeVGndYgjPiEldJdU=; b=SG7QeVPmcKGGQOAXP/HmVy5I4IzjUH9dKXvWYwCAYoTYlooNYOM5EWOxArQhVlvixK wWfJbNhD93yvMhxjv08VLmZJet9c7Il8bE2+Mt/7SboQ9ZIoRvgF3x8f9cW8ss4xib+T P+2eH6JoiVHkrjcgVCUYyRpMzB7yfs33Nrt/bX02ZPNB7+5QVJBvz/k3Wuq+mzlFzGyJ q18WQS897uIKtv3oH9ofxI4VmknSL516xAokmo5U0IJl6mlPCz/pj0Lc5LPQx+o2jWdI K4C8zMwwYmID+t4it3kZOzTbgv1Hu/sWWV8TiedPtiEVkdp58pC2PEBGsgkoIw2WEVRB UOnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671966; x=1788276766; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=QhSwWX2E/V88OSmZiCSHWiUU/JQeVGndYgjPiEldJdU=; b=JjadREC5m7mY7qD/FUzkb0yUmU7i3GBd7rLTIfQYGgMUjSRH0TQPXdA8woxPVs+LMC FXJozV3oZYhGpFbXPXfHCkGjWsy83uuKkdbpzlO2PyM6GxI4vAMrmdLg2lnco/1fI1Xq x1sy0dsPqAt7xZiFrfc+pS3YY+cxgf/o4L71JxK8zzjm4ZMXtoSvZCXnueU9K7cS2wJ8 7XFVyCWA26OR8dNtujEsZsaL0ibCFK1UVM/AMmPPln9LzQm3/FnSQJjHJhwOQ0QLoc3G FwWPetFAMhhL+VUR8pEnq7zQtUgYr5sIjyZziz4ispKelbvZ4SmKyI3bkjJrNwlQ0K5l 1mcA== X-Forwarded-Encrypted: i=1; AHgh+RqraxZCU40sNT8lPHTQoebPEXtaUPNYDUQDrFrDCa+wYM5i+/praymGJhGvvnuKZNje5QeMP7Urq0Sos1A=@vger.kernel.org X-Gm-Message-State: AFuF++mmn16JnfiIJBeNTLzCfDxRRBA7VAk1rrs0LPajyk2HglCMpJSb VO4V/KZptG3G9w050PxDnqfd7hkErLbcRjNm9SohFn+NoDVnNOZX8u9v X-Gm-Gg: AR+sD12toHK8CwkkXx/I/NxzlNZrdCPzPPXo4GOoQJtqYGG1FltVhd2vReslo/dbreI JJzWk3wjGI1GWcSZlr0S7UxzbVcF0F+3JS8dxtsSYneP7tdLnvrmNjTLjYPgofUGIjFYWnzOY8I qZRiAni5L4UgDDVwR+W5Df70QBwunw9t5Sbv6gNdpXRLjH7768p1jvUtMhKb22kBixFOJk6VtxR FpalPR01Mc9uklxD+yDZUOS0T40kR5rxnLm4cpXyHn81ue1esJY4siAxdhSsjsRAkN4a8/oHzJg 5Q90QKxZMDEvaH2VWB+wPB6H21ydr/X/dhZshnKmqdPDRhLzojxLgkWYYSK/V9IYgKvKS9Db+9R PQt2f1O0laItLm6lIaUj+YJTdCcLwjVz/cZmZrM6ZxwWI3tU7dSuX0A9uW7kpYhz3sWVfNVqXeI Ah64CP6/jsC9KiXxbuKKqF/Ocrz0w6PkU6SuN2hDmPE8OVL7iw2KHRAZgOKZ24Xz8cFqmzfgVg7 N/P6pMaxO0= X-Received: by 2002:a05:6808:448d:b0:4b3:3e1:4a25 with SMTP id 5614622812f47-4b350513b86mr7973433b6e.11.1787671966152; Tue, 25 Aug 2026 08:32:46 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:5b::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-4b32c690a5asm6225953b6e.8.2026.08.25.08.32.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:45 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 03/11] mm, swap: prepare the swap IO path for vswap Date: Tue, 25 Aug 2026 08:32:29 -0700 Message-ID: <20260825153238.2695446-4-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for adding a physical swap backend for vswap, make the swap IO path able to submit IO for a swap entry other than folio->swap. The swap IO path derives the target device and sector from folio->swap. For a vswap folio backed by a physical slot that entry is virtual, so it identifies neither the backing device nor the sector to submit IO against. Compute the sector from an explicit entry (swap_folio_sector becomes swap_entry_sector), thread that entry through swap_add_folio, __swap_writepage and ops->can_merge, and stash it in swap_iocb so the submit and completion paths address the IO from it rather than from folio->swap. This lets the batching path serve both vswap entries (backed by a physical slot) and physical entries mapped directly into PTEs. All callers pass folio->swap for now, so there is no functional change. Signed-off-by: Nhat Pham --- include/linux/swap.h | 2 +- include/linux/swap_ops.h | 9 ++++--- mm/page_io.c | 54 +++++++++++++++++++--------------------- mm/swap.h | 3 ++- mm/swapfile.c | 6 ++--- mm/zswap.c | 2 +- 6 files changed, 38 insertions(+), 38 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 5339323486d5..f57f4aeeb822 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -388,7 +388,7 @@ extern int __swap_count(swp_entry_t entry); extern bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t en= try); extern int swp_swapcount(swp_entry_t entry); extern struct swap_info_struct *get_swap_device(swp_entry_t entry); -sector_t swap_folio_sector(struct folio *folio); +sector_t swap_entry_sector(swp_entry_t entry); =20 /* * If there is an existing swap slot reference (swap entry) and the caller diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h index 57ac6c703f68..223c84548bde 100644 --- a/include/linux/swap_ops.h +++ b/include/linux/swap_ops.h @@ -12,6 +12,7 @@ struct swap_iocb { struct bio_vec bvecs[SWAP_CLUSTER_MAX]; int nr_bvecs; int len; + swp_entry_t entry; /* first slot in the batch; addresses the IO */ }; =20 struct swap_io_ctx { @@ -30,15 +31,15 @@ struct swap_io_ctx { struct swap_ops { unsigned int flags; =20 - bool (*can_merge)(struct folio *folio, struct folio *prev_folio, - size_t prev_folio_size, int rw); + bool (*can_merge)(struct folio *folio, swp_entry_t phys, + struct swap_iocb *sio, int rw); void (*submit_write)(struct swap_io_ctx *ctx); void (*submit_read)(struct swap_io_ctx *ctx); }; =20 void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *= iter); -bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, - size_t prev_folio_size, int rw); +bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys, + struct swap_iocb *sio, int rw); int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *= ops); =20 #endif /* _MM_SWAP_OPS_H */ diff --git a/mm/page_io.c b/mm/page_io.c index 3bc69c2dbe4d..a3aaa61d6c79 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -265,7 +265,7 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio= *folio) return AOP_WRITEPAGE_ACTIVATE; } =20 - __swap_writepage(ctx, folio); + __swap_writepage(ctx, folio, folio->swap); return 0; out_unlock: folio_unlock(folio); @@ -334,24 +334,22 @@ int sio_pool_init(void) } =20 static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio, - int rw) + swp_entry_t phys, int rw) { - struct swap_info_struct *sis =3D __swap_entry_to_info(folio->swap); - struct bio_vec *last_bv =3D &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1]; - struct folio *prev_folio =3D bvec_folio(last_bv); - size_t prev_folio_size =3D folio_size(prev_folio); + struct swap_info_struct *sis =3D __swap_entry_to_info(phys); =20 if (ctx->sis !=3D sis) return false; - return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw); + return sis->ops->can_merge(folio, phys, ctx->sio, rw); } =20 -static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, i= nt rw) +static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, + swp_entry_t phys, int rw) { - struct swap_info_struct *sis =3D __swap_entry_to_info(folio->swap); + struct swap_info_struct *sis =3D __swap_entry_to_info(phys); struct swap_iocb *sio =3D ctx->sio; =20 - if (sio && !swap_can_merge(ctx, folio, rw)) { + if (sio && !swap_can_merge(ctx, folio, phys, rw)) { if (rw =3D=3D WRITE) swap_write_submit(ctx); else @@ -364,6 +362,7 @@ static void swap_add_folio(struct swap_io_ctx *ctx, str= uct folio *folio, int rw) ctx->sio =3D sio =3D mempool_alloc(sio_pool, GFP_NOIO); sio->nr_bvecs =3D 0; sio->len =3D 0; + sio->entry =3D phys; } bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0); sio->len +=3D folio_size(folio); @@ -384,7 +383,8 @@ static void swap_add_folio(struct swap_io_ctx *ctx, str= uct folio *folio, int rw) } } =20 -void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio) +void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio, + swp_entry_t phys) { VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); =20 @@ -400,7 +400,7 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct f= olio *folio) =20 folio_start_writeback(folio); folio_unlock(folio); - swap_add_folio(ctx, folio, WRITE); + swap_add_folio(ctx, folio, phys, WRITE); } =20 /* @@ -504,7 +504,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct fo= lio *folio) =20 /* We have to read from slower devices. Increase zswap protection. */ zswap_folio_swapin(folio); - swap_add_folio(ctx, folio, READ); + swap_add_folio(ctx, folio, folio->swap, READ); =20 finish: if (workingset) { @@ -536,8 +536,6 @@ static void swap_fs_write_complete(struct kiocb *iocb, = long ret) bool failed =3D ret !=3D sio->len; =20 if (failed) { - struct page *page =3D sio->bvecs[0].bv_page; - /* * In the case of swap-over-nfs, this can be a temporary failure * if the system has limited memory for allocating transmit @@ -545,7 +543,7 @@ static void swap_fs_write_complete(struct kiocb *iocb, = long ret) * folio_rotate_reclaimable but rate-limit the messages. */ pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n", - ret, swap_dev_pos(page_swap_entry(page))); + ret, swap_dev_pos(sio->entry)); } =20 swap_write_end(sio, failed); @@ -617,7 +615,7 @@ static void swap_bdev_submit_write(struct swap_io_ctx *= ctx) bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs), REQ_OP_WRITE | REQ_SWAP); bio->bi_iter.bi_size =3D sio->len; - bio->bi_iter.bi_sector =3D swap_folio_sector(bio_first_folio_all(bio)); + bio->bi_iter.bi_sector =3D swap_entry_sector(sio->entry); bio_associate_blkg_from_page(bio, bio_first_folio_all(bio)); =20 if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) { @@ -637,7 +635,7 @@ static void swap_bdev_submit_read(struct swap_io_ctx *c= tx) bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs), REQ_OP_READ); bio->bi_iter.bi_size =3D sio->len; - bio->bi_iter.bi_sector =3D swap_folio_sector(bio_first_folio_all(bio)); + bio->bi_iter.bi_sector =3D swap_entry_sector(sio->entry); =20 if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) { /* @@ -655,13 +653,14 @@ static void swap_bdev_submit_read(struct swap_io_ctx = *ctx) } } =20 -static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_fo= lio, - size_t prev_folio_size, int rw) +static bool swap_bdev_can_merge(struct folio *folio, swp_entry_t phys, + struct swap_iocb *sio, int rw) { - if (swap_folio_sector(folio) !=3D - swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT)) + if (swap_entry_sector(phys) !=3D + swap_entry_sector(sio->entry) + (sio->len >> SECTOR_SHIFT)) return false; - if (rw =3D=3D WRITE && !folio_blkg_can_merge(folio, prev_folio)) + if (rw =3D=3D WRITE && !folio_blkg_can_merge(folio, + bvec_folio(&sio->bvecs[sio->nr_bvecs - 1]))) return false; return true; } @@ -677,7 +676,7 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw= , struct iov_iter *iter) struct swap_iocb *sio =3D ctx->sio; =20 init_sync_kiocb(&sio->iocb, ctx->sis->swap_file); - sio->iocb.ki_pos =3D swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap); + sio->iocb.ki_pos =3D swap_dev_pos(sio->entry); if (rw =3D=3D WRITE) sio->iocb.ki_complete =3D swap_fs_write_complete; else @@ -688,11 +687,10 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int = rw, struct iov_iter *iter) } EXPORT_SYMBOL_GPL(swap_fs_prepare_rw); =20 -bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio, - size_t prev_folio_size, int rw) +bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys, + struct swap_iocb *sio, int rw) { - return swap_dev_pos(folio->swap) =3D=3D - swap_dev_pos(prev_folio->swap) + prev_folio_size; + return swap_dev_pos(phys) =3D=3D swap_dev_pos(sio->entry) + sio->len; } EXPORT_SYMBOL_GPL(swap_fs_can_merge); =20 diff --git a/mm/swap.h b/mm/swap.h index 1cc9f3f5105e..f2b64920ae5b 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -320,7 +320,8 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct fo= lio *folio); void swap_read_submit(struct swap_io_ctx *ctx); void swap_write_submit(struct swap_io_ctx *ctx); int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio); -void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio); +void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio, + swp_entry_t phys); =20 /* linux/mm/swap_state.c */ extern struct address_space swap_space __read_mostly; diff --git a/mm/swapfile.c b/mm/swapfile.c index e0b603a247c0..f5fd01307a67 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -343,14 +343,14 @@ offset_to_swap_extent(struct swap_info_struct *sis, u= nsigned long offset) BUG(); } =20 -sector_t swap_folio_sector(struct folio *folio) +sector_t swap_entry_sector(swp_entry_t entry) { - struct swap_info_struct *sis =3D __swap_entry_to_info(folio->swap); + struct swap_info_struct *sis =3D __swap_entry_to_info(entry); struct swap_extent *se; sector_t sector; pgoff_t offset; =20 - offset =3D swp_offset(folio->swap); + offset =3D swp_offset(entry); se =3D offset_to_swap_extent(sis, offset); sector =3D se->start_block + (offset - se->start_page); return sector << (PAGE_SHIFT - 9); diff --git a/mm/zswap.c b/mm/zswap.c index e5a9922b951a..54506935ceba 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1076,7 +1076,7 @@ static int zswap_writeback_entry(struct zswap_entry *= entry, folio_set_reclaim(folio); =20 /* start writeback */ - __swap_writepage(&ctx, folio); + __swap_writepage(&ctx, folio, folio->swap); swap_write_submit(&ctx); =20 out: --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85FCE48987C for ; Tue, 25 Aug 2026 15:32:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671980; cv=none; b=IyHQfNlzsQcWgR8hK0cL5a/QsEMhgO3ro8oX7K8w3AxFJ5antO6KcNm8g7Tkb/5k6s+KltHntWo1ZQhbk1oo+Z+Uk3uJOzh6m+BzKx0ZMAtwOMzlnUnUdk9l+bzyc58+pimrBYH3Zw3FcPkzdjbgHOGZNvg4UGHZ2+VgA+6zH6c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671980; c=relaxed/simple; bh=Ir7mgcdvJ+mFq7iklAS0VdxMoNgKMp+NYlmRq0N0Cfo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LtpZFRmBcdC0jrjLErGrxjeeVKCMqWldGR5lUleiJVCR+IoDxcNGoxawsTDbZRjJfqPMm9Ll9HEe/6mmuDeuIDsaWeSKIGjSwIqRqaaT2fgnq1DrsQu1VfvLAiliTu4V6OV+P29wOHomZqRSnX9uXYGEDKOHvJtNVEtNuZNloVU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JT8IKlTd; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JT8IKlTd" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-7ee125ec926so5387830a34.2 for ; Tue, 25 Aug 2026 08:32:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671970; x=1788276770; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ZiuIUjxHZJqBtUh4LQgpSrUu+vgbaWcl2GsgQdNoZNs=; b=JT8IKlTd8Q2ftVik3OdNOn/dpIN2SX+7Iqk8tY7QZGc20xEwyVwHOG1eUwztw1dBIS kgg5o4Ej5GLswb2dneYGmK/LEPImCpMiB+IBswfPZ1vyWUB3iPxO3Oz91EOgojO5wedA c84GDrU5QaXCecwl6KYg1vB10hnUvN96w7cnwL5e5cAENFv4orm8PLZiidYLNNiaoZl9 f/uQkxuEvZY5+/AfmElm5y/lXq2sdPxDfx+VXaEwlH6LMe5jOsPFQBo0AYOIWz9jnrhy mEZ49iRw1BZrv4wFxZQpoFnuF0vkyCzlpV+JIt2IYOa35zACuitLdvHiKtPNOpW6Dumj 3xkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671970; x=1788276770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ZiuIUjxHZJqBtUh4LQgpSrUu+vgbaWcl2GsgQdNoZNs=; b=sEu/wUXzmoYnWd+yIXz3IswghOlhrb3bKe6n3XgZpH3yBn12jDbaWLijqUv8eHjVy0 96QrlUlxhrCTr6RhzO6d5Eh4+6JK3Z/Gwuk5tL6FBNY/srj6p1tINI5ps4xNC8JuFnb4 m86qZB1IoaRDnbmkx9Ib/1qvu8hr8y2lwsni+0PA3xLZAxWdmSHyYNhzVixGHV9Fc6p4 13pf+apEULeOexgGwBfJu81BkO+JxDqJ3G23TMbLAYtRJY0rbvPOdPa7PaRaRn37092f LaO8ZRs7y7WxjHKU8E2qMCw2S5Zazv2aLg/rP01gY93cmW6F673/3A8rY6lzUczcdWx4 jh9g== X-Forwarded-Encrypted: i=1; AHgh+RqEGACBAub/dPsD/X4/OyUd25PuYG8Sn9n3L/mB97pfPhEfqwJANgOjBB+z4/tUYxarIeu3qoOPwG9Kvo4=@vger.kernel.org X-Gm-Message-State: AFuF++mzxGdqxY/XfEPeONfAyxn15o1Nw0Z391SLv6FXfj0QthiYs+A5 I6HkogN3I3yf37vuWsjvpTNnWOx9vWRl+A5TKcCkk+Rtywe+dVEXSyjq X-Gm-Gg: AR+sD10qZ6KYvwIj0nls4HxpE0GdwUrBd3lJ2W3ysUJuR1ZD/SC+roy3I5w/OQCatW6 pOqmbcx/frE6S9pcyIX07BIEXY29hdk11lzenx71jh8q57xbNhQCTFxWrOv3WW+JW4Ju2rmn03A +VkQWS81BbpVjSewErc+HscBq+RGB6bG49s0R0ghVo/LEsfcnqatgTaDsdfNH/XeWJc+2BNlw2p GXalrZtkPuIEx7nlq8N32DdCg/cJNCvWy+ghQSg4umDiqeVMJc71zvEeIYGj/LeerCH/sg5irj2 9syRiC8x8hj4dAYv9VxGybMZrmuQfogdUkNF1ne4bpB6K6UtdsZ0dGCPPoBhZ6YHYmM5EXJWq8h ZNVP0GSoUGU3ncvKpckr5teLJFMdZUgL7BpfBFKw0/YzYPOxVpg0e7YlmCJYkLur7AWYQJnV3oA bglK3o448Hw7xTEXXvjIB2XNV56KWGCmRjVI9AZ8PMMgHE4jvGWLnLcB7z92MxhaG8JCogFdUqI huS+p9jdjo= X-Received: by 2002:a05:6830:2a03:b0:7e6:8955:c52d with SMTP id 46e09a7af769-7f4613368demr40060364a34.5.1787671968548; Tue, 25 Aug 2026 08:32:48 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:5b::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7f48fc7fb48sm7732206a34.19.2026.08.25.08.32.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:47 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 04/11] mm, swap: support physical swap as a vswap backend Date: Tue, 25 Aug 2026 08:32:30 -0700 Message-ID: <20260825153238.2695446-5-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add physical swap as a backend for the virtual swap layer. When zswap declines a page, the swapout path allocates a physical slot on demand for swap out. Each vswap entry's physical slot is tracked via a pointer-tagged swap_table entry on the physical cluster (an rmap back to the vswap entry). Physical readahead scans a whole offset window and would trip over these rmap slots, so __swap_cache_add_check() now skips swp_tb_is_pointer() entries. Nothing is lost: a backing slot is faulted through its owning vswap entry, never through the physical offset. swapoff reads each vswap entry back through its rmap slot before freeing the physical slot. A failed read leaves the folio not uptodate, so drop it instead of marking it dirty: dirtying would write uninitialised memory out to swap, and the loss is now reported as SIGBUS on the next fault rather than silently returning stale data. If zswap is disabled at the host level or for the folio's cgroup, the folio still gets a physical swap slot, bypassing vswap and mapping the slot directly into the PTEs. In practice the swapfile backend is only reached when zswap declines the folio at swap_writeout() time, most often because the pool is full. The machinery is in place, but its main consumer is not. Zswap writeback to physical swap is added in a following patch. Reclaim of physical slots backing cache-only vswap entries follows it. Suggested-by: Kairui Song Signed-off-by: Nhat Pham --- mm/memory.c | 13 +- mm/page_io.c | 42 ++++-- mm/swap_state.c | 6 +- mm/swap_table.h | 42 +++++- mm/swapfile.c | 358 +++++++++++++++++++++++++++++++++++++++++++----- mm/vmscan.c | 2 +- mm/vswap.h | 198 +++++++++++++++++++++++++- mm/zswap.c | 2 +- 8 files changed, 602 insertions(+), 61 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 181a90fa2211..dc4dd72ce73b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4660,10 +4660,13 @@ static inline bool should_try_to_free_swap(struct s= wap_info_struct *si, * are fast, and meanwhile, swap cache pinning the slot deferring the * release of metadata or fragmentation is a more critical issue. */ - if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) + if (swap_entry_backend_has_flag(si, folio->swap, SWP_SYNCHRONOUS_IO)) return true; - /* A vswap entry holds no physical slot, so keeping it saves no IO. */ - if (is_vswap_entry(folio->swap)) + /* + * Non-swapfile backends cannot be reused for future swapouts. + * Free the swap slot unless backed by contiguous physical swap. + */ + if (!folio_phys_swap_backed(folio)) return true; if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || folio_test_mlocked(folio)) @@ -4970,7 +4973,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swap_update_readahead(folio, vma, vmf->address); if (!folio) { /* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */ - if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) + if (swap_entry_backend_has_flag(si, entry, SWP_SYNCHRONOUS_IO)) folio =3D swapin_sync(entry, GFP_HIGHUSER_MOVABLE, thp_swapin_suitable_orders(vmf) | BIT(0), vmf, NULL, 0); @@ -5135,7 +5138,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ exclusive =3D true; } else if (exclusive && folio_test_writeback(folio) && - data_race(si->flags & SWP_STABLE_WRITES)) { + swap_entry_backend_has_flag(si, entry, SWP_STABLE_WRITES)) { /* * This is tricky: not all swap backends support * concurrent page modifications while under writeback. diff --git a/mm/page_io.c b/mm/page_io.c index a3aaa61d6c79..b36a898358c6 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -209,6 +209,7 @@ static void swap_zeromap_folio_clear(struct folio *foli= o) */ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio) { + swp_entry_t phys; int ret =3D 0; =20 if (folio_free_swap(folio)) @@ -241,8 +242,14 @@ int swap_writeout(struct swap_io_ctx *ctx, struct foli= o *folio) */ swap_zeromap_folio_clear(folio); =20 + /* + * For vswap: release stale non-swapfile backings (e.g. ZSWAP from a + * previous swapout cycle) so zswap_store or folio_realloc_swap + * starts on clean slots. Contiguous PHYS backing is preserved for + * reuse by folio_realloc_swap. + */ if (is_vswap_entry(folio->swap)) - folio_release_vswap_backing(folio); + folio_release_non_phys_swap_backing(folio); =20 if (zswap_store(folio)) { count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT); @@ -258,11 +265,19 @@ int swap_writeout(struct swap_io_ctx *ctx, struct fol= io *folio) rcu_read_unlock(); =20 /* - * A vswap folio has no physical slot to write to, so keep it dirty. + * A vswap folio with no backend needs a physical slot to write to. + * zswap_store rolled back any partial vtable state on failure, so + * PHYS backing from a prior cycle is still there to reuse. If none + * is free, keep it dirty. */ if (is_vswap_entry(folio->swap)) { - folio_mark_dirty(folio); - return AOP_WRITEPAGE_ACTIVATE; + phys =3D folio_realloc_swap(folio); + if (!phys.val) { + folio_mark_dirty(folio); + return AOP_WRITEPAGE_ACTIVATE; + } + __swap_writepage(ctx, folio, phys); + return 0; } =20 __swap_writepage(ctx, folio, folio->swap); @@ -473,6 +488,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct fo= lio *folio) bool workingset =3D folio_test_workingset(folio); unsigned long pflags; bool in_thrashing; + swp_entry_t phys; =20 VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -497,14 +513,24 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct = folio *folio) if (zswap_load(folio) !=3D -ENOENT) goto finish; =20 - if (unlikely(swap_is_vswap(sis))) { - folio_unlock(folio); - goto finish; + /* + * Resolve the physical slot to read from. A vswap entry keeps + * folio->swap virtual, so map it to its physical backing; a folio with + * no backing has nothing to read. + */ + if (swap_is_vswap(sis)) { + phys =3D vswap_to_phys(folio->swap); + if (!phys.val) { + folio_unlock(folio); + goto finish; + } + } else { + phys =3D folio->swap; } =20 /* We have to read from slower devices. Increase zswap protection. */ zswap_folio_swapin(folio); - swap_add_folio(ctx, folio, folio->swap, READ); + swap_add_folio(ctx, folio, phys, READ); =20 finish: if (workingset) { diff --git a/mm/swap_state.c b/mm/swap_state.c index 0385cc50ff93..c0441783b8e7 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -186,6 +186,9 @@ static int __swap_cache_add_check(struct swap_cluster_i= nfo *ci, return -ENOENT; ci_off =3D swp_cluster_offset(targ_entry); old_tb =3D __swap_table_get(ci, ci_off); + /* Physical readahead can hit a vswap-backing rmap slot; skip it. */ + if (swp_tb_is_pointer(old_tb)) + return -ENOENT; if (swp_tb_is_folio(old_tb)) return -EEXIST; if (!__swp_tb_get_count(old_tb)) @@ -210,7 +213,8 @@ static int __swap_cache_add_check(struct swap_cluster_i= nfo *ci, ci_end =3D ci_off + nr; do { old_tb =3D __swap_table_get(ci, ci_off); - if (unlikely(swp_tb_is_folio(old_tb) || + if (unlikely(swp_tb_is_pointer(old_tb) || + swp_tb_is_folio(old_tb) || !__swp_tb_get_count(old_tb) || is_zero !=3D __swap_table_test_zero(ci, ci_off) || (memcg_id && *memcg_id !=3D __swap_cgroup_get(ci, ci_off)))) diff --git a/mm/swap_table.h b/mm/swap_table.h index 868aae6c820f..08d2494a8ee6 100644 --- a/mm/swap_table.h +++ b/mm/swap_table.h @@ -4,6 +4,7 @@ =20 #include #include +#include #include "swap.h" =20 extern struct swap_info_struct *vswap_si; @@ -30,7 +31,7 @@ struct swap_memcg_table { * NULL: |---------------- 0 ---------------| - Free slot * Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot * PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot - * Pointer: |----------- Pointer ----------|100| - (Unused) + * Pointer: |-------- vswap offset --------|100| - vswap rmap * Bad: |------------- 1 -------------|1000| - Bad slot * * COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit, @@ -51,9 +52,8 @@ struct swap_memcg_table { * - PFN: Swap slot is in use, and cached. Memcg info is recorded on the p= age * struct. * - * - Pointer: Unused yet. `0b100` is reserved for potential pointer usage - * because only the lower three bits can be used as a marker for 8 bytes - * aligned pointers. + * - Pointer: Reverse map from a physical slot to the vswap entry that owns + * it. See the layout below. * * - Bad: Swap slot is reserved, protects swap header or holes on swap dev= ices. */ @@ -370,4 +370,38 @@ static inline unsigned short __swap_cgroup_clear(struc= t swap_cluster_info *ci, } #endif =20 +/* + * Pointer-tagged swap table entry: rmap for vswap-backing physical slots. + * + * On physical clusters, a Pointer-tagged entry stores the offset of the + * vswap entry that owns this physical slot (the reverse map). Only the + * offset is stored; the swap type is implicit (always vswap_si->type, + * since there is exactly one vswap device). + * + * Pointer: |---- vswap offset ----|100| + */ +#define SWP_TB_PTR_MARK_BITS 3 +#define SWP_TB_PTR_MARK 0b100UL +#define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1) +#define SWP_RMAP_ENTRY_MASK (~SWP_TB_PTR_MARK_MASK) + +static inline bool swp_tb_is_pointer(unsigned long swp_tb) +{ + return (swp_tb & SWP_TB_PTR_MARK_MASK) =3D=3D SWP_TB_PTR_MARK; +} + +static inline unsigned long swp_entry_to_swp_tb_ptr(swp_entry_t entry) +{ + return (swp_offset(entry) << SWP_TB_PTR_MARK_BITS) | SWP_TB_PTR_MARK; +} + +static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb) +{ + unsigned long offset; + + VM_WARN_ON(!swp_tb_is_pointer(swp_tb)); + offset =3D (swp_tb & SWP_RMAP_ENTRY_MASK) >> SWP_TB_PTR_MARK_BITS; + return swp_entry(vswap_si->type, offset); +} + #endif diff --git a/mm/swapfile.c b/mm/swapfile.c index f5fd01307a67..66bcbb112142 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -249,7 +249,7 @@ static int __try_to_reclaim_swap(struct swap_info_struc= t *si, need_reclaim =3D ((flags & TTRS_ANYWAY) || ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) || ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) && - !is_vswap_entry(folio->swap))); + folio_phys_swap_backed(folio))); if (!need_reclaim || !folio_swapcache_freeable(folio)) goto out_unlock; =20 @@ -955,6 +955,8 @@ static bool __swap_cluster_alloc_entries(struct swap_in= fo_struct *si, { unsigned int order; unsigned long nr_pages; + swp_entry_t vswap_entry, v; + unsigned int i; =20 lockdep_assert_held(&ci->lock); =20 @@ -974,8 +976,26 @@ static bool __swap_cluster_alloc_entries(struct swap_i= nfo_struct *si, order =3D folio_order(folio); nr_pages =3D 1 << order; swap_cluster_assert_empty(ci, ci_off, nr_pages, false); - __swap_cache_add_folio(ci, folio, swp_entry(si->type, - ci_off + cluster_offset(si, ci))); + if (folio_test_swapcache(folio)) { + /* + * Folio already in the swap cache: we are allocating + * physical backing for its vswap entry. Point each + * physical slot back at its own vswap entry + * (Pointer-tagged rmap). + */ + VM_WARN_ON(!is_vswap_entry(folio->swap)); + vswap_entry =3D folio->swap; + for (i =3D 0; i < nr_pages; i++) { + v =3D vswap_entry; + v.val +=3D i; + __swap_table_set(ci, ci_off + i, + swp_entry_to_swp_tb_ptr(v)); + } + } else { + __swap_cache_add_folio(ci, folio, + swp_entry(si->type, + ci_off + cluster_offset(si, ci))); + } } else if (IS_ENABLED(CONFIG_HIBERNATION)) { order =3D 0; nr_pages =3D 1; @@ -1474,12 +1494,14 @@ static bool get_swap_device_info(struct swap_info_s= truct *si) * Fast path try to get swap entries with specified order from current * CPU's swap entry pool (a cluster). */ -static bool swap_alloc_fast(struct folio *folio) +static swp_entry_t swap_alloc_fast(struct folio *folio) { unsigned int order =3D folio_order(folio); struct swap_cluster_info *ci; struct swap_info_struct *si; - unsigned int offset; + unsigned long offset, found =3D 0; + + lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock); =20 /* * Once allocated, swap_info_struct will never be completely freed, @@ -1488,25 +1510,28 @@ static bool swap_alloc_fast(struct folio *folio) si =3D this_cpu_read(percpu_swap_cluster.si[order]); offset =3D this_cpu_read(percpu_swap_cluster.offset[order]); if (!si || !offset || !get_swap_device_info(si)) - return false; + return (swp_entry_t){}; =20 ci =3D swap_cluster_lock(si, offset); if (ci && cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset =3D cluster_offset(si, ci); - alloc_swap_scan_cluster(si, ci, folio, offset); + found =3D alloc_swap_scan_cluster(si, ci, folio, offset); } else if (ci) { swap_cluster_unlock(ci); } =20 put_swap_device(si); - return folio_test_swapcache(folio); + if (found) + return swp_entry(si->type, found); + return (swp_entry_t){}; } =20 /* Rotate the device and switch to a new cluster */ -static void swap_alloc_slow(struct folio *folio) +static swp_entry_t swap_alloc_slow(struct folio *folio) { struct swap_info_struct *si, *next; + unsigned long found; =20 spin_lock(&swap_avail_lock); start_over: @@ -1515,12 +1540,12 @@ static void swap_alloc_slow(struct folio *folio) plist_requeue(&si->avail_list, &swap_avail_head); spin_unlock(&swap_avail_lock); if (get_swap_device_info(si)) { - cluster_alloc_swap_entry(si, folio); + found =3D cluster_alloc_swap_entry(si, folio); put_swap_device(si); - if (folio_test_swapcache(folio)) - return; + if (found) + return swp_entry(si->type, found); if (folio_test_large(folio)) - return; + return (swp_entry_t){}; } =20 spin_lock(&swap_avail_lock); @@ -1538,6 +1563,7 @@ static void swap_alloc_slow(struct folio *folio) goto start_over; } spin_unlock(&swap_avail_lock); + return (swp_entry_t){}; } =20 /* @@ -1889,6 +1915,23 @@ static bool vswap_alloc(struct folio *folio) return false; } =20 +static swp_entry_t folio_alloc_phys_swap(struct folio *folio) +{ + swp_entry_t entry; + +again: + local_lock(&percpu_swap_cluster.lock); + entry =3D swap_alloc_fast(folio); + if (!entry.val) + entry =3D swap_alloc_slow(folio); + local_unlock(&percpu_swap_cluster.lock); + + if (!entry.val && !folio_order(folio) && swap_sync_discard()) + goto again; + + return entry; +} + /** * folio_alloc_swap - allocate swap space for a folio * @folio: folio we want to move to swap @@ -1925,20 +1968,8 @@ int folio_alloc_swap(struct folio *folio) } } =20 - if (vswap_alloc(folio)) - goto done; - -again: - local_lock(&percpu_swap_cluster.lock); - if (!swap_alloc_fast(folio)) - swap_alloc_slow(folio); - local_unlock(&percpu_swap_cluster.lock); - -done: - if (!order && unlikely(!folio_test_swapcache(folio))) { - if (swap_sync_discard()) - goto again; - } + if (!vswap_alloc(folio)) + folio_alloc_phys_swap(folio); =20 /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */ if (unlikely(mem_cgroup_try_charge_swap(folio))) @@ -1950,6 +1981,11 @@ int folio_alloc_swap(struct folio *folio) return 0; } =20 +static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi, + struct swap_cluster_info *pci, + unsigned int ci_start, + unsigned int nr_pages); + /** * __vswap_release_backing - release the backing of a range of vtable slots * @ci: the locked vswap cluster @@ -1965,8 +2001,12 @@ void __vswap_release_backing(struct swap_cluster_inf= o *ci, unsigned int ci_start, unsigned int nr) { struct swap_cluster_info_dynamic *ci_dyn; + struct swap_info_struct *psi; + unsigned long phys_start =3D 0, phys_end =3D 0; + unsigned int phys_type =3D 0; unsigned int ci_off; unsigned long vt; + swp_entry_t phys; =20 lockdep_assert_held(&ci->lock); ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); @@ -1974,7 +2014,32 @@ void __vswap_release_backing(struct swap_cluster_inf= o *ci, for (ci_off =3D ci_start; ci_off < ci_start + nr; ci_off++) { vt =3D __vtable_get(ci_dyn, ci_off); =20 + /* The free helper takes one contiguous run within one cluster. */ + if (phys_start !=3D phys_end && + (vtable_type(vt) !=3D VSWAP_SWAPFILE || + swp_type(vtable_to_phys(vt)) !=3D phys_type || + swp_offset(vtable_to_phys(vt)) !=3D phys_end || + phys_end % SWAPFILE_CLUSTER =3D=3D 0)) { + psi =3D __swap_type_to_info(phys_type); + __swap_cluster_free_phys_backing(psi, + __swap_entry_to_cluster( + swp_entry(phys_type, phys_start)), + phys_start % SWAPFILE_CLUSTER, + phys_end - phys_start); + phys_start =3D phys_end =3D 0; + } + switch (vtable_type(vt)) { + case VSWAP_SWAPFILE: + if (phys_start =3D=3D phys_end) { + phys =3D vtable_to_phys(vt); + phys_start =3D swp_offset(phys); + phys_end =3D phys_start + 1; + phys_type =3D swp_type(phys); + } else { + phys_end++; + } + break; case VSWAP_ZSWAP: zswap_entry_free(vtable_to_zswap(vt)); break; @@ -1990,6 +2055,15 @@ void __vswap_release_backing(struct swap_cluster_inf= o *ci, if (__swap_table_test_zero(ci, ci_off)) __swap_table_clear_zero(ci, ci_off); } + + if (phys_start !=3D phys_end) { + psi =3D __swap_type_to_info(phys_type); + __swap_cluster_free_phys_backing(psi, + __swap_entry_to_cluster( + swp_entry(phys_type, phys_start)), + phys_start % SWAPFILE_CLUSTER, + phys_end - phys_start); + } } =20 /** @@ -2017,6 +2091,100 @@ void folio_release_vswap_backing(struct folio *foli= o) spin_unlock(&ci->lock); } =20 +/** + * folio_release_non_phys_swap_backing() - Drop a folio's non-physical vsw= ap backing. + * @folio: the folio, occupying a virtual swap entry. + * + * Release the zswap backing recorded for @folio's virtual swap entry, + * leaving the slots empty so the writeout path can install fresh physical + * backing. Does nothing when the entry is already backed by physical + * swapfile slots, which are kept for reuse, or when it has no backing + * beyond the swap cache folio itself. + * + * Context: Caller must hold the folio lock; @folio must be in the swap ca= che + * and occupy a virtual swap entry. + */ +void folio_release_non_phys_swap_backing(struct folio *folio) +{ + struct swap_cluster_info *ci; + struct swap_cluster_info_dynamic *ci_dyn; + int nr =3D folio_nr_pages(folio); + unsigned int voff; + unsigned long vt; + enum vswap_backing_type type; + + ci =3D __swap_entry_to_cluster(folio->swap); + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + voff =3D swp_cluster_offset(folio->swap); + + spin_lock(&ci->lock); + /* A folio's slots cannot mix swapfile with other backends. */ + vt =3D __vtable_get(ci_dyn, voff); + type =3D vtable_type(vt); + + if (type =3D=3D VSWAP_SWAPFILE || type =3D=3D VSWAP_NONE) { + spin_unlock(&ci->lock); + return; + } + + __vswap_release_backing(ci, voff, nr); + spin_unlock(&ci->lock); +} + +/** + * folio_realloc_swap() - Back a virtual swap folio with a physical swap s= lot. + * @folio: the folio, occupying a virtual swap entry. + * + * Ensure @folio's virtual swap entry has physical (swapfile) backing, + * allocating a physical slot on demand if it has none. If @folio is + * already physically backed, the existing physical entry is returned + * unchanged. + * + * Context: Caller must hold the folio lock; @folio must be in the swap ca= che + * and occupy a virtual swap entry. + * Return: The physical swap entry now backing @folio, or an empty entry + * (.val =3D=3D 0) on failure. + */ +swp_entry_t folio_realloc_swap(struct folio *folio) +{ + swp_entry_t vswap_entry =3D folio->swap; + struct swap_cluster_info *ci; + struct swap_cluster_info_dynamic *ci_dyn; + unsigned int voff; + swp_entry_t phys_entry =3D {}; + swp_entry_t pe; + int i, nr =3D folio_nr_pages(folio); + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); + VM_WARN_ON(!is_vswap_entry(vswap_entry)); + + phys_entry =3D vswap_to_phys(vswap_entry); + if (phys_entry.val) + return phys_entry; + + phys_entry =3D folio_alloc_phys_swap(folio); + if (!phys_entry.val) + return (swp_entry_t){}; + + voff =3D swp_cluster_offset(vswap_entry); + + ci =3D __swap_entry_to_cluster(vswap_entry); + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + spin_lock(&ci->lock); + /* + * Install PHYS backing without freeing any prior contents of the + * vtable. Releasing the old backing is the caller's job. + */ + for (i =3D 0; i < nr; i++) { + pe.val =3D phys_entry.val + i; + __vtable_set(ci_dyn, voff + i, vtable_mk_phys(pe)); + } + spin_unlock(&ci->lock); + + return phys_entry; +} + /** * folio_dup_swap() - Increase swap count of swap entries of a folio. * @folio: folio with swap entries bounded. @@ -2141,6 +2309,47 @@ struct swap_info_struct *get_swap_device(swp_entry_t= entry) return NULL; } =20 +/* + * Common tail for freeing swap slots: device-level accounting + * and cluster list management. + */ +static void __swap_cluster_finish_free(struct swap_info_struct *si, + struct swap_cluster_info *ci, + unsigned int ci_start, + unsigned int nr_pages) +{ + lockdep_assert_held(&ci->lock); + swap_range_free(si, cluster_offset(si, ci) + ci_start, nr_pages); + swap_cluster_assert_empty(ci, ci_start, nr_pages, false); + + if (!ci->count) + free_cluster(si, ci); + else + partial_free_cluster(si, ci); +} + +/* + * Free physical swap slots that were backing vswap entries (Pointer-tagge= d). + */ +static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi, + struct swap_cluster_info *pci, + unsigned int ci_start, + unsigned int nr_pages) +{ + unsigned int ci_off; + + spin_lock_nested(&pci->lock, SINGLE_DEPTH_NESTING); + VM_WARN_ON(pci->count < nr_pages); + pci->count -=3D nr_pages; + for (ci_off =3D ci_start; ci_off < ci_start + nr_pages; ci_off++) { + __swap_table_set(pci, ci_off, null_to_swp_tb()); + if (!SWAP_TABLE_HAS_ZEROFLAG) + __swap_table_clear_zero(pci, ci_off); + } + __swap_cluster_finish_free(psi, pci, ci_start, nr_pages); + swap_cluster_unlock(pci); +} + /* * Free a set of swap slots after their swap count dropped to zero, or wil= l be * zero after putting the last ref (saves one __swap_cluster_put_entry cal= l). @@ -2152,7 +2361,6 @@ void __swap_cluster_free_entries(struct swap_info_str= uct *si, unsigned long old_tb; unsigned short batch_id =3D 0, id_cur; unsigned int ci_off =3D ci_start, ci_end =3D ci_start + nr_pages; - unsigned long ci_head =3D cluster_offset(si, ci); unsigned int batch_off =3D ci_off; =20 VM_WARN_ON(ci->count < nr_pages); @@ -2190,13 +2398,7 @@ void __swap_cluster_free_entries(struct swap_info_st= ruct *si, if (batch_id) mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off); =20 - swap_range_free(si, ci_head + ci_start, nr_pages); - swap_cluster_assert_empty(ci, ci_start, nr_pages, false); - - if (!ci->count) - free_cluster(si, ci); - else - partial_free_cluster(si, ci); + __swap_cluster_finish_free(si, ci, ci_start, nr_pages); } =20 int __swap_count(swp_entry_t entry) @@ -2982,19 +3184,99 @@ static unsigned int find_next_to_unuse(struct swap_= info_struct *si, =20 static int try_to_unuse(unsigned int type) { + struct mempolicy *mpol =3D get_task_policy(current); struct mm_struct *prev_mm; struct mm_struct *mm; struct list_head *p; int retval =3D 0; struct swap_info_struct *si =3D swap_info[type]; struct folio *folio; - swp_entry_t entry; - unsigned int i; + struct swap_io_ctx ctx; + swp_entry_t entry, vswap_entry; + unsigned long swp_tb; + unsigned int i, j; =20 if (!swap_usage_in_pages(si)) goto success; =20 retry: + /* + * Free vswap-backing slots (Pointer-tagged) first. Walk physical + * clusters, read the vswap entry from the rmap, ensure the data + * is in the swap cache, and transition PHYS to FOLIO. Freeing the + * physical backing is enough, so no page table walk is needed. + */ + i =3D 0; + while (vswap_is_enabled() && + swap_usage_in_pages(si) && + !signal_pending(current) && + (i =3D find_next_to_unuse(si, i)) !=3D 0) { + swp_entry_t phys; + + swp_tb =3D swap_table_get(__swap_offset_to_cluster(si, i), + i % SWAPFILE_CLUSTER); + if (!swp_tb_is_pointer(swp_tb)) + continue; + + vswap_entry =3D swp_tb_ptr_to_swp_entry(swp_tb); + + folio =3D swap_cache_get_folio(vswap_entry); + if (!folio) { + folio =3D swap_cache_alloc_folio(vswap_entry, + GFP_HIGHUSER_MOVABLE, + BIT(0), NULL, mpol, + NO_INTERLEAVE_INDEX); + if (IS_ERR(folio)) { + if (PTR_ERR(folio) =3D=3D -ENOMEM) + return -ENOMEM; + continue; + } + ctx =3D (struct swap_io_ctx){}; + swap_read_folio(&ctx, folio); + swap_read_submit(&ctx); + folio_lock(folio); + } else { + folio_lock(folio); + } + + if (!folio_matches_swap_entry(folio, vswap_entry)) { + folio_unlock(folio); + folio_put(folio); + continue; + } + + /* + * Re-validate under folio lock: rmap holds folio->swap + j + * for some j in [0, nr_pages). Check folio->swap still maps + * to the contiguous physical run that includes our slot i. + */ + j =3D vswap_entry.val - folio->swap.val; + phys =3D vswap_to_phys(folio->swap); + if (!phys.val || swp_type(phys) !=3D type || + swp_offset(phys) + j !=3D i) { + folio_unlock(folio); + folio_put(folio); + continue; + } + + folio_wait_writeback(folio); + folio_release_vswap_backing(folio); + /* + * Drop a folio whose read failed rather than dirtying + * uninitialised memory; the next fault finds no backing and + * gets SIGBUS. + */ + if (unlikely(!folio_test_uptodate(folio))) + swap_cache_del_folio(folio); + else + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); + } + + if (!swap_usage_in_pages(si)) + goto success; + retval =3D shmem_unuse(type); if (retval) return retval; @@ -3037,6 +3319,8 @@ static int try_to_unuse(unsigned int type) (i =3D find_next_to_unuse(si, i)) !=3D 0) { =20 entry =3D swp_entry(type, i); + + /* Pointer-tagged rmap slots have no folio; the pre-pass took them. */ folio =3D swap_cache_get_folio(entry); if (!folio) continue; diff --git a/mm/vmscan.c b/mm/vmscan.c index 5cc40db37f6b..7960cc489ea0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1528,7 +1528,7 @@ static unsigned int shrink_folio_list(struct list_hea= d *folio_list, activate_locked: /* Not a candidate for swapping, so reclaim swap space. */ if (folio_test_swapcache(folio) && - ((mem_cgroup_swap_full(folio) && !is_vswap_entry(folio->swap)) || + ((mem_cgroup_swap_full(folio) && folio_phys_swap_backed(folio)) || folio_test_mlocked(folio))) folio_free_swap(folio); VM_BUG_ON_FOLIO(folio_test_active(folio), folio); diff --git a/mm/vswap.h b/mm/vswap.h index 5334c77b6b84..ce35a8381ca0 100644 --- a/mm/vswap.h +++ b/mm/vswap.h @@ -20,6 +20,7 @@ struct zswap_entry; enum vswap_backing_type { VSWAP_NONE =3D 0, VSWAP_ZSWAP =3D 1, + VSWAP_SWAPFILE =3D 2, VSWAP_ZERO, VSWAP_FOLIO, }; @@ -50,11 +51,15 @@ static inline bool is_vswap_entry(swp_entry_t entry) * pointer for a virtual swap slot. Tag in low 3 bits, payload in * upper 61 bits. * - * NONE: |----- 0000 ------|000| - no separate backend pointer - * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bi= ts) + * NONE: |----- 0000 ------|000| - no separate backend pointer + * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low = bits) + * SWAPFILE: |- type:5,off:56 -|010| - on a physical swapfile * - * Pointer payloads (ZSWAP) are stored directly with the tag OR'd into the - * low bits (kernel pointers are >=3D 8-byte aligned, same approach as xar= ray). + * SWAPFILE packs swp_type in the top MAX_SWAPFILES_SHIFT bits and swp_off= set in + * the middle VTABLE_PHYS_OFF_BITS bits, both above the tag, so the type is + * not shifted off the word. Pointer payloads (ZSWAP) are stored directly = with + * the tag OR'd into the low bits (kernel pointers are >=3D 8-byte aligned= , same + * approach as xarray). * * vtable[i] =3D NONE does not by itself mean "free". The swap_table entry * and the per-slot zero flag carry the rest of the state. The full @@ -68,6 +73,8 @@ static inline bool is_vswap_entry(swp_entry_t entry) * NONE | * | set | zero-backed; cached if PFN set * ZSWAP | PFN | clear | folio cached + zswap entry * ZSWAP | shadow / NULL | clear | evicted, only in zswap + * SWAPFILE | PFN | clear | folio cached + physical slot + * SWAPFILE | shadow / NULL | clear | evicted, only on the swapfile * * Locking: a slot's vtable entry (the vswap entry's backend) is only * stable while the caller owns and holds the lock on that entry's swap @@ -93,6 +100,23 @@ static inline enum vswap_backing_type vtable_type(unsig= ned long vt) return vt & VTABLE_TAG_MASK; } =20 +/* swp_offset field width in a physical backend slot; layout described abo= ve. */ +#define VTABLE_PHYS_OFF_BITS (BITS_PER_LONG - VTABLE_TAG_BITS - MAX_SWAPFI= LES_SHIFT) + +static inline unsigned long vtable_mk_phys(swp_entry_t entry) +{ + VM_WARN_ON_ONCE(swp_offset(entry) >> VTABLE_PHYS_OFF_BITS); + return ((unsigned long)swp_type(entry) << (VTABLE_TAG_BITS + VTABLE_PHYS_= OFF_BITS)) | + (swp_offset(entry) << VTABLE_TAG_BITS) | VSWAP_SWAPFILE; +} + +static inline swp_entry_t vtable_to_phys(unsigned long vt) +{ + VM_WARN_ON(vtable_type(vt) !=3D VSWAP_SWAPFILE); + return swp_entry(vt >> (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS), + (vt >> VTABLE_TAG_BITS) & ((1UL << VTABLE_PHYS_OFF_BITS) - 1)); +} + static inline struct zswap_entry *vtable_to_zswap(unsigned long vt) { VM_WARN_ON(vtable_type(vt) !=3D VSWAP_ZSWAP); @@ -134,6 +158,33 @@ vswap_lock_cluster(swp_entry_t entry, unsigned int *vo= ff) return container_of(ci, struct swap_cluster_info_dynamic, ci); } =20 +/** + * vswap_to_phys - resolve a vswap entry's physical swap backing + * @entry: the virtual swap entry + * + * Context: takes and drops the vswap cluster lock internally. + * Return: the backing physical swp_entry_t, or the null entry (.val =3D= =3D 0) + * when @entry has no physical backing (NONE/ZSWAP/ZERO). + */ +static inline swp_entry_t vswap_to_phys(swp_entry_t entry) +{ + struct swap_cluster_info_dynamic *ci_dyn; + unsigned int voff; + unsigned long vt; + + ci_dyn =3D vswap_lock_cluster(entry, &voff); + if (!ci_dyn) + return (swp_entry_t){}; + + vt =3D __vtable_get(ci_dyn, voff); + swap_cluster_unlock(&ci_dyn->ci); + + if (vtable_type(vt) !=3D VSWAP_SWAPFILE) + return (swp_entry_t){}; + + return vtable_to_phys(vt); +} + void __vswap_release_backing(struct swap_cluster_info *ci, unsigned int ci_start, unsigned int nr); =20 @@ -184,6 +235,104 @@ static inline struct zswap_entry *vswap_zswap_load(sw= p_entry_t entry) } =20 void folio_release_vswap_backing(struct folio *folio); +swp_entry_t folio_realloc_swap(struct folio *folio); +void folio_release_non_phys_swap_backing(struct folio *folio); + +/* + * Walk nr vtable slots starting at voff in ci_dyn. Returns the prefix + * length of slots sharing one effective backing type. For SWAPFILE, + * the prefix is also restricted to contiguous offsets in the same + * swapfile. + * + * Effective type per slot: + * vtable=3DNONE + zero flag set -> VSWAP_ZERO + * vtable=3DNONE + swap_table PFN tag -> VSWAP_FOLIO + * vtable=3DNONE + neither -> VSWAP_NONE + * vtable=3DSWAPFILE -> VSWAP_SWAPFILE + * vtable=3DZSWAP -> VSWAP_ZSWAP + * + * *typep returns the effective type of slot 0. Caller holds + * ci_dyn->ci.lock. + */ +static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *= ci_dyn, + unsigned int voff, int nr, + enum vswap_backing_type *typep) +{ + enum vswap_backing_type first_type =3D VSWAP_NONE; + enum vswap_backing_type slot_type; + swp_entry_t first_phys =3D {}; + unsigned long vt, swap_tb; + int i; + + lockdep_assert_held(&ci_dyn->ci.lock); + + for (i =3D 0; i < nr; i++) { + vt =3D __vtable_get(ci_dyn, voff + i); + if (vtable_type(vt) =3D=3D VSWAP_NONE) { + swap_tb =3D __swap_table_get(&ci_dyn->ci, voff + i); + if (__swap_table_test_zero(&ci_dyn->ci, voff + i)) + slot_type =3D VSWAP_ZERO; + else if (swp_tb_is_folio(swap_tb)) + slot_type =3D VSWAP_FOLIO; + else + slot_type =3D VSWAP_NONE; + } else { + slot_type =3D vtable_type(vt); + } + + if (!i) { + first_type =3D slot_type; + if (first_type =3D=3D VSWAP_SWAPFILE) + first_phys =3D vtable_to_phys(vt); + } else if (slot_type !=3D first_type) { + break; + } else if (first_type =3D=3D VSWAP_SWAPFILE && + vtable_to_phys(vt).val !=3D first_phys.val + i) { + break; + } + } + + if (typep) + *typep =3D first_type; + return i; +} + +static inline int vswap_check_backing(swp_entry_t entry, int nr, + enum vswap_backing_type *typep) +{ + struct swap_cluster_info_dynamic *ci_dyn; + unsigned int voff; + int ret; + + ci_dyn =3D vswap_lock_cluster(entry, &voff); + if (!ci_dyn) { + if (typep) + *typep =3D VSWAP_NONE; + return 0; + } + ret =3D __vswap_check_backing(ci_dyn, voff, nr, typep); + swap_cluster_unlock(&ci_dyn->ci); + return ret; +} + +/** + * folio_phys_swap_backed - test whether a folio is backed by a contiguous + * range of physical swap slots. + * @folio: a swap-cache resident folio + * + * Return: %true if @folio->swap is not a vswap entry, or if these vswap + * entries are backed by a contiguous range of physical slots. + */ +static inline bool folio_phys_swap_backed(struct folio *folio) +{ + swp_entry_t entry =3D folio->swap; + int nr =3D folio_nr_pages(folio); + enum vswap_backing_type type; + + return !is_vswap_entry(entry) || + (vswap_check_backing(entry, nr, &type) =3D=3D nr && + type =3D=3D VSWAP_SWAPFILE); +} =20 static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dyna= mic *ci_dyn, gfp_t gfp) @@ -214,6 +363,47 @@ static inline bool is_vswap_entry(swp_entry_t entry) return false; } =20 +static inline swp_entry_t vswap_to_phys(swp_entry_t entry) +{ + return (swp_entry_t){}; +} + +static inline bool folio_phys_swap_backed(struct folio *folio) +{ + return true; +} + #endif /* CONFIG_SWAP */ =20 +/* + * Test a per-backend swap flag (SWP_SYNCHRONOUS_IO, SWP_STABLE_WRITES, ..= .) + * for @entry. For a vswap entry the property belongs to the current + * physical backing rather than vswap_si itself; resolve to the backing + * and test there. Returns false for zswap/zero/unbacked vswap entries + * as they don't have a backing bdev. + */ +static inline bool swap_entry_backend_has_flag(struct swap_info_struct *si, + swp_entry_t entry, + unsigned long flag) +{ + struct swap_info_struct *phys_si; + swp_entry_t phys; + bool has_flag; + + if (!swap_is_vswap(si)) + return data_race(si->flags & flag); + + phys =3D vswap_to_phys(entry); + if (!phys.val) + return false; + + phys_si =3D get_swap_device(phys); + if (!phys_si) + return false; + + has_flag =3D data_race(phys_si->flags & flag); + put_swap_device(phys_si); + return has_flag; +} + #endif /* _MM_VSWAP_H */ diff --git a/mm/zswap.c b/mm/zswap.c index 54506935ceba..9a00ee049cf4 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1598,7 +1598,7 @@ bool zswap_store(struct folio *folio) */ if (is_vswap_entry(swp)) { if (index > 0) - folio_release_vswap_backing(folio); + folio_release_non_phys_swap_backing(folio); } else { unsigned type =3D swp_type(swp); pgoff_t offset =3D swp_offset(swp); --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oi1-f169.google.com (mail-oi1-f169.google.com [209.85.167.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C5EA486650 for ; Tue, 25 Aug 2026 15:32:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671978; cv=none; b=LbDllzUsJ+v80+9epMmQ4xZ4YNi7EeIQs2IKy1S/3cTZ3m2iwrrtroDeJi30ITxIHGyVba27pM/0mW2G4qE+haQOUX2Gq3gnNuwqjzkJG1msYDKm446EM5x9RzGQSt3cgj+tbPS0mxgwt050wDsDAMHM6Al3HUYquEkoLz1mf8o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671978; c=relaxed/simple; bh=Ng2X2En18m+Jw7WfQX3hDEj1QTM7CjxLX2o8m/jXjDs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IW4bOXOiK4j2zmAecN613GUKoKBa8zlB65mn0pnO5sbjV/ZXs8cEIM8lj7F8JGmQ3IsLNeJ+EP/HQ0IYe1zEoD15p8ctMAZq9TdKK07NViwP4OCo8dFa1SVh3YtQm4nDoXC1Qb503eGrYqPXqOfYjs3A0EY2hcmd0VxqD5zcINc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=njJBwHq5; arc=none smtp.client-ip=209.85.167.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="njJBwHq5" Received: by mail-oi1-f169.google.com with SMTP id 5614622812f47-4a483a552efso2622496b6e.1 for ; Tue, 25 Aug 2026 08:32:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671970; x=1788276770; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=efoGnqCiYbC3GQtG+3aDF+oz3AA7LAoLSzZXQ/arlkM=; b=njJBwHq55vl5xsBKNmbe7vPbr9FdmrN0dn31gyEqpiMrdw5saRqggzSEBJv3/i9O7M cwkaljQUKeFVkBZpWdwFa0SJte77QB8DrE6Uey0wJu+PQqlBCWjARqe14r6ECgtTXNSE GO81B9pL0WIXGicwSwMCRJOEgzhr4D3S1eNhQhD/qhlB/dlbjGZm/6paAmikDk+d/PIB ZcfuqHggJQfQjBphpV9J6NzpCjaJtA2hZ4nF6qo2TBhj6qQWe4lPSdBzIJs2G6aN/189 JpbAtVE7gHuejqDTrrRBvbMnzsOqm7HotEGo3n1hxLsD50f9tbHGmVt90H3mToFq4b0Q 5v4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671970; x=1788276770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=efoGnqCiYbC3GQtG+3aDF+oz3AA7LAoLSzZXQ/arlkM=; b=HYWyGikrv6dj/ddeL9Rayy4KrD7EqoYIjnJTJBY3vxKqUrtGKvg1kpfNCYxDnSdpi7 vO8pzmD0ukpx3lOXqbCa9qkezoIpMstds/T+ow6CFkiXOdUYIxb5hunClNhcHBd0bDkX RXEnQ3DI12Vc0ieLmeJmKeF3DKPOg6mF53I78BXPsscSG1UQ8SXyXQWTXxzznIgrUgA6 xfAKcWZCsjghYx+HLXam4+K1dB4EIRWbzDkLmNrF/FovXDvOVp0ievPwPiCGSxUU7ObK LmFBNg2SdpJBVFT3S3fbQTSIAkRBkAquNn80ymHPq3xWoPjaqtiGQVIUWHudose1sI5l Tz/g== X-Forwarded-Encrypted: i=1; AHgh+RpErw3gNYRGMnFLQ90JoH806QCyYB4EShIHRjJNSZCdzp9U+WnHuDlPfQgYTTaanpk9sDKqVb3j9hiWRVw=@vger.kernel.org X-Gm-Message-State: AFuF++llkJFgLhmJRVcvEYIOeMhFiwnEAdiju0GzB9kzO+cBu/V8Y7zb hAbG1d5VXdd6gPOkLFdI1sZlmp9c8zJWcKVEzBx4197/3TlAXd1oPyG0 X-Gm-Gg: AR+sD126C+z06rOG/ViIh9/OsFAlJ/hOsCW3OQ8z7erzEG8C3Son4i2qGi2JXy5/ZX0 08viAD9845Dxk85A3byizTvjrj/LC5RHwj5oToH9qXQWLHyC+b/fRoAipSDkRrlTld0RY+9yrcD 2HRLyuUgJqrTBHubUXhMPRdeOsNPRdyls7SodF80Foa5Pk4xMIkKfFnk9/APJCQ4mlrYSPvMSoW 2Mv3u+b6ueeQMcUTwaLHg3NV3bVuQ4AvpXa6O0K4Hv2l7/nm99dT4gvjl7XYkHfvV2wr0XjVnUU kUwlLH/4OaMTNjXagNpmXqccPfXJ5dAWzMq720COyYWnHryU1SYzK+S/x2OVYKZZZnb96AatCxK YKwOcxBCL8xkUEiAKVDM8HRUG2zd7q+QxUil2gRqa1FGr1oPsvYWmsCLZoW/2LdPbUkjjUq/ab8 rlvJx2ZntSWq9eml0p/PsGLbXBlP4PIsz7C/OIRaBwrcxIKqNXgSfOJhkJBM61EtiVhWB34hSD4 RZ4L5XK/L4Aad0dDpl5NQ== X-Received: by 2002:a4a:dd18:0:b0:6aa:de9e:e223 with SMTP id 006d021491bc7-6b16b209b13mr18223935eaf.9.1787671969879; Tue, 25 Aug 2026 08:32:49 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:5e::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-463831d1e61sm7536265fac.8.2026.08.25.08.32.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:49 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 05/11] mm, swap: enable THP swapin for vswap entries Date: Tue, 25 Aug 2026 08:32:31 -0700 Message-ID: <20260825153238.2695446-6-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Swap a large anon folio back in as a unit when its vswap entries share a contiguous run of physical swap slots on a synchronous IO device, instead of always falling back to order-0 faults. A zswap-backed or mixed-backing batch is still refused, and the fault retries at a smaller order. Signed-off-by: Nhat Pham --- mm/memory.c | 5 +++-- mm/swap_state.c | 17 +++++++++++++---- mm/zswap.c | 19 +++++++++++++------ 3 files changed, 29 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index dc4dd72ce73b..62f7b82427e2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4823,9 +4823,10 @@ static unsigned long thp_swapin_suitable_orders(stru= ct vm_fault *vmf) * lack handling for such cases, so fallback to swapping in order-0 * folio. * - * THP swapin for vswap is not supported yet either. + * Vswap entries are checked later, under the cluster lock in + * __swap_cache_add_check(). */ - if (is_vswap_entry(entry) || !zswap_never_enabled()) + if (!is_vswap_entry(entry) && !zswap_never_enabled()) return 0; =20 /* diff --git a/mm/swap_state.c b/mm/swap_state.c index c0441783b8e7..5cfddec8633b 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -174,6 +174,9 @@ static int __swap_cache_add_check(struct swap_cluster_i= nfo *ci, unsigned int ci_off, ci_end; unsigned long old_tb; bool is_zero; + struct swap_cluster_info_dynamic *ci_dyn; + enum vswap_backing_type type; + int ret; =20 lockdep_assert_held(&ci->lock); =20 @@ -202,11 +205,17 @@ static int __swap_cache_add_check(struct swap_cluster= _info *ci, return 0; =20 /* - * Reject a vswap batch so swap_cache_alloc_folio falls back to - * order 0. + * For a vswap entry batch, reject if the backing is not THP-amenable + * (e.g. uniformly ZSWAP, or mixed). The order-fallback loop in + * swap_cache_alloc_folio will retry with a smaller order on -EBUSY. */ - if (is_vswap_entry(targ_entry)) - return -EBUSY; + if (is_vswap_entry(targ_entry)) { + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + ret =3D __vswap_check_backing(ci_dyn, round_down(ci_off, nr), + nr, &type); + if (ret !=3D nr || type =3D=3D VSWAP_ZSWAP) + return -EBUSY; + } =20 is_zero =3D __swap_table_test_zero(ci, ci_off); ci_off =3D round_down(ci_off, nr); diff --git a/mm/zswap.c b/mm/zswap.c index 9a00ee049cf4..f16c0b44b5d5 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1629,9 +1629,9 @@ bool zswap_store(struct folio *folio) * will SIGBUS). * * -EINVAL: if the swapped out content was in zswap, but the page belongs - * to a large folio, which is not supported by zswap. The folio is unlock= ed, - * but NOT marked up-to-date, so that an IO error is emitted (e.g. - * do_swap_page() will SIGBUS). + * to a large non-vswap folio, which is not supported by zswap. The folio + * is unlocked, but NOT marked up-to-date, so that an IO error is emitted + * (e.g. do_swap_page() will SIGBUS). * * -ENOENT: if the swapped out content was not in zswap. The folio remains * locked on return. @@ -1652,10 +1652,17 @@ int zswap_load(struct folio *folio) * Large folios should not be swapped in while zswap is being used, as * they are not properly handled. Zswap does not properly load large * folios, and a large folio may only be partially in zswap. + * + * A large vswap folio cannot reach here ZSWAP-backed, since + * __swap_cache_add_check() refuses such a batch, so hand it to the + * phys path without warning. */ - if (WARN_ON_ONCE(folio_test_large(folio))) { - folio_unlock(folio); - return -EINVAL; + if (folio_test_large(folio)) { + if (WARN_ON_ONCE(!swap_is_vswap(si))) { + folio_unlock(folio); + return -EINVAL; + } + return -ENOENT; } =20 entry =3D zswap_entry_load(swp); --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A922E486E78 for ; Tue, 25 Aug 2026 15:32:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671976; cv=none; b=Afbiqo0TpdqWgocM2kjpYeUMRXbte4fgEvWdEfEJ75YZ+WTWLD2e3vWqPa8kjglIWwbOV48XhpJrbixrT61VKNT0N6Bx88mdrKoOAPZ411B0iE0JxttStsWLpaRkXKqcBxS3mMpFhb/8FA5+aYJGUdcHPL0Y3jWyn4Krede2so8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671976; c=relaxed/simple; bh=dzGLx7hmRStu+44RUZgGTXgypJylAAmdzWxuKyL5v4U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NpVLn9oSo3AW0r+jXy92ztYCu0u92OM7j1ZkVwonPmAeQ58X4dof0uEtjexCh+NzS/IXgMSSY/UAqTCHaEJs9vtdPyRlrt7zXxuxRS14S85cQAac/zwA9VRwtFbXu/sYWPYCHcJZjGoLKeXA1KYVUmEGzMhRqR5T1GuWFYdwo00= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=V5f9hGjb; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="V5f9hGjb" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-7e9ecd7216cso3782574a34.3 for ; Tue, 25 Aug 2026 08:32:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671971; x=1788276771; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=1juMVOhJOgcK/kg6XX4IK6P14Ov4AD2HCrXjeUjkxQU=; b=V5f9hGjbPzOwoM5i8m1SfnkswVKQj7oUtFUKPpLoQN5qlTiBJcuyhGbxyMf/unnn8j X0Bvazpe/PLXAcIlstjDndiUMdTt4FuCcuJFEJ4uSsBx1BbmRUwRqBW3ERs7h04X8tBm jVtR8iVlJSMRfJNaA2PjpuAT3KVLnNM1i93nWsWjBi8cU1FM+72xiUmBgOtCd7ZkHsfo 0HQv8hbxr1uCzSGlj9suqem4c2EUzA5v0WRA3f1GOW+LdJSUbzwLGr3BSXz6vUd1m5p5 Jz8AsEu/eOz/u2eb5dqukWVKreV18VBKx4k4zRyRWIYVh5Qra8Fy2LRzzFX5r00eoFUc 4L1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671971; x=1788276771; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=1juMVOhJOgcK/kg6XX4IK6P14Ov4AD2HCrXjeUjkxQU=; b=gsXnCxSkzzugs8b2iqwauKKmiNMXn5YFzn5rE7Jk6F+p+b14waLMmnVOSTpCeJKuJa AxxKF261oTqpmWEoRz6KIcZe7atxuZC0woTGgU3z6I9YCupGmv+3BfQyBkmBZ5Xd1gPF XWPgF5+by2r1Juy8ZoHTxd01j23TYerpRMnUiNR58J4oHrEWpDOJIm5MCfNLmNIm1Cqr KPaZuwSpvr77mjfSiItzn+A6/vRvIRt+BjQfLh7W9o5Fb9X0kWmqhrRy+CarLQ89nFgD R/StuObrEahq1CBMnY+IhIlsEBsQ4CpQFILuaMAuIOdDBDUavEUWefEN0CoZoTBm6KJ0 wxcA== X-Forwarded-Encrypted: i=1; AHgh+RqOPFyrjdDN2bsJw/BDApUe9CBa8FSOyw09uglWxkjSabQvyadz6N9e8EwkRzvh9jDoeT+zi1KHtmU5wCE=@vger.kernel.org X-Gm-Message-State: AFuF++kpnnsdPG+1GJW2yy7vLNvlem+ysK1d/C7y0z2xn+VuSItMoKy6 o3X5j4Fs7J0fUrPG3lA7HeTzXScc47taglaRzqObr5J5cNw1CjFjWcgW X-Gm-Gg: AR+sD12PpcwT2qjoEHdO886eNHhlqFmS+tPIyPdJuHI47dC0hfn7J7UtU+ZTIKQ6na+ aqzqSFKQBsnOLacLagYa8WQORFRnxehaAUdCphRy0mQVDQshwpwF0jLqcc+91yCGJD1MAiDsjjS B1u9KmsNFNjst7Yl14kqL2FKKbXx64Utdj21IeoNu87DxUmKkxSZK/WBy8xiU9Sh5UnKvW+3DSE 4ZXO1wT7qBD4ZyrjxJwDpBEynAc3xeGbTmX9/95spgMkASnPFoFvoXmusKeG3zYT1FOfkHc7jBJ ABGqRVAgqi+sGHU/Id6+Gf3PaAq6rVyjA2Ey4JEVd9voofcNevG+RlBVpjje5SMIy8e4fnGAtCo hlktc5eIektKCjcFSax+cBbha3tv6ND8F7oTe1tEEz9J6X/DtV3WxxMsQAfKF97qfBfQUWK465+ Qbm/OXIunMae5IoFkvSyd5+3hW4cxlFUf+SX5l9Fu0I9i6ziDzWdyAHfpmmHnQM451r10FSpEZu bM5envx4tM= X-Received: by 2002:a05:6830:448f:b0:7eb:3af8:8c1a with SMTP id 46e09a7af769-7f4af99833dmr6541276a34.9.1787671971139; Tue, 25 Aug 2026 08:32:51 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:70::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7f48fc4f427sm6969227a34.16.2026.08.25.08.32.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:50 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 06/11] mm, swap: write back vswap zswap entries to physical swap Date: Tue, 25 Aug 2026 08:32:32 -0700 Message-ID: <20260825153238.2695446-7-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add support for writing back zswap-backed vswap entries to physical swap. The mechanism mirrors the existing zswap writeback path, except the backing physical slot is allocated on demand at writeback time rather than already being pinned by the PTE. The zswap shrinker no longer skips vswap entries, unless we are out of physical swap space. Signed-off-by: Nhat Pham --- mm/zswap.c | 67 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 21 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f16c0b44b5d5..70ad8010f18a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1007,12 +1007,13 @@ static bool zswap_decompress(struct zswap_entry *en= try, struct folio *folio) static int zswap_writeback_entry(struct zswap_entry *entry, swp_entry_t swpentry) { - struct xarray *tree; pgoff_t offset =3D swp_offset(swpentry); struct folio *folio; struct mempolicy *mpol; struct swap_info_struct *si; struct swap_io_ctx ctx =3D {}; + swp_entry_t phys =3D {}; + bool is_vswap; int ret =3D 0; =20 /* try to allocate swap cache folio */ @@ -1020,12 +1021,7 @@ static int zswap_writeback_entry(struct zswap_entry = *entry, if (!si) return -EEXIST; =20 - /* Vswap entries have no physical backing to write to. */ - if (swap_is_vswap(si)) { - put_swap_device(si); - return -EINVAL; - } - + is_vswap =3D swap_is_vswap(si); mpol =3D get_task_policy(current); folio =3D swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol, NO_INTERLEAVE_INDEX); @@ -1044,24 +1040,44 @@ static int zswap_writeback_entry(struct zswap_entry= *entry, /* * folio is locked, and the swapcache is now secured against * concurrent swapping to and from the slot, and concurrent - * swapoff so we can safely dereference the zswap tree here. + * swapoff so we can safely dereference the zswap tree (or vswap + * vtable) here. * Verify that the swap entry hasn't been invalidated and recycled * behind our backs, to avoid overwriting a new swap folio with * old compressed data. Only when this is successful can the entry * be dereferenced. */ - tree =3D swap_zswap_tree(swpentry); - if (entry !=3D xa_load(tree, offset)) { + if (entry !=3D zswap_entry_load(swpentry)) { ret =3D -ENOMEM; goto out; } =20 + if (is_vswap) { + /* + * Allocate physical backing before decompress so a failure + * wastes no work. + */ + phys =3D folio_realloc_swap(folio); + if (!phys.val) { + ret =3D -ENOMEM; + goto out; + } + } + if (!zswap_decompress(entry, folio)) { ret =3D -EIO; + /* + * The phys allocation above took the entry out of the vtable. + * Restore the zswap entry to the vtable, which also frees the + * allocated physical swap space. + */ + if (is_vswap) + vswap_zswap_store(swpentry, entry); goto out; } =20 - xa_erase(tree, offset); + if (!is_vswap) + xa_erase(swap_zswap_tree(swpentry), offset); =20 count_vm_event(ZSWPWB); if (entry->objcg) @@ -1076,7 +1092,10 @@ static int zswap_writeback_entry(struct zswap_entry = *entry, folio_set_reclaim(folio); =20 /* start writeback */ - __swap_writepage(&ctx, folio, folio->swap); + if (is_vswap) + __swap_writepage(&ctx, folio, phys); + else + __swap_writepage(&ctx, folio, folio->swap); swap_write_submit(&ctx); =20 out: @@ -1091,6 +1110,15 @@ static int zswap_writeback_entry(struct zswap_entry = *entry, /********************************* * shrinker functions **********************************/ +/* + * vswap zswap entries get a physical slot allocated on demand at writeback + * time. Skip the shrinker when none is available. + */ +static bool zswap_writeback_possible(void) +{ + return !vswap_is_enabled() || get_nr_swap_pages() > 0; +} + /* * The dynamic shrinker is modulated by the following factors: * @@ -1228,7 +1256,7 @@ static unsigned long zswap_shrinker_count(struct shri= nker *shrinker, if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg)) return 0; =20 - if (vswap_is_enabled()) + if (!zswap_writeback_possible()) return 0; =20 /* @@ -1314,7 +1342,8 @@ static struct shrinker *zswap_alloc_shrinker(void) * were scanned but none could be written back, or -ENOENT if @memcg has * writeback disabled, is a zombie cgroup, or has empty zswap LRUs. * - * Also returns -ENOENT when vswap is enabled. + * Also returns -ENOENT when vswap is enabled and there is no physical + * swap to write back to. */ static int shrink_memcg(struct mem_cgroup *memcg) { @@ -1323,7 +1352,7 @@ static int shrink_memcg(struct mem_cgroup *memcg) if (!mem_cgroup_zswap_writeback_enabled(memcg)) return -ENOENT; =20 - if (vswap_is_enabled()) + if (!zswap_writeback_possible()) return -ENOENT; =20 /* @@ -1354,11 +1383,7 @@ static void shrink_worker(struct work_struct *w) int ret, failures =3D 0, attempts =3D 0; unsigned long thr; =20 - /* - * When vswap is enabled, zswap entries are almost all vswap backed, - * with no slot to write back to. - */ - if (vswap_is_enabled()) + if (!zswap_writeback_possible()) return; =20 /* Reclaim down to the accept threshold */ @@ -1439,7 +1464,7 @@ static void shrink_worker(struct work_struct *w) break; resched: cond_resched(); - } while (zswap_total_pages() > thr); + } while (zswap_total_pages() > thr && zswap_writeback_possible()); } =20 /********************************* --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AC2D48A8B6 for ; Tue, 25 Aug 2026 15:32:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671984; cv=none; b=aE2h+5CPZJQo2iefrKnV2f9aXDIShOoz2OvYycPLezTYotdiTrdeSF/ZG64W5SaaaR0+yl3LRzbN1SAEqAbcRcfmhUsc87HnamB9hik/EhnePYbVGD6jMJu/jBAD1t4I6EBthdZj16GveSwdnlwD5qHVNkhpkXuBgcl0gLMQpu4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671984; c=relaxed/simple; bh=4B5CVeeW+H8xhmPHjOyG0zUr0e2Whn+FP3FajcUNunw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X2WSsIAKRWjg1GqT8MIwd7Clmh8rLtd5y2+/BRNtdkmuM7JGXyLyb1QnZncWont1ALZfOgFuTXvmekdfKyiyoiBnU97NoxTQyI3i/kvbE4FVcIEnX25e1ONeQfavkJxsaLVeVhuxF83yN9VVusLDmpH6crA73Qvt12X4FMeHLEs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=c2KnE9Es; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c2KnE9Es" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-7eb4d532e65so2765215a34.0 for ; Tue, 25 Aug 2026 08:32:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671973; x=1788276773; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8+Xifm4yMHmmJc4W3PmSLP73v1OoKmGISMUVeCn1vQA=; b=c2KnE9Esmn4fhxz7kJ7PPEBvEyDgOv63Q9a7ULtXLhMypQrRUUUz7AableiL9FwXGt DGyuT8yIKvyGO3PcP2xtHSEsKux0XNCJKKPlIm1RDGOKpfJLWH8cAdDRKf2QJ3EdrRcM svF4NORS84xqmhG35MVQVafsYXshK/Wnh+nwnoFmCRVr17Pb0XmxW2uMLpSiLCsIB79F MhZ4ZAGWhwpouSR2JW0e+uE4ti48tg8xVYGi7Qx4M6qQHMcaSjzn9CdEbWdXGU3oA89u VtaQkdx+el1xpWu9MrUB0/3WiU+2IyTfXNwfZW5lXjT2xPW8My/S8vefqgtxDADklhS8 dXzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671973; x=1788276773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=8+Xifm4yMHmmJc4W3PmSLP73v1OoKmGISMUVeCn1vQA=; b=WW9YGCkiMzEiVd6QAqAbUzsCcrriQvH3PYHuc8Tf3/XYjt/MnXELTocAfj/jH9HPvJ lCs+zV+erRsQ2R/dACcvzd1ZzoCpraHSGq3UcTmv8xzCQV+KK1SRbE2ugwdSq5GFLkQd d5eK0NoxEm0a4frs6VJpry5xpQDPjX7J+JHTf/qBos/xGZWmXsdsKhrXFmIbHCAQVVGs JXxuWoGdlngrrYs1ObIzGyh7bSSyEqzHujRsklTDIsJ4Ks3oyGeEPfYe26mHdP45N+B8 RTshBh0L0RPgvLTwvR8dMZ6WTtvPZ3ARsUXK/NTzc8wipi0ixDkYPjhx3wipqNUKd7Mg 4/Uw== X-Forwarded-Encrypted: i=1; AHgh+RpmrsXef0BR+CDQpjhJtTh0SjmCvOev5NKWQ1lsYbuONnyzpvo78j+kBYLj2kUR7AF8i9CbwRe8KqHyFGg=@vger.kernel.org X-Gm-Message-State: AFuF++m3IybzluljlTa4r3pSLlCkD4D/4s9/1QHym3IZQ6xEct614kyF MkUkOYxfbfUFxxMrhy7lNWQ7/3szEZNYyCKinzoO1HTp5+5QZp9dk26O X-Gm-Gg: AR+sD101E2qRpVWN7B7jjVf0X1qEA+/a4IJ7fxn0rAyly7wiN84oU0TwhYOmIIiqZZf r0nzctMGr9u58UyPgM7Kb4MGuL0cx82AP/4PvHZbi4/4L7fjDUM5kLZK2OZkyfzSHsaeUc8FlFF JP7L+MjAvGUoJOqaQI8qCpJpPZZwsI4tIuNBhlWPAKuzFCRVkd0eWgUnxAhQV9gO0sIT7CbfpS5 bA1R1CDshGyb4c5ZLR77wX/cGlr6WSlF40hZ0hFWV0uLiI5gfpuzRFBrxCv7kPxAho15h/z9blc ++CaX4IFhuK/upL7MUuVxWJxgajqeCOgzTZ/LxH+fLFp3KvdCxUjk9c9rs/jAfSnoJ+CRLE00vO HPuU70R0P7+AfZ0sodGIvddd/KcI72IELo+K9sO3wBO+Kzv5EFK8EmUSloTT3Ve2ScafI0iUOXW TKPByfQT6Rkk9FF812TYUfUkd0btUmm15yW0vuEIfKnAm3t61xiv1CBojImj7G1Q017WqMTSKRC 2qC/UKe5ActPOkWyDK1ruU= X-Received: by 2002:a05:6830:6af8:b0:7e9:e313:4fdb with SMTP id 46e09a7af769-7f47639d520mr29228026a34.2.1787671972520; Tue, 25 Aug 2026 08:32:52 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:42::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7f48fb1db63sm8021692a34.12.2026.08.25.08.32.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:52 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries Date: Tue, 25 Aug 2026 08:32:33 -0700 Message-ID: <20260825153238.2695446-8-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When a vswap entry's swap_count drops to 0 while its folio is still in the swap cache, the entry is cache-only and its physical slot is redundant. Until now such a slot was only freed when the vswap entry itself was freed, pinning otherwise reclaimable physical capacity. Reclaim such slots from the physical reclaim scanner, once swap is more than half used (vm_swap_full()), to free physical capacity for new allocations. Signed-off-by: Nhat Pham --- mm/swap_table.h | 12 +++-- mm/swapfile.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++++ mm/vswap.h | 31 +++++++++++++ 3 files changed, 154 insertions(+), 4 deletions(-) diff --git a/mm/swap_table.h b/mm/swap_table.h index 08d2494a8ee6..b614b1989fd9 100644 --- a/mm/swap_table.h +++ b/mm/swap_table.h @@ -31,7 +31,7 @@ struct swap_memcg_table { * NULL: |---------------- 0 ---------------| - Free slot * Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot * PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot - * Pointer: |-------- vswap offset --------|100| - vswap rmap + * Pointer: |C|------- vswap offset -------|100| - vswap rmap * Bad: |------------- 1 -------------|1000| - Bad slot * * COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit, @@ -376,14 +376,18 @@ static inline unsigned short __swap_cgroup_clear(stru= ct swap_cluster_info *ci, * On physical clusters, a Pointer-tagged entry stores the offset of the * vswap entry that owns this physical slot (the reverse map). Only the * offset is stored; the swap type is implicit (always vswap_si->type, - * since there is exactly one vswap device). + * since there is exactly one vswap device). The top bit is reserved as + * a cache-only flag, set when vswap swap_count drops to 0 but the folio + * is still in swap cache. * - * Pointer: |---- vswap offset ----|100| + * Pointer: |C|---- vswap offset ----|100| + * C =3D SWP_RMAP_CACHE_ONLY (bit 63) */ #define SWP_TB_PTR_MARK_BITS 3 #define SWP_TB_PTR_MARK 0b100UL #define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1) -#define SWP_RMAP_ENTRY_MASK (~SWP_TB_PTR_MARK_MASK) +#define SWP_RMAP_CACHE_ONLY (1UL << (BITS_PER_LONG - 1)) +#define SWP_RMAP_ENTRY_MASK (~(SWP_RMAP_CACHE_ONLY | SWP_TB_PTR_MARK_MASK)) =20 static inline bool swp_tb_is_pointer(unsigned long swp_tb) { diff --git a/mm/swapfile.c b/mm/swapfile.c index 66bcbb112142..6ec439462490 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -142,6 +142,10 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, per= cpu_vswap_cluster) =3D { }; =20 static bool vswap_alloc(struct folio *folio); +static void vswap_mark_cache_only(struct swap_cluster_info *ci, + unsigned int ci_off); +static void vswap_clear_cache_only(struct swap_cluster_info *ci, + unsigned int ci_start, int nr); =20 /* May return NULL on invalid type, caller must check for NULL return */ static struct swap_info_struct *swap_type_to_info(int type) @@ -867,6 +871,54 @@ static int swap_cluster_setup_bad_slot(struct swap_inf= o_struct *si, return ret; } =20 +/* + * Try to reclaim a Pointer-tagged physical slot backing a vswap entry. + * The physical cluster lock must NOT be held. Returns the backing folio's + * page count, negated if the slots could not be reclaimed, or 0 if the + * folio could not be shown to own @offset (i.e. there is a race). + */ +static int try_to_reclaim_vswap_backing(struct swap_info_struct *si, + unsigned long offset, + swp_entry_t vswap_entry) +{ + swp_entry_t phys_base; + struct folio *folio; + unsigned int i; + int ret; + + folio =3D swap_cache_get_folio(vswap_entry); + if (!folio) + return 0; + + if (!folio_trylock(folio)) { + folio_put(folio); + return 0; + } + + if (!folio_matches_swap_entry(folio, vswap_entry)) { + folio_unlock(folio); + folio_put(folio); + return 0; + } + + i =3D vswap_entry.val - folio->swap.val; + phys_base =3D vswap_to_phys(folio->swap); + if (!phys_base.val || swp_type(phys_base) !=3D si->type || + swp_offset(phys_base) + i !=3D offset) { + folio_unlock(folio); + folio_put(folio); + return 0; + } + + /* The run is ours: skip it all, whether or not the free succeeds. */ + ret =3D folio_nr_pages(folio); + if (!folio_free_swap(folio)) + ret =3D -ret; + folio_unlock(folio); + folio_put(folio); + return ret; +} + /* * Reclaim drops the ci lock, so the cluster may become unusable (freed or * stolen by a lower order). @usable will be set to false if that happens. @@ -1148,6 +1200,7 @@ static void swap_reclaim_full_clusters(struct swap_in= fo_struct *si, bool force) long to_scan =3D 1; unsigned long offset, end; struct swap_cluster_info *ci; + swp_entry_t vswap_entry; unsigned long swp_tb; int nr_reclaim; =20 @@ -1172,6 +1225,19 @@ static void swap_reclaim_full_clusters(struct swap_i= nfo_struct *si, bool force) offset +=3D abs(nr_reclaim); continue; } + } else if (swp_tb_is_pointer(swp_tb) && + (swp_tb & SWP_RMAP_CACHE_ONLY)) { + vswap_entry =3D swp_tb_ptr_to_swp_entry(swp_tb); + spin_unlock(&ci->lock); + nr_reclaim =3D try_to_reclaim_vswap_backing(si, offset, + vswap_entry); + ci =3D swap_cluster_lock(si, offset); + if (!ci) + goto next; + if (nr_reclaim) { + offset +=3D abs(nr_reclaim); + continue; + } } offset++; } @@ -1748,6 +1814,8 @@ static void swap_put_entries_cluster(struct swap_info= _struct *si, } /* count will be 0 after put, slot can be reclaimed */ need_reclaim =3D true; + if (swap_is_vswap(si)) + vswap_mark_cache_only(ci, ci_off); } /* * A count !=3D 1 or cached slot can't be freed. Put its swap @@ -1854,6 +1922,8 @@ static int swap_dup_entries_cluster(struct swap_info_= struct *si, goto failed; } } while (++ci_off < ci_end); + if (swap_is_vswap(si)) + vswap_clear_cache_only(ci, ci_start, nr); swap_cluster_unlock(ci); return 0; failed: @@ -1981,6 +2051,51 @@ int folio_alloc_swap(struct folio *folio) return 0; } =20 +static void vswap_mark_cache_only(struct swap_cluster_info *ci, + unsigned int ci_off) +{ + struct swap_cluster_info_dynamic *ci_dyn; + struct swap_cluster_info *pci; + swp_entry_t phys; + unsigned long vt; + + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + vt =3D __vtable_get(ci_dyn, ci_off); + + if (vtable_type(vt) =3D=3D VSWAP_SWAPFILE) { + phys =3D vtable_to_phys(vt); + pci =3D __swap_entry_to_cluster(phys); + swap_rmap_mark_cache_only(pci, swp_cluster_offset(phys)); + } +} + +/* + * Clear the cache-only rmap hint for entries re-referenced from count 0 t= o 1 + * (no longer reclaimable), so the physical reclaim scanner skips them. + */ +static void vswap_clear_cache_only(struct swap_cluster_info *ci, + unsigned int ci_start, int nr) +{ + struct swap_cluster_info_dynamic *ci_dyn; + struct swap_cluster_info *pci; + unsigned long swp_tb, vt; + swp_entry_t phys; + unsigned int off; + + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + for (off =3D ci_start; off < ci_start + nr; off++) { + swp_tb =3D __swap_table_get(ci, off); + if (!swp_tb_is_folio(swp_tb) || swp_tb_get_count(swp_tb) !=3D 1) + continue; + vt =3D __vtable_get(ci_dyn, off); + if (vtable_type(vt) !=3D VSWAP_SWAPFILE) + continue; + phys =3D vtable_to_phys(vt); + pci =3D __swap_entry_to_cluster(phys); + swap_rmap_clear_cache_only(pci, swp_cluster_offset(phys)); + } +} + static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi, struct swap_cluster_info *pci, unsigned int ci_start, diff --git a/mm/vswap.h b/mm/vswap.h index ce35a8381ca0..c66fa34e2e60 100644 --- a/mm/vswap.h +++ b/mm/vswap.h @@ -44,6 +44,37 @@ static inline bool is_vswap_entry(swp_entry_t entry) return swap_is_vswap(__swap_entry_to_info(entry)); } =20 +/* + * Rmap cache-only helpers for physical cluster Pointer-tagged entries. + * SWP_RMAP_CACHE_ONLY records, inline on the physical swap_table entry, + * that the backing vswap entry has swap_count =3D=3D 0 (swap-cache-only, = so + * reclaimable). The physical reclaim scanner reads it directly instead of + * chasing the rmap into the vswap layer and paying the cluster-lookup + * indirection. + * + * Callers hold the vswap cluster lock, not the physical one. The rmap is + * only touched while the vtable holds the slot as SWAPFILE, and that + * window is opened and closed under the vswap cluster lock, so the + * allocator has finished writing the entry by then. + */ +static inline void swap_rmap_mark_cache_only(struct swap_cluster_info *ci, + unsigned int off) +{ + atomic_long_t *table; + + table =3D rcu_dereference_check(ci->table, true); + atomic_long_or(SWP_RMAP_CACHE_ONLY, &table[off]); +} + +static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci, + unsigned int off) +{ + atomic_long_t *table; + + table =3D rcu_dereference_check(ci->table, true); + atomic_long_and(~SWP_RMAP_CACHE_ONLY, &table[off]); +} + /* * Virtual table entry encoding for vswap clusters. * --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oa1-f41.google.com (mail-oa1-f41.google.com [209.85.160.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7E4A488777 for ; Tue, 25 Aug 2026 15:32:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671983; cv=none; b=bFwo0zF78PsYNZtUKO6LfARsa1NpS/Q3gzU92mzMe64DTyDxWFTXPUf3YA5B8SFuWpZMiQ6K9EiRUF229dtqLoPgnv9LZBiTA1JHs6GiXdr520xd1a9gYWq5RwEonYG7OGH3w7GVUszQRW6MqQ3F59PK5HnfjNcIfbNmiEOc8vA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671983; c=relaxed/simple; bh=OBXKrqbaXwvmCx1KQK9eEvnh7PwXgK4ThqqYLp1zz5g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CkUqEL+SyRju+Nx24218hUWUGnuQRAFIVrOHJ23R8nqMcGhPplISNAfP1A3yQuziJKquoz6hwrYpPHDKqja1tLcLBqiC9W725K3r0GZlT8MNSrsrwhOPVWH0VtVrLm7D0rjwFOPw+sLkTUd5/4jV+ii5D121UuAjO/+hhH2YhmM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XVnVECjM; arc=none smtp.client-ip=209.85.160.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XVnVECjM" Received: by mail-oa1-f41.google.com with SMTP id 586e51a60fabf-43bf9548df4so4930001fac.0 for ; Tue, 25 Aug 2026 08:32:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671974; x=1788276774; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=zlGXhprJFuKNO99yNrTPz5qKVmgEA/flDFbSYzcy7Gk=; b=XVnVECjMhN4SbdauoVI+j9MRsd9y2Rep+95kjF/jWSPF0M+WK47XdBviZoPHaIcZiP dT+w83p0RqR8tcRn0npsn75xkor19gWKlqGnCjIUHRq3V0RK6zZ3PY1UTfrXNrZfNSYE BAuAABHx3Y9uoMk7q7J5KuXgBn4MsQmncCa1KBt1EXKty6mpxq1yLPT3MYWIO2X9w0Gq pW6MfEW9gA9KExyg7bRj+mzfy1upIbPbThZnidcKxT/cze+ua54bmVmlppxbGEVHSOQn ZMk7sck8nHdrBnbCyhZumf/8prKDrA+VPidUCkHH2N4N/msn7aoAYU+3fDuB2d+zH3EJ o+EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671974; x=1788276774; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=zlGXhprJFuKNO99yNrTPz5qKVmgEA/flDFbSYzcy7Gk=; b=iNFSsLVvk9WuoG6gSA20uLfrIoD2IFkPkH+j2tXmg8xc0v7OUME+6z9SWx8CvOkjpd cTG696nuITR0nWUIkopr2O1fHzG0RfsEcZKeVcbYG0ONmX/qHOGW917L3F38ZlJO7HJg xEIa00cMZW18Y2wOfjcBEiasxZEWtAUiYK3LsmIEXrhtZT2ql+jGUMZXUpC23gsr/2og gxUlRDzrDxAmt2VyYFu+hoq2I6xgUwVljAR1P/4enT5bKbfSI+eQ1ZcuKRniP9V2Saxa uuCZPuyuL8zTvRcIy7/odkfR8eeGaZdnF8HxPsvEhXzNL0f3If4OB+PO693wrJY0vkuk sP2Q== X-Forwarded-Encrypted: i=1; AHgh+RoON5RAOek5mInPuSrItHHJoVDns+NfkAJYZk0Ex1Fa6k7e8LIYKmItYL3rIh0PosEbJJxMJ1bUZ9L1Hn0=@vger.kernel.org X-Gm-Message-State: AFuF++nW2NupQrQ+Vh11Z2a0PItMXsFPFgQZ8p3gZJ01c4JVOWnUik9A RzSQ0EHlAvmQlx3hkhGUlroiffm3pNcBocADaqcw6TMEjccbrtV+MQF0 X-Gm-Gg: AR+sD11w1sI7y484pyqRwBKZAhLpE57uTvBa/4iglDfqyyql/O1Ga/N69JMbXBMLBC0 G/S226F8HP7ClDpgB+5Sul5jOaF98jW3xp8NgWbDMGGUSBAgJ62cL/HzjarxbcqioOMqMvdz79E U7/qdC8ReL8LKCduGdbwtTObj5k9MXTn9sXh17VXPZtCIBCiPxqDtdG5EZ2sS3K2PJ/vFa7PzaD XvOwA3MaT7YG6NZROoQFjuKRH652l08bRDFrOAVzFKTeOl5NdXQcqx+LMIe6az2/MnFDDx9fvuU 4zvJMkUUhJ15lQcbIhx0LSWwjgXOqPxn8SZEz1GNvMqm+Ygqi7txhEYxFzYctxiYQOKYNYqZTxh Y6f/5MScAOy2F9HvavFWUuVGf/ZmGlGRNc8Cyc+Cfa3rQaemfgL4Qrk3RoUg4rTjnTAj7VOkeVZ LDTBpjXFevjIAjNrgXdVu11c6R2MU076DxAoWA4esVk11CKcHeWZi5cGXCpTCGhL7gTl8SC6adx 9fPi7ujvC8= X-Received: by 2002:a05:6820:5507:b0:6ac:8e23:3078 with SMTP id 006d021491bc7-6b1591d99b6mr26775658eaf.5.1787671973954; Tue, 25 Aug 2026 08:32:53 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:52::]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-6b17c8328c2sm6742797eaf.4.2026.08.25.08.32.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:53 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 08/11] mm, swap: only charge physical swap entries Date: Tue, 25 Aug 2026 08:32:34 -0700 Message-ID: <20260825153238.2695446-9-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Zswap-backed and zero-filled pages occupy no swap space, but were charged against memcg->swap as though they did. Charge memcg->swap when a vswap entry acquires physical backing rather than when it is allocated. memory.swap.current therefore counts only on-disk swap usage, not zswap-backed or zero-filled pages. When vswap is enabled, a cgroup can reclaim its anon memory even with memory.swap.max set to 0, provided zswap is allowed for it. Also refactor the swap memcg operations into separate get, record, charge, uncharge and put helpers, since recording the owner and charging it no longer happen at the same time. Direct-mapped physical swap charging is unchanged. So is cgroup v1 memsw accounting: the folio's memsw charge is retained across swapout regardless of backing, and released when the entry is freed. Suggested-by: Johannes Weiner Signed-off-by: Nhat Pham --- .../admin-guide/cgroup-v1/memcg_test.rst | 2 +- include/linux/memcontrol.h | 6 + include/linux/swap.h | 61 +++++++- mm/memcontrol-v1.c | 10 +- mm/memcontrol.c | 148 +++++++++++------- mm/swapfile.c | 128 +++++++++++++-- 6 files changed, 279 insertions(+), 76 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v1/memcg_test.rst b/Documenta= tion/admin-guide/cgroup-v1/memcg_test.rst index ebedbc3c3f9c..13b9ae800b72 100644 --- a/Documentation/admin-guide/cgroup-v1/memcg_test.rst +++ b/Documentation/admin-guide/cgroup-v1/memcg_test.rst @@ -43,7 +43,7 @@ Please note that implementation details can be changed. mem_cgroup_uncharge() Called when a page's refcount goes down to 0. =20 - mem_cgroup_uncharge_swap() + mem_cgroup_swap_uncharge() Called when swp_entry's refcnt goes down to 0. A charge against swap disappears. =20 diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 215e2e87f42b..4d89a35f49ff 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1905,6 +1905,7 @@ static inline bool memcg_is_dying(struct mem_cgroup *= memcg) =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP) bool obj_cgroup_may_zswap(struct obj_cgroup *objcg); +bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush); void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size); void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size); bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg); @@ -1913,6 +1914,11 @@ static inline bool obj_cgroup_may_zswap(struct obj_c= group *objcg) { return true; } + +static inline bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may= _flush) +{ + return true; +} static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size) { diff --git a/include/linux/swap.h b/include/linux/swap.h index f57f4aeeb822..95386a86d3fd 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -501,35 +501,80 @@ static inline void folio_throttle_swaprate(struct fol= io *folio, gfp_t gfp) #endif =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) -int __mem_cgroup_try_charge_swap(struct folio *folio); -static inline int mem_cgroup_try_charge_swap(struct folio *folio) +struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio); +static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio) +{ + if (mem_cgroup_disabled()) + return NULL; + return __mem_cgroup_swap_get(folio); +} + +int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pag= es); +static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg, + unsigned int nr_pages) { if (mem_cgroup_disabled()) return 0; - return __mem_cgroup_try_charge_swap(folio); + return __mem_cgroup_swap_charge(memcg, nr_pages); } =20 -extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_= pages); -static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned in= t nr_pages) +void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memc= g); +static inline void mem_cgroup_swap_record(struct folio *folio, + struct mem_cgroup *memcg) { if (mem_cgroup_disabled()) return; - __mem_cgroup_uncharge_swap(id, nr_pages); + __mem_cgroup_swap_record(folio, memcg); +} + +void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, + unsigned int nr_pages); +static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (mem_cgroup_disabled()) + return; + __mem_cgroup_swap_uncharge(memcg, nr_pages); +} + +void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages= ); +static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (mem_cgroup_disabled()) + return; + __mem_cgroup_swap_put(memcg, nr_pages); } =20 extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg); extern bool mem_cgroup_swap_full(struct folio *folio); #else -static inline int mem_cgroup_try_charge_swap(struct folio *folio) +static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio) +{ + return NULL; +} + +static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg, + unsigned int nr_pages) { return 0; } =20 -static inline void mem_cgroup_uncharge_swap(unsigned short id, +static inline void mem_cgroup_swap_record(struct folio *folio, + struct mem_cgroup *memcg) +{ +} + +static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages) { } =20 +static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ +} + static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) { return get_nr_swap_pages(); diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index 05ef55cae4dc..88016c8f22a2 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -690,6 +690,7 @@ void __memcg1_swapout(struct folio *folio, struct swap_= cluster_info *ci) void memcg1_swapin(struct folio *folio) { struct swap_cluster_info *ci; + struct mem_cgroup *memcg; unsigned long nr_pages; unsigned short id; =20 @@ -721,7 +722,14 @@ void memcg1_swapin(struct folio *folio) id =3D __swap_cgroup_clear(ci, swp_cluster_offset(folio->swap), nr_pages); swap_cluster_unlock(ci); - mem_cgroup_uncharge_swap(id, nr_pages); + + rcu_read_lock(); + memcg =3D mem_cgroup_from_private_id(id); + if (memcg) { + mem_cgroup_swap_uncharge(memcg, nr_pages); + mem_cgroup_swap_put(memcg, nr_pages); + } + rcu_read_unlock(); } #endif =20 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8508fc7e2dfd..6ae0a4191d88 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5722,93 +5722,129 @@ int __init mem_cgroup_init(void) =20 #ifdef CONFIG_SWAP /** - * __mem_cgroup_try_charge_swap - try charging swap space for a folio + * __mem_cgroup_swap_get - pin the memcg to account a folio's swap slots to * @folio: folio being added to swap * - * Try to charge @folio's memcg for the swap space at folio->swap. + * Pins one private ID ref per page of @folio on its memcg, or on its clos= est + * online ancestor if it has been offlined. The caller charges and records + * against whichever memcg is returned, so both land on the same one. * - * Returns 0 on success, -ENOMEM on failure. + * Return: the pinned memcg, or NULL if there is nothing to account. Drop = the + * pins with __mem_cgroup_swap_put(). */ -int __mem_cgroup_try_charge_swap(struct folio *folio) +struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio) { unsigned int nr_pages =3D folio_nr_pages(folio); - struct swap_cluster_info *ci; - struct page_counter *counter; struct mem_cgroup *memcg; struct obj_cgroup *objcg; =20 if (do_memsw_account()) - return 0; + return NULL; =20 objcg =3D folio_objcg(folio); VM_WARN_ON_ONCE_FOLIO(!objcg, folio); if (!objcg) - return 0; + return NULL; =20 rcu_read_lock(); memcg =3D obj_cgroup_memcg(objcg); if (!folio_test_swapcache(folio)) { memcg_memory_event(memcg, MEMCG_SWAP_FAIL); rcu_read_unlock(); - return 0; + return NULL; } =20 memcg =3D mem_cgroup_private_id_get_online(memcg, nr_pages); /* memcg is pined by memcg ID. */ rcu_read_unlock(); =20 + return memcg; +} + +/** + * __mem_cgroup_swap_charge - charge physical swap space + * @memcg: the mem_cgroup to charge (may be NULL) + * @nr_pages: the amount of swap space to charge + * + * Return: 0 on success, -ENOMEM if memory.swap.max is exceeded. + */ +int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pag= es) +{ + struct page_counter *counter; + + if (do_memsw_account() || !memcg) + return 0; + if (!mem_cgroup_is_root(memcg) && !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - mem_cgroup_private_id_put(memcg, nr_pages); return -ENOMEM; } mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); + return 0; +} + +/** + * __mem_cgroup_swap_record - record the owner of a folio's swap slots + * @folio: folio being added to swap + * @memcg: the memcg pinned by __mem_cgroup_swap_get() + */ +void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memc= g) +{ + struct swap_cluster_info *ci; =20 ci =3D swap_cluster_get_and_lock(folio); - __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages, - mem_cgroup_private_id(memcg)); + __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), + folio_nr_pages(folio), mem_cgroup_private_id(memcg)); swap_cluster_unlock(ci); - - return 0; } =20 /** - * __mem_cgroup_uncharge_swap - uncharge swap space - * @id: cgroup id to uncharge + * __mem_cgroup_swap_uncharge - uncharge physical swap space + * @memcg: the mem_cgroup to uncharge (may be NULL) * @nr_pages: the amount of swap space to uncharge */ -void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages) +void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, unsigned int nr_= pages) { - struct mem_cgroup *memcg; + if (!memcg) + return; =20 - rcu_read_lock(); - memcg =3D mem_cgroup_from_private_id(id); - if (memcg) { - if (!mem_cgroup_is_root(memcg)) { - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); - else - page_counter_uncharge(&memcg->swap, nr_pages); - } - mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages); - mem_cgroup_private_id_put(memcg, nr_pages); + if (!mem_cgroup_is_root(memcg)) { + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); + else + page_counter_uncharge(&memcg->swap, nr_pages); } - rcu_read_unlock(); + mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages); +} + +/** + * __mem_cgroup_swap_put - drop the private ID refs taken for swap slots + * @memcg: the pinned mem_cgroup + * @nr_pages: number of refs to drop + */ +void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages) +{ + mem_cgroup_private_id_put(memcg, nr_pages); } =20 long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) { - long nr_swap_pages =3D get_nr_swap_pages(); + long nr_swap_pages; =20 /* - * vswap zswap-backed swapout needs no physical slot, so gate anon - * reclaim on the swap.max headroom instead of the physical free count. + * vswap charges physical backing, not allocation, so virtual swap is + * unbounded for a zswap-capable memcg and the swap.max walk below + * would starve anon reclaim. swap.max is still enforced when the + * backing is charged. */ - if (vswap_is_enabled() && zswap_is_enabled()) - nr_swap_pages =3D PAGE_COUNTER_MAX; + if (vswap_is_enabled() && zswap_is_enabled() && + (mem_cgroup_disabled() || do_memsw_account() || + mem_cgroup_may_zswap(memcg, false))) + return PAGE_COUNTER_MAX; =20 + nr_swap_pages =3D get_nr_swap_pages(); if (mem_cgroup_disabled() || do_memsw_account()) return nr_swap_pages; for (; !mem_cgroup_is_root(memcg); memcg =3D parent_mem_cgroup(memcg)) @@ -5980,8 +6016,10 @@ static struct cftype swap_files[] =3D { =20 #ifdef CONFIG_ZSWAP /** - * obj_cgroup_may_zswap - check if this cgroup can zswap - * @objcg: the object cgroup + * mem_cgroup_may_zswap - check if this cgroup can zswap + * @memcg: the memcg to query + * @may_flush: force-flush stats for an accurate check (sleeps). Pass false + * from atomic contexts; the check is then best-effort. * * Check if the hierarchical zswap limit has been reached. * @@ -5991,36 +6029,38 @@ static struct cftype swap_files[] =3D { * spending cycles on compression when there is already no room left * or zswap is disabled altogether somewhere in the hierarchy. */ -bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) +bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush) { - struct mem_cgroup *memcg, *original_memcg; - bool ret =3D true; - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return true; =20 - original_memcg =3D get_mem_cgroup_from_objcg(objcg); - for (memcg =3D original_memcg; !mem_cgroup_is_root(memcg); - memcg =3D parent_mem_cgroup(memcg)) { + for (; !mem_cgroup_is_root(memcg); memcg =3D parent_mem_cgroup(memcg)) { unsigned long max =3D READ_ONCE(memcg->zswap_max); unsigned long pages; =20 if (max =3D=3D PAGE_COUNTER_MAX) continue; - if (max =3D=3D 0) { - ret =3D false; - break; - } + if (max =3D=3D 0) + return false; =20 /* Force flush to get accurate stats for charging */ - __mem_cgroup_flush_stats(memcg, true); + if (may_flush) + __mem_cgroup_flush_stats(memcg, true); pages =3D memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; - if (pages < max) - continue; - ret =3D false; - break; + if (pages >=3D max) + return false; } - mem_cgroup_put(original_memcg); + return true; +} + +bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) +{ + struct mem_cgroup *memcg; + bool ret; + + memcg =3D get_mem_cgroup_from_objcg(objcg); + ret =3D mem_cgroup_may_zswap(memcg, true); + mem_cgroup_put(memcg); return ret; } =20 diff --git a/mm/swapfile.c b/mm/swapfile.c index 6ec439462490..517fa58f8022 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -46,6 +46,7 @@ =20 #include #include +#include "memcontrol-v1.h" #include "swap_table.h" #include "vswap.h" #include "internal.h" @@ -2015,6 +2016,7 @@ static swp_entry_t folio_alloc_phys_swap(struct folio= *folio) int folio_alloc_swap(struct folio *folio) { unsigned int order =3D folio_order(folio); + struct mem_cgroup *memcg; unsigned int size =3D 1 << order; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); @@ -2041,9 +2043,21 @@ int folio_alloc_swap(struct folio *folio) if (!vswap_alloc(folio)) folio_alloc_phys_swap(folio); =20 - /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */ - if (unlikely(mem_cgroup_try_charge_swap(folio))) - swap_cache_del_folio(folio); + /* + * Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. + * A vswap entry has no physical swap yet, so only record the memcg. + * folio_realloc_swap() charges it once backing is allocated. + */ + memcg =3D mem_cgroup_swap_get(folio); + if (memcg) { + if (!is_vswap_entry(folio->swap) && + unlikely(mem_cgroup_swap_charge(memcg, size))) { + mem_cgroup_swap_put(memcg, size); + swap_cache_del_folio(folio); + } else { + mem_cgroup_swap_record(folio, memcg); + } + } =20 if (unlikely(!folio_test_swapcache(folio))) return -ENOMEM; @@ -2101,6 +2115,36 @@ static void __swap_cluster_free_phys_backing(struct = swap_info_struct *psi, unsigned int ci_start, unsigned int nr_pages); =20 +static void vswap_uncharge_cgroup_batch(unsigned short memcg_id, + unsigned int batch_nr, + unsigned int batch_nr_swapfile) +{ + struct mem_cgroup *memcg; + unsigned int n; + + /* + * v1 (memsw): entries keep their memsw charge across swapout + * regardless of backing, so uncharge all of them. v2: only + * swapfile-backed entries are charged, so uncharge just those. + * + * On v1 the id is written by __memcg1_swapout() as the folio leaves the + * swap cache and cleared by memcg1_swapin() when it comes back, both + * under the cluster lock. Callers still holding a cached folio are + * outside that window and see @memcg_id =3D=3D 0, so only the free path + * uncharges. On v2 the id is set when swap is allocated, so those + * callers do uncharge, which balances the charge folio_realloc_swap() + * took. + */ + n =3D do_memsw_account() ? batch_nr : batch_nr_swapfile; + if (!n) + return; + + rcu_read_lock(); + memcg =3D memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL; + rcu_read_unlock(); + mem_cgroup_swap_uncharge(memcg, n); +} + /** * __vswap_release_backing - release the backing of a range of vtable slots * @ci: the locked vswap cluster @@ -2122,12 +2166,27 @@ void __vswap_release_backing(struct swap_cluster_in= fo *ci, unsigned int ci_off; unsigned long vt; swp_entry_t phys; + unsigned short batch_id; + unsigned int batch_nr =3D 0, batch_nr_swapfile =3D 0; =20 lockdep_assert_held(&ci->lock); ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + batch_id =3D __swap_cgroup_get(ci, ci_start); =20 for (ci_off =3D ci_start; ci_off < ci_start + nr; ci_off++) { + unsigned short cur_id; + vt =3D __vtable_get(ci_dyn, ci_off); + cur_id =3D __swap_cgroup_get(ci, ci_off); + + if (cur_id !=3D batch_id) { + vswap_uncharge_cgroup_batch(batch_id, batch_nr, + batch_nr_swapfile); + batch_id =3D cur_id; + batch_nr =3D 0; + batch_nr_swapfile =3D 0; + } + batch_nr++; =20 /* The free helper takes one contiguous run within one cluster. */ if (phys_start !=3D phys_end && @@ -2146,6 +2205,7 @@ void __vswap_release_backing(struct swap_cluster_info= *ci, =20 switch (vtable_type(vt)) { case VSWAP_SWAPFILE: + batch_nr_swapfile++; if (phys_start =3D=3D phys_end) { phys =3D vtable_to_phys(vt); phys_start =3D swp_offset(phys); @@ -2179,6 +2239,8 @@ void __vswap_release_backing(struct swap_cluster_info= *ci, phys_start % SWAPFILE_CLUSTER, phys_end - phys_start); } + + vswap_uncharge_cgroup_batch(batch_id, batch_nr, batch_nr_swapfile); } =20 /** @@ -2265,7 +2327,10 @@ swp_entry_t folio_realloc_swap(struct folio *folio) swp_entry_t vswap_entry =3D folio->swap; struct swap_cluster_info *ci; struct swap_cluster_info_dynamic *ci_dyn; + struct mem_cgroup *memcg; unsigned int voff; + unsigned long vt; + unsigned short memcg_id; swp_entry_t phys_entry =3D {}; swp_entry_t pe; int i, nr =3D folio_nr_pages(folio); @@ -2274,18 +2339,37 @@ swp_entry_t folio_realloc_swap(struct folio *folio) VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); VM_WARN_ON(!is_vswap_entry(vswap_entry)); =20 - phys_entry =3D vswap_to_phys(vswap_entry); - if (phys_entry.val) - return phys_entry; + voff =3D swp_cluster_offset(vswap_entry); + ci =3D __swap_entry_to_cluster(vswap_entry); + ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); + + spin_lock(&ci->lock); + vt =3D __vtable_get(ci_dyn, voff); + if (vtable_type(vt) =3D=3D VSWAP_SWAPFILE) { + spin_unlock(&ci->lock); + return vtable_to_phys(vt); + } + memcg_id =3D __swap_cgroup_get(ci, voff); + spin_unlock(&ci->lock); =20 phys_entry =3D folio_alloc_phys_swap(folio); if (!phys_entry.val) return (swp_entry_t){}; =20 - voff =3D swp_cluster_offset(vswap_entry); + rcu_read_lock(); + memcg =3D folio_memcg(folio); + if (!memcg || mem_cgroup_private_id(memcg) !=3D memcg_id) + memcg =3D memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL; + rcu_read_unlock(); + + if (mem_cgroup_swap_charge(memcg, nr)) { + __swap_cluster_free_phys_backing(__swap_entry_to_info(phys_entry), + __swap_entry_to_cluster(phys_entry), + swp_cluster_offset(phys_entry), + nr); + return (swp_entry_t){}; + } =20 - ci =3D __swap_entry_to_cluster(vswap_entry); - ci_dyn =3D container_of(ci, struct swap_cluster_info_dynamic, ci); spin_lock(&ci->lock); /* * Install PHYS backing without freeing any prior contents of the @@ -2465,6 +2549,25 @@ static void __swap_cluster_free_phys_backing(struct = swap_info_struct *psi, swap_cluster_unlock(pci); } =20 +/* + * Release the cgroup accounting of a batch of freed slots. For vswap the + * physical swap was already uncharged by __vswap_release_backing(), so on= ly + * the ID ref is left to drop. + */ +static void memcg_swap_free(unsigned short id, unsigned int nr, bool is_vs= wap) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); + memcg =3D mem_cgroup_from_private_id(id); + if (memcg) { + if (!is_vswap) + mem_cgroup_swap_uncharge(memcg, nr); + mem_cgroup_swap_put(memcg, nr); + } + rcu_read_unlock(); +} + /* * Free a set of swap slots after their swap count dropped to zero, or wil= l be * zero after putting the last ref (saves one __swap_cluster_put_entry cal= l). @@ -2477,10 +2580,11 @@ void __swap_cluster_free_entries(struct swap_info_s= truct *si, unsigned short batch_id =3D 0, id_cur; unsigned int ci_off =3D ci_start, ci_end =3D ci_start + nr_pages; unsigned int batch_off =3D ci_off; + bool is_vswap =3D swap_is_vswap(si); =20 VM_WARN_ON(ci->count < nr_pages); =20 - if (swap_is_vswap(si)) + if (is_vswap) __vswap_release_backing(ci, ci_start, nr_pages); =20 ci->count -=3D nr_pages; @@ -2504,14 +2608,14 @@ void __swap_cluster_free_entries(struct swap_info_s= truct *si, id_cur =3D __swap_cgroup_clear(ci, ci_off, 1); if (batch_id !=3D id_cur) { if (batch_id) - mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off); + memcg_swap_free(batch_id, ci_off - batch_off, is_vswap); batch_id =3D id_cur; batch_off =3D ci_off; } } while (++ci_off < ci_end); =20 if (batch_id) - mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off); + memcg_swap_free(batch_id, ci_off - batch_off, is_vswap); =20 __swap_cluster_finish_free(si, ci, ci_start, nr_pages); } --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oa1-f48.google.com (mail-oa1-f48.google.com [209.85.160.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE58A2F747A for ; Tue, 25 Aug 2026 15:32:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671984; cv=none; b=qf4VWVr2gM24hpLbwBNmOXxb//Bk9VG8vaQdlk0KaGAz3aZ8CVUvHAGbYmi0jOE4Y94vDBuNa7qCzDpL80U3Ths4ZUR0K8UOu2ejJUqW6x2rtbtsVgGiOZF1g6N9S/+k+0XOPYwW7wuCZtVptOXeXQsNHGAXbjKH0YBknJpVo9I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671984; c=relaxed/simple; bh=7RdfGNwJGIhjQira6vfwXUtsmsJVS3QW6ObOGHjEZac=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gMFTwUHkPmZ1Mx3ULrG5BxxkXv9XFq/a/nxCM1fWZ/WuaWhQ/diFxuJYfvZ6i8EK/FQoujv8qvW+KdIBY6k7KWZ9AAJIs08JBaBVbH9loDVskys3yoJ4EdGEEJV5pRGVjjiILSLIqOCXEUm4NRHmF4pB7wLX6RVJm4O3lMiFtQs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JUeJopBE; arc=none smtp.client-ip=209.85.160.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JUeJopBE" Received: by mail-oa1-f48.google.com with SMTP id 586e51a60fabf-4563ac048f4so4727127fac.0 for ; Tue, 25 Aug 2026 08:32:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671976; x=1788276776; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=rRVrRpujJcoHdsF0W6IfC5VyfQfzFj4+BdlWYD2xQfM=; b=JUeJopBEk2oDKXIstUSpBzgQvbrlelqGvN4OwaZc9kIkXLXenGx2VBEBckocKtHjDT efbXIqnTecWRPIA40NF+phHEbBpWwMTOiyBUyRa/3y5lLzUZ/0rZ434dv9LPio6dBj0N ldWW9MhAHESkcEB5ECWvdBsgPdvSy2XQn/UFJldTsQ1ixOjvUqGdsm+4SeQySYIJlgyH 5mKOZb99+ReXXdyoVmvYlKbDZrACgkP8HOgQbbg+MljbBv5W5JDWEM2sSUca4tQqI6qe 6sphLU4Fq1nG1ifHHtCdspwDgAePkssPid7/VGOW37UTWUXN2kVzho0DACm9U3SLMNd0 XyTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671976; x=1788276776; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=rRVrRpujJcoHdsF0W6IfC5VyfQfzFj4+BdlWYD2xQfM=; b=Zcc0aKWyVd9wBNq4t9kE349AUWC9kQC33zJvlE8xcXMxXopW/E9XklkOaBOkvjH3aP ltp087du5B9YPk1rkwiJzcKI0QGUO4E8IJV71ijQF5uLnQdzcnUboo11u7bYRSWaGH73 tyamh2VHaPBMYGhSZy8iiZB7Wbjz05TMHRuFLwUfL0IeZGNzGKcixP7cTS1y8LtEwTgF 7BN9ox7h6+7VGVxsr2b8dx5zK36ZPvSEk1ZSDKBxTytPxTJn8I+XbB2FUtbCq6rnQbTP SeD0vZ9h0dtgb7+JSF4Oaf/KYvGjbtwfPQDpZjoNOZZ3F5RfOJj0DnD2J8nxum7leLHQ IgXg== X-Forwarded-Encrypted: i=1; AHgh+RofpOYrFbsNkuC7g9CZ7scpT1elDNoWi3Sl9jfBCeaNGHqefNlvABHzbOyF+G3FXCAPkGIy0aVQbiaClGY=@vger.kernel.org X-Gm-Message-State: AFuF++nCjL7f0t3oojxJh+VjrIKZosG9I3weec1M+oKUqnTnvXPWbPv4 34Fnk95NUz7LU83JV48ON2eTW+PcVyYhS1cMMqnCAFh7adbzkNuwxOMH X-Gm-Gg: AR+sD10peJqOPSsQ9+dy+OSV+zHs5naHjcb7j0esk4io60lzkpoHx5bftQ/De7bJ+pM 0pKDMtgC1BFgDl+IHp3XIpfQvgHHjr8t1gmuRyfyK596Dx8YRlLgcbo5Dn35/+Q169p8jUzr1T0 g1O4TM75XQ4xQMa8upjal39NOy2okHhbZcqDczPvcoRCS+LT0UmDNsRdM8UUh72tU2O2CNppeAB lw+k0sW0w3z1nRzsv3SdiPmChsWKdhCCcE1ZVdfHOWbw0F+tVQkfWKHzDF9mi6RCmWjBG4+Zg5T mwiNUCobMU72pXCRw9gUqZ+l1CBZJLLJF8Cwjg/e0h2QfW+AstXSKuS6ANlBg+ueZsF1mV6bTlI h3YFOZ+hNiEFOolABvS/8+nB73sJcpUKc0Zx/ycMYlvHOm+cCMf/3ApoamODFB71OqucRbUtpkB dToENtMhNZHAeXF4YS39fPgkE08PQGrF1YGZeIWJTTZfpR5FVW3uuhpNW4bIouzDyNeJTEkOyOz IzFo/i6dW1VMsKcX0wEIQ== X-Received: by 2002:a05:6820:8106:b0:6ae:5250:dd1c with SMTP id 006d021491bc7-6b1904fd029mr6471337eaf.29.1787671975541; Tue, 25 Aug 2026 08:32:55 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:42::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4638353a37asm7759186fac.15.2026.08.25.08.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:54 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 09/11] mm, swap: add debugfs counters for vswap Date: Tue, 25 Aug 2026 08:32:35 -0700 Message-ID: <20260825153238.2695446-10-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add /sys/kernel/debug/vswap/ with two counters: * used: virtual swap slots (pages) currently allocated * alloc_reject: cumulative pages that failed to get a vswap slot Signed-off-by: Nhat Pham --- mm/swapfile.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/swapfile.c b/mm/swapfile.c index 517fa58f8022..3b10173cddc3 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -7,6 +7,7 @@ */ =20 #include +#include #include #include #include @@ -142,6 +143,8 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, perc= pu_vswap_cluster) =3D { .lock =3D INIT_LOCAL_LOCK(), }; =20 +static atomic_long_t vswap_alloc_reject =3D ATOMIC_LONG_INIT(0); + static bool vswap_alloc(struct folio *folio); static void vswap_mark_cache_only(struct swap_cluster_info *ci, unsigned int ci_off); @@ -1983,6 +1986,7 @@ static bool vswap_alloc(struct folio *folio) =20 this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID); local_unlock(&percpu_vswap_cluster.lock); + atomic_long_add(folio_nr_pages(folio), &vswap_alloc_reject); return false; } =20 @@ -4795,9 +4799,25 @@ early_param("vswap", early_vswap); /* vswap does no IO on its own. */ static const struct swap_ops vswap_ops =3D { }; =20 +static int vswap_used_get(void *data, u64 *val) +{ + *val =3D swap_usage_in_pages(vswap_si); + return 0; +} +DEFINE_DEBUGFS_ATTRIBUTE(vswap_used_fops, vswap_used_get, NULL, "%llu\n"); + +static int vswap_alloc_reject_get(void *data, u64 *val) +{ + *val =3D atomic_long_read(&vswap_alloc_reject); + return 0; +} +DEFINE_DEBUGFS_ATTRIBUTE(vswap_alloc_reject_fops, vswap_alloc_reject_get, = NULL, + "%llu\n"); + static int __init vswap_init(void) { struct swap_info_struct *si; + struct dentry *root; unsigned long maxpages; int err; =20 @@ -4841,6 +4861,12 @@ static int __init vswap_init(void) mutex_unlock(&swapon_mutex); =20 vswap_si =3D si; + + root =3D debugfs_create_dir("vswap", NULL); + debugfs_create_file("used", 0444, root, NULL, &vswap_used_fops); + debugfs_create_file("alloc_reject", 0444, root, NULL, + &vswap_alloc_reject_fops); + pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages); =20 /* Last: everything above must be visible before routing starts. */ --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oa1-f42.google.com (mail-oa1-f42.google.com [209.85.160.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1952148A8C2 for ; Tue, 25 Aug 2026 15:33:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671986; cv=none; b=QMRUH64t7loXEdtGkBLDe6cxOO/gWYnCDyNjS2RF8yQaLzQmFey/NJSfsWUYqUi5gKKcHzBMFv1mMxfT+qUdVEk3cLEwMhOaHPjgkv6KEnumoXwvtOO13UxBtRoSfBNlhe3mIBXWGhV6dPz7otaYhlES3i1sRt2wBl2paG3VAOs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671986; c=relaxed/simple; bh=nDSTRjBO6H+M0nB3bfo7Tt9POfOY5EN48fG/RNTOvO0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Nc9KRHKGRT1Eg6UjqRbLK1kO+1NJvhEE8iZntp6YRx+w/iJQSEZtcM76yjdGfVA8g08GA2UHgOdfG0XC+cVP3PI92X6RFOoQhi69F+D67WcDN8Dv8lztM+UfBhkp1tgzjkmBoI7I1hSpUnJUV3pcBsDbR+yETPDGa4LwLIe0hcY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lU8bPhZo; arc=none smtp.client-ip=209.85.160.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lU8bPhZo" Received: by mail-oa1-f42.google.com with SMTP id 586e51a60fabf-46556b9e02cso290616fac.1 for ; Tue, 25 Aug 2026 08:33:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671977; x=1788276777; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=e+G2HY1nMuhTy8GReU5FQmdFCwNyu4BLzojPY+0AOwU=; b=lU8bPhZoAnQ520vS9cRFOrNTx219eDjaXHH9twsqSfxVw93X6hug5TKlAiDEGyyRUk WcMNFZQtuxbuLiNO6ONDPTY4DRBKiI6tShv5N0B6I3SOBmOwa1rwbGFc6zx6fA/zu98N sJfjaGWHtLHkyXzSm9euZQpZrUFDM6/HStbHUBofioLIe4WCh+1IjjWLQ0cJl1XuUSEb PXhRNFVuFBlM011JhEg6AoHxbOwCfLmfGTZ/m6/sijXRxG2290+ERZVF1GQvRym+1OK3 nnTDolMFHHofj2BpR7S/rC6cHsIu/M7wMAdC0MlyAtI1kmszb7byRx8n4R/5bsGFvTNN u9ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671977; x=1788276777; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=e+G2HY1nMuhTy8GReU5FQmdFCwNyu4BLzojPY+0AOwU=; b=hUbg2iNWOjA9lGfKIaeYcfXJyK3qZ1VmgOqy11qi/fs8EaDIqTDzlxXThSem1LZDLM xHdeoH4NFGjgvqUp5MWQIuJ8HlB1JCoCF85y7YdjMoQlkbsDau2jd05QcGWqzf5cebh7 Mek7j9BOqV7IqwJ6tyx524xBGM4GwQ4oHXMkINhWjtuUn0ZhUPp0PI8K0L+vuYl3R5z8 HBdxWUImFnqrRq1VGW/0Z0Gpth4cOW7JFXfmieLZmg6agW7GyzU4YWwyd/4sOe49slYS MXN2B6qlgzumxqfdQxHY3YekpA84x3/H/Jhte8jYXfYhxyGjxytM6Dp5ow9nQVcCmwtz zljA== X-Forwarded-Encrypted: i=1; AHgh+RoRarWgYRLaPqs0h2ZOsKFdpZKAmJ5AKWsraNNaDCTh+JpdPEa4JeNvst29NFLLp5Xl/+7+AaeM+I9gBcc=@vger.kernel.org X-Gm-Message-State: AFuF++ky9UWQJ52aQO5BruMesF3Z7KJ9Si9atQoOIEOqnj6Hsj9e0pTb 132mWQprhjMO5c6op/H3l5vfeIx+7nvb9EeJonpUb+IG8bN6o2761CJY X-Gm-Gg: AR+sD10xbVwuyhbEKYn/qI5fVLgML2SVaaVuKlCzXkcDb/KBs8M7DgbE8D1oBQLP+4g ticEqhRA+hrv+En9h94lwmKvdYT7Oo/kINi/nOh6Le9FXil1ytsUnKfbLMEkeUgD+jbO/lVg1Vc VjHXwlI0z6GuN2wwkybq3FneYqfxwQXlgSFlzxDN57FqEmtGm2cRYeFC2nnuv+ilpxMgGvuV6PV JDI392JBLhuzHCDnxgvzUq9NFZx3IxSRzhSZogctsXW3parT9MuWGU2hD7uXp4o+gOEg4wS5FYN 1lNagAWeFp4UHGNCnHaLCVufiAB3nj+GQkh3Wmw9x3Ja9fDnyjur9fKaoKkMT0qlnWn44ae+JK7 BDEXCUjw0rnFKI32/+h7NdFqwsp3wRM3KE0lIRxxlq/K82UToaRkDVLmbyZDX58CwosyAsFRO0T wR/wKn4zhvunfbjtmwhiGH58d6R+60SkzNDpqg/euBnQ6vnMyQSG+t3C4alFluofarTIg0BZ7e7 PU9EfnKvA== X-Received: by 2002:a05:6871:d607:b0:451:fbeb:e9a6 with SMTP id 586e51a60fabf-464443cbc91mr6041524fac.0.1787671977026; Tue, 25 Aug 2026 08:32:57 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:2::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-463831397bbsm7562679fac.4.2026.08.25.08.32.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:56 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 10/11] mm, swap: defer memcg_table allocation for physical swap clusters Date: Tue, 25 Aug 2026 08:32:36 -0700 Message-ID: <20260825153238.2695446-11-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Stop allocating a memcg table for every physical swap cluster that only ever holds vswap backings. The table costs SWAPFILE_CLUSTER * sizeof(unsigned short) per cluster, 1 KB per 2 MB of swap on a 64-bit kernel with 4 KB pages. On a vswap-heavy workload, where zswap writeback is the only consumer of physical swap, that is the common case. Such clusters never have their memcg_table read or written: vswap-layer charging records on the vswap cluster's table, not the physical one. Allocate eagerly only where the table is known to be needed: every vswap cluster, and, when vswap is off, every physical cluster, since all of its slots then map directly into the PTEs. A physical cluster otherwise allocates on its first direct-use slot, and skips entirely if it only holds vswap backings. That deferred allocation can fail, though rarely. Signed-off-by: Nhat Pham --- mm/swapfile.c | 85 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 66 insertions(+), 19 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 3b10173cddc3..8c80544a7d25 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -471,7 +471,8 @@ static void swap_cluster_free_table(struct swap_cluster= _info *ci) swap_cluster_free_table_folio_rcu_cb); } =20 -static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gf= p) +static int swap_cluster_alloc_table(struct swap_info_struct *si, + struct swap_cluster_info *ci, gfp_t gfp) { struct swap_table *table =3D NULL; struct folio *folio; @@ -494,7 +495,14 @@ static int swap_cluster_alloc_table(struct swap_cluste= r_info *ci, gfp_t gfp) rcu_assign_pointer(ci->table, table); =20 #ifdef CONFIG_MEMCG - if (!mem_cgroup_disabled()) { + /* + * A physical cluster under vswap may hold only vswap backings, which + * record their memcg on the vswap cluster's table, not this one. Such + * clusters defer memcg_table allocation until they hand out a slot + * that maps directly into the PTEs. + */ + if ((!vswap_is_enabled() || swap_is_vswap(si)) && + !mem_cgroup_disabled()) { VM_WARN_ON_ONCE(ci->memcg_table); ci->memcg_table =3D kzalloc_obj(*ci->memcg_table, gfp); if (!ci->memcg_table) { @@ -565,8 +573,8 @@ swap_cluster_populate(struct swap_info_struct *si, lockdep_assert_held(&si->global_cluster_lock); lockdep_assert_held(&ci->lock); =20 - if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC | - __GFP_NOWARN)) + if (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC | + __GFP_NOWARN)) return ci; =20 /* @@ -579,8 +587,8 @@ swap_cluster_populate(struct swap_info_struct *si, spin_unlock(&si->global_cluster_lock); local_unlock(&percpu_swap_cluster.lock); =20 - ret =3D swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC | - GFP_KERNEL); + ret =3D swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC | + GFP_KERNEL); =20 /* * Back to atomic context. We might have migrated to a new CPU with a @@ -857,7 +865,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info= _struct *si, =20 ci =3D cluster_info + idx; /* Need to allocate swap table first for initial bad slot marking. */ - if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL)) + if (!ci->count && swap_cluster_alloc_table(si, ci, GFP_KERNEL)) return -ENOMEM; spin_lock(&ci->lock); /* Check for duplicated bad swap slots. */ @@ -1079,7 +1087,9 @@ static bool __swap_cluster_alloc_entries(struct swap_= info_struct *si, /* Try use a new cluster for current CPU and allocate from it. */ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci, - struct folio *folio, unsigned long offset) + struct folio *folio, + unsigned long offset, + bool *nomem) { unsigned int next =3D SWAP_ENTRY_INVALID, found =3D SWAP_ENTRY_INVALID; unsigned long start =3D ALIGN_DOWN(offset, SWAPFILE_CLUSTER); @@ -1108,6 +1118,24 @@ static unsigned int alloc_swap_scan_cluster(struct s= wap_info_struct *si, if (!ret) continue; } +#ifdef CONFIG_MEMCG + /* + * Lazy-allocate memcg_table on the first direct-use slot of a + * physical cluster. + */ + if (vswap_is_enabled() && folio && + !folio_test_swapcache(folio) && !mem_cgroup_disabled() && + !ci->memcg_table) { + ci->memcg_table =3D kzalloc_obj(*ci->memcg_table, + GFP_ATOMIC | __GFP_NOMEMALLOC | + __GFP_NOWARN); + if (!ci->memcg_table) { + if (nomem) + *nomem =3D true; + goto out; + } + } +#endif if (!__swap_cluster_alloc_entries(si, ci, folio, offset % SWAPFILE_CLUST= ER)) break; found =3D offset; @@ -1117,7 +1145,15 @@ static unsigned int alloc_swap_scan_cluster(struct s= wap_info_struct *si, break; } out: - relocate_cluster(si, ci); + /* + * On a discard-capable device, relocating a cluster whose memcg_table + * allocation failed queues a discard for slots that were never used, + * which folio_alloc_phys_swap() reads as progress and retries on. + */ + if (nomem && *nomem && !ci->count) + __free_cluster(si, ci); + else + relocate_cluster(si, ci); swap_cluster_unlock(ci); if (swap_is_vswap(si)) { this_cpu_write(percpu_vswap_cluster.offset[order], next); @@ -1138,7 +1174,13 @@ static unsigned int alloc_swap_scan_list(struct swap= _info_struct *si, bool scan_all) { unsigned int found =3D SWAP_ENTRY_INVALID; + bool nomem =3D false; =20 + /* + * In rare cases alloc_swap_scan_cluster() can fail due to + * memcg_table allocation failure. Short-circuit to avoid looping + * over the list indefinitely. + */ do { struct swap_cluster_info *ci =3D isolate_lock_cluster(si, list); unsigned long offset; @@ -1146,10 +1188,10 @@ static unsigned int alloc_swap_scan_list(struct swa= p_info_struct *si, if (!ci) break; offset =3D cluster_offset(si, ci); - found =3D alloc_swap_scan_cluster(si, ci, folio, offset); + found =3D alloc_swap_scan_cluster(si, ci, folio, offset, &nomem); if (found) break; - } while (scan_all); + } while (scan_all && !nomem); =20 return found; } @@ -1170,7 +1212,7 @@ static unsigned int vswap_alloc_cluster(struct swap_i= nfo_struct *si, spin_lock_init(&ci_dyn->ci.lock); INIT_LIST_HEAD(&ci_dyn->ci.list); =20 - if (swap_cluster_alloc_table(&ci_dyn->ci, GFP_ATOMIC)) { + if (swap_cluster_alloc_table(si, &ci_dyn->ci, GFP_ATOMIC)) { kfree(ci_dyn); return SWAP_ENTRY_INVALID; } @@ -1196,7 +1238,7 @@ static unsigned int vswap_alloc_cluster(struct swap_i= nfo_struct *si, } =20 offset =3D cluster_offset(si, ci); - return alloc_swap_scan_cluster(si, ci, folio, offset); + return alloc_swap_scan_cluster(si, ci, folio, offset, NULL); } =20 static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool f= orce) @@ -1303,7 +1345,8 @@ static unsigned long cluster_alloc_swap_entry(struct = swap_info_struct *si, if (cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset =3D cluster_offset(si, ci); - found =3D alloc_swap_scan_cluster(si, ci, folio, offset); + found =3D alloc_swap_scan_cluster(si, ci, folio, offset, + NULL); } else { swap_cluster_unlock(ci); } @@ -1347,7 +1390,8 @@ static unsigned long cluster_alloc_swap_entry(struct = swap_info_struct *si, if (order < PMD_ORDER) { /* * Scan only one fragment cluster is good enough. Order 0 - * allocation will surely success, and large allocation + * allocation will surely success unless the memcg table + * allocation fails, which is rare, and large allocation * failure is not critical. Scanning one cluster still * keeps the list rotated and reclaimed (for clean swap cache). */ @@ -1363,7 +1407,8 @@ static unsigned long cluster_alloc_swap_entry(struct = swap_info_struct *si, for (int o =3D 1; o < SWAP_NR_ORDERS; o++) { /* * Clusters here have at least one usable slots and can't fail order 0 - * allocation, but reclaim may drop si->lock and race with another user. + * allocation, but reclaim may drop si->lock and race with another user, + * and the memcg table allocation may fail. */ found =3D alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true); if (found) @@ -1586,7 +1631,7 @@ static swp_entry_t swap_alloc_fast(struct folio *foli= o) if (ci && cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset =3D cluster_offset(si, ci); - found =3D alloc_swap_scan_cluster(si, ci, folio, offset); + found =3D alloc_swap_scan_cluster(si, ci, folio, offset, NULL); } else if (ci) { swap_cluster_unlock(ci); } @@ -1969,7 +2014,8 @@ static bool vswap_alloc(struct folio *folio) if (ci && cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset =3D cluster_offset(vswap_si, ci); - alloc_swap_scan_cluster(vswap_si, ci, folio, offset); + alloc_swap_scan_cluster(vswap_si, ci, folio, offset, + NULL); } else if (ci) { swap_cluster_unlock(ci); } @@ -2836,7 +2882,8 @@ swp_entry_t swap_alloc_hibernation_slot(int type) if (pcp_si =3D=3D si && pcp_offset) { ci =3D swap_cluster_lock(si, pcp_offset); if (cluster_is_usable(ci, 0)) - offset =3D alloc_swap_scan_cluster(si, ci, NULL, pcp_offset); + offset =3D alloc_swap_scan_cluster(si, ci, NULL, + pcp_offset, NULL); else swap_cluster_unlock(ci); } --=20 2.53.0-Meta From nobody Wed Sep 23 16:31:19 2026 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDCAD48A2BE for ; Tue, 25 Aug 2026 15:33:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671988; cv=none; b=JF+CarYuc4FyAcsuFY+XOU+KNfHGeaQeRWsPcYT/od8gDYTjh9WzIJAHJKzgmdjUYsCzgT73avikdfPaXHPSyaqHu8FgKptbogFIFTrulVFVTvNj18S/msvWCrhGetXlMG7UqA1ksI/rFa/W0Fpkfe5HjmNv7CbGaoKViJxsg2o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671988; c=relaxed/simple; bh=ESa/aJb+Zcb+vI6rmyLQaTfjj0+Al0OL+w/Lgyn3Yu0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=adB2n1DHRPgH7iGxMYD8KAY7Fs4leScM/pv9itZxxNarKEcUKmSb4cUG8GXPMJfTGJ3xYlUBx4LdsTkVQuIbSlIYgVJ1BLI6Q8xiY7j7P5LDsq0kj042AmIrQoHX8v6e6t9nZbUnWcn8OgnJBisCyhI5b4MiHf+vdOfRuNmjRcg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hjYysJX3; arc=none smtp.client-ip=209.85.167.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hjYysJX3" Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-4af173320f9so4007053b6e.2 for ; Tue, 25 Aug 2026 08:33:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787671979; x=1788276779; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uAKqLnS2duDg8f3jGAvM1PgrDRSN4wgtQcZNf8+IpfY=; b=hjYysJX3IVpEhW0E6s0hQ6MM5lz7Gr5QgJO+Z38p3ugGbP3op1OHJorZsL+4FNXd9i Zh7DtUPcIMP+tgyQPfvAYEk1MmEANWLOSODa4hmBhshAcP4Bs6ey76yuSTLDupxwWx65 sZL53TSsyXSZAx/HrBbUtHnkkR+EsLFBYNFfezy60oc3kAvH3Bs7H5zKkdBdJeX77zbT AnquNzbPqdPllOMVV9TlcLmx0Ia1Zw+Cf+B+ApESH4qpKSTNrnbuF3OBK7kj1eODtgQU dsjfOe08atULZl9xgFqModnWExflixhwG8SPVrQoF/ffaXgxKuT6ZCGbGc/WruJJlMet TiRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787671979; x=1788276779; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=uAKqLnS2duDg8f3jGAvM1PgrDRSN4wgtQcZNf8+IpfY=; b=VkUUEulEjGNgLd8OIaorV67A6cyiQRu1tQHzYo36sNDYC4D+OyNennVvpF4tRhBoyK pUQdibaF3SNmRcJKVBAE/ZVYc6EF7WbcQnJtwdP3TNDRtV3HdhQSRE1KQytUg0FHvEdl 9on8RO6ir9ObtbJRviVCp6ub7EwDPWX+1OicQm/N9g8qh8VTWjDwcVOrAGwhh9CVXI4Z hfnHJ077+baxg3MI/jmyWVW5f1tvY1mJXUPsGOzx3MS0vspRlGB+lOILw5VtxS5uR22r MFQVF6kijURjhBswH0Yubp7PMR2hhgXw3EEI1QU2VNgovbLXTrK/AdcBUbghd2h04FHk ki0w== X-Forwarded-Encrypted: i=1; AHgh+RrdUuP6bEc0qqvlXGiPJITkRZqRHnhECVWJ3TixejevGAnLf/k66vM+/2ojo+4/tRyrb8j0bg+wwFkA9Wk=@vger.kernel.org X-Gm-Message-State: AFuF++nMLN+8jApq2keKqBiWzFnzeHq4CeZXWeBbLkDRIXtwcRfXNvnA xsFyPjn/DNxhYQXUNxg4RP2rkkFlZ1R/GfLgxapSomWb3VKJduAi1c2Y X-Gm-Gg: AR+sD13TVsle5hUdGbLYp8ZwSTAhkCdyagdDMX0ePNU8eGBDFVmxpuOxNB2wP03GVpd AYi6EadVItxlIhmyr/NnNyAiOtDnrY+j6wADMd2vl/B4WuHFUkLBgiFbslJtTHt+DSMrTY4DoME 2JqXGabVm/AJ8knBQApGG/FChTwxh6CAyCS9Jj7Ur2msSGwkEMAg/V7WmlaAHQEBR1pc1R7KReu GCu6GQoV36ufYMs4a8+b2ytHTFQ099F+ofFYiJFLWqInYonKMLs1WcX/XbgJKTo0iDUWyGQGVAO 2bKIdf19BUwhOoXWDM2eB/XzwyOEbBbe1lqNe0QROMefQaHP+6K9JM1HfxorsCZ+mmLVzveU6b6 F2SiqHzfxMoLWOxM7KV2NikQqg0D6rEq/nAHrm5Eh8biCjTgQyI5WVR4gWPzf9j9MT3BH1rVACZ ndZGztyyx0Tk6oi8wfg69uq9GYiVzFKCTzVP1Sn5X7dpKtY2mrOWZ14oZgckYyMzwEh8Xa94HfB 5Ure2qgJHI= X-Received: by 2002:a05:6808:4f49:b0:495:ff85:d33d with SMTP id 5614622812f47-4b35059a824mr7724933b6e.12.1787671978714; Tue, 25 Aug 2026 08:32:58 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:28::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-4b32c690f27sm5952581b6e.7.2026.08.25.08.32.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2026 08:32:58 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: chrisl@kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, yosry@kernel.org, david@kernel.org, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, chengming.zhou@linux.dev, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, qi.zheng@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, riel@surriel.com, gourry@gourry.net, haowenchao22@gmail.com, corbet@lwn.net, hughd@google.com, baolin.wang@linux.alibaba.com, tj@kernel.org, mkoutny@suse.com, skhan@linuxfoundation.org, kunwu.chan@linux.dev, kernel-team@meta.com, nphamcs@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 11/11] mm, swap: widen swap_info_struct max/pages to unsigned long Date: Tue, 25 Aug 2026 08:32:37 -0700 Message-ID: <20260825153238.2695446-12-nphamcs@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260825153238.2695446-1-nphamcs@gmail.com> References: <20260825153238.2695446-1-nphamcs@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Widen swap_info_struct->max and ->pages from unsigned int to unsigned long so the vswap device can exceed the previous 16 TB cap with 4 KB pages (ALIGN_DOWN(UINT_MAX, SWAPFILE_CLUSTER) pages). The new vswap cap is the cluster_info_pool xarray's allocator limit. XA_FLAGS_ALLOC stores allocated IDs in u32, so max_pages =3D UINT_MAX * SWAPFILE_CLUSTER (~8 PB at the typical SWAPFILE_CLUSTER=3D512 layout). Signed-off-by: Nhat Pham --- include/linux/swap.h | 4 +-- mm/swapfile.c | 63 ++++++++++++++++++++++---------------------- 2 files changed, 34 insertions(+), 33 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 95386a86d3fd..5310ac0f3faa 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -246,7 +246,7 @@ struct swap_info_struct { signed short prio; /* swap priority of this type */ struct plist_node list; /* entry in swap_active_head */ signed char type; /* strange name for an index */ - unsigned int max; /* size of this swap device */ + unsigned long max; /* size of this swap device */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ struct list_head free_clusters; /* free clusters list */ struct list_head full_clusters; /* full clusters list */ @@ -254,7 +254,7 @@ struct swap_info_struct { /* list of cluster that contains at least one free slot */ struct list_head frag_clusters[SWAP_NR_ORDERS]; /* list of cluster that are fragmented or contented */ - unsigned int pages; /* total of usable pages of swap */ + unsigned long pages; /* total of usable pages of swap */ atomic_long_t inuse_pages; /* number of those currently in use */ struct swap_sequential_cluster *global_cluster; /* Use one global cluster= for rotating device */ spinlock_t global_cluster_lock; /* Serialize usage of global cluster */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 8c80544a7d25..9860fb3b079f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -429,10 +429,10 @@ static inline unsigned int cluster_index(struct swap_= info_struct *si, return ci - si->cluster_info; } =20 -static inline unsigned int cluster_offset(struct swap_info_struct *si, - struct swap_cluster_info *ci) +static inline unsigned long cluster_offset(struct swap_info_struct *si, + struct swap_cluster_info *ci) { - return cluster_index(si, ci) * SWAPFILE_CLUSTER; + return (unsigned long)cluster_index(si, ci) * SWAPFILE_CLUSTER; } =20 static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head) @@ -848,7 +848,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info= _struct *si, =20 /* si->max may got shrunk by swap swap_activate() */ if (offset >=3D si->max && !mask) { - pr_debug("Ignoring bad slot %u (max: %u)\n", offset, si->max); + pr_debug("Ignoring bad slot %u (max: %lu)\n", offset, si->max); return 0; } /* @@ -1085,13 +1085,13 @@ static bool __swap_cluster_alloc_entries(struct swa= p_info_struct *si, } =20 /* Try use a new cluster for current CPU and allocate from it. */ -static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, - struct swap_cluster_info *ci, - struct folio *folio, - unsigned long offset, - bool *nomem) +static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si, + struct swap_cluster_info *ci, + struct folio *folio, + unsigned long offset, + bool *nomem) { - unsigned int next =3D SWAP_ENTRY_INVALID, found =3D SWAP_ENTRY_INVALID; + unsigned long next =3D SWAP_ENTRY_INVALID, found =3D SWAP_ENTRY_INVALID; unsigned long start =3D ALIGN_DOWN(offset, SWAPFILE_CLUSTER); unsigned int order =3D likely(folio) ? folio_order(folio) : 0; unsigned long end =3D start + SWAPFILE_CLUSTER; @@ -1168,12 +1168,12 @@ static unsigned int alloc_swap_scan_cluster(struct = swap_info_struct *si, return found; } =20 -static unsigned int alloc_swap_scan_list(struct swap_info_struct *si, - struct list_head *list, - struct folio *folio, - bool scan_all) +static unsigned long alloc_swap_scan_list(struct swap_info_struct *si, + struct list_head *list, + struct folio *folio, + bool scan_all) { - unsigned int found =3D SWAP_ENTRY_INVALID; + unsigned long found =3D SWAP_ENTRY_INVALID; bool nomem =3D false; =20 /* @@ -1196,8 +1196,8 @@ static unsigned int alloc_swap_scan_list(struct swap_= info_struct *si, return found; } =20 -static unsigned int vswap_alloc_cluster(struct swap_info_struct *si, - struct folio *folio) +static unsigned long vswap_alloc_cluster(struct swap_info_struct *si, + struct folio *folio) { struct swap_cluster_info_dynamic *ci_dyn; struct swap_cluster_info *ci; @@ -1324,7 +1324,7 @@ static unsigned long cluster_alloc_swap_entry(struct = swap_info_struct *si, { struct swap_cluster_info *ci; unsigned int order =3D likely(folio) ? folio_order(folio) : 0; - unsigned int offset =3D SWAP_ENTRY_INVALID, found =3D SWAP_ENTRY_INVALID; + unsigned long offset =3D SWAP_ENTRY_INVALID, found =3D SWAP_ENTRY_INVALID; =20 /* * Swapfile is not block device so unable @@ -3423,10 +3423,10 @@ static int unuse_mm(struct mm_struct *mm, unsigned = int type) * Return 0 if there are no inuse entries after prev till end of * the map. */ -static unsigned int find_next_to_unuse(struct swap_info_struct *si, - unsigned int prev) +static unsigned long find_next_to_unuse(struct swap_info_struct *si, + unsigned long prev) { - unsigned int i; + unsigned long i; unsigned long swp_tb; =20 /* @@ -3464,7 +3464,8 @@ static int try_to_unuse(unsigned int type) struct swap_io_ctx ctx; swp_entry_t entry, vswap_entry; unsigned long swp_tb; - unsigned int i, j; + unsigned long i; + unsigned int j; =20 if (!swap_usage_in_pages(si)) goto success; @@ -3921,7 +3922,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, special= file) struct file *swap_file, *victim; struct address_space *mapping; struct inode *inode; - unsigned int maxpages; + unsigned long maxpages; int err, found =3D 0; =20 if (!capable(CAP_SYS_ADMIN)) @@ -4343,12 +4344,8 @@ static unsigned long read_swap_header(struct swap_in= fo_struct *si, pr_warn("Truncating oversized swap area, only using %luk out of %luk\n", K(maxpages), K(last_page)); } - if (maxpages > last_page) { + if (maxpages > last_page) maxpages =3D last_page + 1; - /* p->max is an unsigned int: don't overflow it */ - if ((unsigned int)maxpages =3D=3D 0) - maxpages =3D UINT_MAX; - } =20 if (!maxpages) return 0; @@ -4589,7 +4586,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialf= ile, int, swap_flags) goto bad_swap_unlock_inode; } if (si->pages !=3D si->max - 1) { - pr_err("swap:%u !=3D (max:%u - 1)\n", si->pages, si->max); + pr_err("swap:%lu !=3D (max:%lu - 1)\n", si->pages, si->max); error =3D -EINVAL; goto bad_swap_unlock_inode; } @@ -4677,7 +4674,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialf= ile, int, swap_flags) /* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */ enable_swap_info(si); =20 - pr_info("Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s= %s%s\n", + pr_info("Adding %luk swap on %s. Priority:%d extents:%d across:%lluk %s%= s%s%s\n", K(si->pages), name->name, si->prio, nr_extents, K((unsigned long long)span), (si->flags & SWP_SOLIDSTATE) ? "SS" : "", @@ -4884,8 +4881,12 @@ static int __init vswap_init(void) return 0; } =20 + /* + * One u32 xarray ID per cluster, so the device cannot be larger + * than UINT_MAX clusters. + */ maxpages =3D min(swapfile_maximum_size, - ALIGN_DOWN((unsigned long)UINT_MAX, SWAPFILE_CLUSTER)); + (unsigned long)UINT_MAX * SWAPFILE_CLUSTER); /* * SWP_WRITEOK enables slot allocation. SWP_SOLIDSTATE selects * per-CPU cluster allocation; vswap has no si->global_cluster. --=20 2.53.0-Meta