From nobody Wed Sep 10 10:58:05 2025 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 368E5280339 for ; Fri, 5 Sep 2025 19:14:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757099685; cv=none; b=dNgPkmaIFyZTUSrRthHrm9ffOss7Kmp/OeCbx5YW1kZ4VV80yifw7cB04u8daZ2MpqFeCtoFi0czkYr+/vSU30zBtoLHWuosKZRwAPX7Y5LPuX/VpYpIbNBBvUpEfyIPKdvBCDBocIxmXUadrMeTKDkGay4b9Kzw9xjs5Q8gEE8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757099685; c=relaxed/simple; bh=HjAkCiKreUoJyDEivAofk+EGvLIortYa6b1SPhd5BSc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DCUPejG8a33RDLzhl6ZtLesljdrep9QeQ6fk7lDmzuXwBuyYucjtj8/b4SlND9Z2wfUcWHF30qUiIEZK4fF4suEBVmlIt1LsN9F3TQoEjpIGdJlo4YM5Atul9xul/RZHgAXvPpj+GCEdIbpLG6UlWzrqv0NIshL9fgwiaQWFvic= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Eh0CTWXy; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eh0CTWXy" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-772488c78bcso2456523b3a.1 for ; Fri, 05 Sep 2025 12:14:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1757099682; x=1757704482; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ZXCjnErSqW7lrOTzOAfqkITRvdAoHNXzorUyab5yxLY=; b=Eh0CTWXysjaHdiBR69ens2c8h3wfQ5hebFNRklR9sCwmZfGnc3z+C7QQvd3LnLS3SP PrIv3l9iFRLJrsfAcmem0NKwQqFFPTR96GpLZErRrTLPtxK2qCCGUCl74PQo+XbiyB+y N85u96e3rg6b1opPeF9eiV3dIij6KC48x9mUx5aDt+ynoUrAsdQqmR+U0QH3YftRKCaS Vwkim0BaURTrTphkjg9WUpjYG09GN5L+TuBO0KqDDXKvUW7qJs/+1tvHeKsThoy//Pcy 7usHmWOo4WLJCh2jYqEjLmTdwIs9yY2IlEIk/qOsEp0WqAGta5N1fegrBl8k7TI2h9sj 4UWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1757099682; x=1757704482; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZXCjnErSqW7lrOTzOAfqkITRvdAoHNXzorUyab5yxLY=; b=cBN2ZhiwiB8geewbwJhDjbqLTvpWYfGgT8cbbo1Zgg92KyKMbDxKvnXjhWHJrDUCM2 LYHEjf4fT5DhSVmntvhUnv92G9/RQKK5iIFAYS8SA4Yce26CAPdCC7H1thbKvCFREq5t 3QxCWYp02chcuwY1Mzj87hSRe3PpXBKC9uCBvxLrJDY4hDEoszulgQvI6Jd5rYQicDBU OF6/QhE8V3PScvKQWurt/WFrWcUKSnJdEKTjSV8r+uT8G14TkxP9CFQFrtp6eDZ1Lmma xixQzaEnv8lBdxEQQLksfpJp39PjPugo0JAfcB2MW6GLrVEmj/E2HNXaNn4A257YTOlv f+7g== X-Forwarded-Encrypted: i=1; AJvYcCUNqRzeJK4Rm0E4DU+EYrpFwS5/SqkPFrUTvnXIiGIZXnLm/3Y0knThoDy/ox7su3dfPL/5JygOEMZJEe0=@vger.kernel.org X-Gm-Message-State: AOJu0YyLDvsSLx/J520bldh08AM+5yDT5U5efS8XIHVhjZUbxZ/mIMaS S6y7chy5+jo90xg4y/xB9lqiLBTvMgsjqnswUXsLsHYNBo/KgaE0Tzzm X-Gm-Gg: ASbGncsSzAUvu6Pejb5UA7YF1jrEbvbtjPerwfCCjXq+bmjeNqmowiDNTw2v5jNEXGs 2qB1Kl8jEd0l/D+3TUiWAAS/E5HTNc0yipw1BBioHiosEy0rOLcxo3ZH3gUqIWr6w4vsYIQDKns rMFT8mT27TlEvPJLyUh94jmfHKod/HHyePGPEMUXdcyJknRB6mLvZJh4jWeaAy3knjSQID1DnHb bnQitMnljx40arJwLPIjCVWy1QVxnY4L2B9fN9n8z1zfLpphKZC9e0QPDg5X83k1+D7pFKFOXYI t2+tExPz2djS45GdMoL73Fekf8LI1zaCyyRA51mekNyCHuT0LPQoekadu2VrBAgBJY7Eb2vv7jh gIo9Zcp2D5wlN6GqULV3dfxHv2G4DaINVZh3Z5/nIi1u3hyIjMY+rybm+Sg== X-Google-Smtp-Source: AGHT+IEBYA0uesip3wIKaccDeR63V4mHmwyoEjZ2FLF3Pja8XPH7B2M+y/xcl5swlQp/VY6U0RNV9A== X-Received: by 2002:a05:6a00:2e2a:b0:772:5fad:e6d7 with SMTP id d2e1a72fcca58-7725fadf0a9mr29719836b3a.32.1757099682203; Fri, 05 Sep 2025 12:14:42 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-77256a0f916sm15871442b3a.63.2025.09.05.12.14.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 05 Sep 2025 12:14:41 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Hugh Dickins , Chris Li , Barry Song , Baoquan He , Nhat Pham , Kemeng Shi , Baolin Wang , Ying Huang , Johannes Weiner , David Hildenbrand , Yosry Ahmed , Lorenzo Stoakes , Zi Yan , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 06/15] mm, swap: rename and move some swap cluster definition and helpers Date: Sat, 6 Sep 2025 03:13:48 +0800 Message-ID: <20250905191357.78298-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20250905191357.78298-1-ryncsn@gmail.com> References: <20250905191357.78298-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song No feature change, move cluster related definitions and helpers to mm/swap.h, also tidy up and add a "swap_" prefix for cluster lock/unlock helpers, so they can be used outside of swap files. And while at it, add kerneldoc. Signed-off-by: Kairui Song Acked-by: Chris Li Reviewed-by: Barry Song Acked-by: David Hildenbrand Reviewed-by: Baolin Wang --- include/linux/swap.h | 34 ---------------- mm/swap.h | 70 ++++++++++++++++++++++++++++++++ mm/swapfile.c | 97 +++++++++++++------------------------------- 3 files changed, 99 insertions(+), 102 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 23452f014ca1..7e1fe4ff3d30 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -235,40 +235,6 @@ enum { /* Special value in each swap_map continuation */ #define SWAP_CONT_MAX 0x7f /* Max count */ =20 -/* - * We use this to track usage of a cluster. A cluster is a block of swap d= isk - * space with SWAPFILE_CLUSTER pages long and naturally aligns in disk. All - * free clusters are organized into a list. We fetch an entry from the lis= t to - * get a free cluster. - * - * The flags field determines if a cluster is free. This is - * protected by cluster lock. - */ -struct swap_cluster_info { - spinlock_t lock; /* - * Protect swap_cluster_info fields - * other than list, and swap_info_struct->swap_map - * elements corresponding to the swap cluster. - */ - u16 count; - u8 flags; - u8 order; - struct list_head list; -}; - -/* All on-list cluster must have a non-zero flag. */ -enum swap_cluster_flags { - CLUSTER_FLAG_NONE =3D 0, /* For temporary off-list cluster */ - CLUSTER_FLAG_FREE, - CLUSTER_FLAG_NONFULL, - CLUSTER_FLAG_FRAG, - /* Clusters with flags above are allocatable */ - CLUSTER_FLAG_USABLE =3D CLUSTER_FLAG_FRAG, - CLUSTER_FLAG_FULL, - CLUSTER_FLAG_DISCARD, - CLUSTER_FLAG_MAX, -}; - /* * The first page in the swap file is the swap header, which is always mar= ked * bad to prevent it from being allocated as an entry. This also prevents = the diff --git a/mm/swap.h b/mm/swap.h index a69e18b12b45..39b27337bc0a 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -7,10 +7,80 @@ struct swap_iocb; =20 extern int page_cluster; =20 +#ifdef CONFIG_THP_SWAP +#define SWAPFILE_CLUSTER HPAGE_PMD_NR +#define swap_entry_order(order) (order) +#else +#define SWAPFILE_CLUSTER 256 +#define swap_entry_order(order) 0 +#endif + +/* + * We use this to track usage of a cluster. A cluster is a block of swap d= isk + * space with SWAPFILE_CLUSTER pages long and naturally aligns in disk. All + * free clusters are organized into a list. We fetch an entry from the lis= t to + * get a free cluster. + * + * The flags field determines if a cluster is free. This is + * protected by cluster lock. + */ +struct swap_cluster_info { + spinlock_t lock; /* + * Protect swap_cluster_info fields + * other than list, and swap_info_struct->swap_map + * elements corresponding to the swap cluster. + */ + u16 count; + u8 flags; + u8 order; + struct list_head list; +}; + +/* All on-list cluster must have a non-zero flag. */ +enum swap_cluster_flags { + CLUSTER_FLAG_NONE =3D 0, /* For temporary off-list cluster */ + CLUSTER_FLAG_FREE, + CLUSTER_FLAG_NONFULL, + CLUSTER_FLAG_FRAG, + /* Clusters with flags above are allocatable */ + CLUSTER_FLAG_USABLE =3D CLUSTER_FLAG_FRAG, + CLUSTER_FLAG_FULL, + CLUSTER_FLAG_DISCARD, + CLUSTER_FLAG_MAX, +}; + #ifdef CONFIG_SWAP #include /* for swp_offset */ #include /* for bio_end_io_t */ =20 +static inline struct swap_cluster_info *swp_offset_cluster( + struct swap_info_struct *si, pgoff_t offset) +{ + return &si->cluster_info[offset / SWAPFILE_CLUSTER]; +} + +/** + * swap_cluster_lock - Lock and return the swap cluster of given offset. + * @si: swap device the cluster belongs to. + * @offset: the swap entry offset, pointing to a valid slot. + * + * Context: The caller must ensure the offset is in the valid range and + * protect the swap device with reference count or locks. + */ +static inline struct swap_cluster_info *swap_cluster_lock( + struct swap_info_struct *si, unsigned long offset) +{ + struct swap_cluster_info *ci =3D swp_offset_cluster(si, offset); + + spin_lock(&ci->lock); + return ci; +} + +static inline void swap_cluster_unlock(struct swap_cluster_info *ci) +{ + spin_unlock(&ci->lock); +} + /* linux/mm/page_io.c */ int sio_pool_init(void); struct swap_iocb; diff --git a/mm/swapfile.c b/mm/swapfile.c index 1bd90f17440f..547ad4bfe1d8 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -58,9 +58,6 @@ static void swap_entries_free(struct swap_info_struct *si, static void swap_range_alloc(struct swap_info_struct *si, unsigned int nr_entries); static bool folio_swapcache_freeable(struct folio *folio); -static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si, - unsigned long offset); -static inline void unlock_cluster(struct swap_cluster_info *ci); =20 static DEFINE_SPINLOCK(swap_lock); static unsigned int nr_swapfiles; @@ -257,9 +254,9 @@ static int __try_to_reclaim_swap(struct swap_info_struc= t *si, * swap_map is HAS_CACHE only, which means the slots have no page table * reference or pending writeback, and can't be allocated to others. */ - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); need_reclaim =3D swap_only_has_cache(si, offset, nr_pages); - unlock_cluster(ci); + swap_cluster_unlock(ci); if (!need_reclaim) goto out_unlock; =20 @@ -384,19 +381,6 @@ static void discard_swap_cluster(struct swap_info_stru= ct *si, } } =20 -#ifdef CONFIG_THP_SWAP -#define SWAPFILE_CLUSTER HPAGE_PMD_NR - -#define swap_entry_order(order) (order) -#else -#define SWAPFILE_CLUSTER 256 - -/* - * Define swap_entry_order() as constant to let compiler to optimize - * out some code if !CONFIG_THP_SWAP - */ -#define swap_entry_order(order) 0 -#endif #define LATENCY_LIMIT 256 =20 static inline bool cluster_is_empty(struct swap_cluster_info *info) @@ -424,34 +408,12 @@ static inline unsigned int cluster_index(struct swap_= info_struct *si, return ci - si->cluster_info; } =20 -static inline struct swap_cluster_info *offset_to_cluster(struct swap_info= _struct *si, - unsigned long offset) -{ - return &si->cluster_info[offset / SWAPFILE_CLUSTER]; -} - static inline unsigned int cluster_offset(struct swap_info_struct *si, struct swap_cluster_info *ci) { return cluster_index(si, ci) * SWAPFILE_CLUSTER; } =20 -static inline struct swap_cluster_info *lock_cluster(struct swap_info_stru= ct *si, - unsigned long offset) -{ - struct swap_cluster_info *ci; - - ci =3D offset_to_cluster(si, offset); - spin_lock(&ci->lock); - - return ci; -} - -static inline void unlock_cluster(struct swap_cluster_info *ci) -{ - spin_unlock(&ci->lock); -} - static void move_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci, struct list_head *list, enum swap_cluster_flags new_flags) @@ -807,7 +769,7 @@ static unsigned int alloc_swap_scan_cluster(struct swap= _info_struct *si, } out: relocate_cluster(si, ci); - unlock_cluster(ci); + swap_cluster_unlock(ci); if (si->flags & SWP_SOLIDSTATE) { this_cpu_write(percpu_swap_cluster.offset[order], next); this_cpu_write(percpu_swap_cluster.si[order], si); @@ -874,7 +836,7 @@ static void swap_reclaim_full_clusters(struct swap_info= _struct *si, bool force) if (ci->flags =3D=3D CLUSTER_FLAG_NONE) relocate_cluster(si, ci); =20 - unlock_cluster(ci); + swap_cluster_unlock(ci); if (to_scan <=3D 0) break; } @@ -913,7 +875,7 @@ static unsigned long cluster_alloc_swap_entry(struct sw= ap_info_struct *si, int o if (offset =3D=3D SWAP_ENTRY_INVALID) goto new_cluster; =20 - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); /* Cluster could have been used by another order */ if (cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) @@ -921,7 +883,7 @@ static unsigned long cluster_alloc_swap_entry(struct sw= ap_info_struct *si, int o found =3D alloc_swap_scan_cluster(si, ci, offset, order, usage); } else { - unlock_cluster(ci); + swap_cluster_unlock(ci); } if (found) goto done; @@ -1202,7 +1164,7 @@ static bool swap_alloc_fast(swp_entry_t *entry, if (!si || !offset || !get_swap_device_info(si)) return false; =20 - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); if (cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset =3D cluster_offset(si, ci); @@ -1210,7 +1172,7 @@ static bool swap_alloc_fast(swp_entry_t *entry, if (found) *entry =3D swp_entry(si->type, found); } else { - unlock_cluster(ci); + swap_cluster_unlock(ci); } =20 put_swap_device(si); @@ -1478,14 +1440,14 @@ static void swap_entries_put_cache(struct swap_info= _struct *si, unsigned long offset =3D swp_offset(entry); struct swap_cluster_info *ci; =20 - ci =3D lock_cluster(si, offset); - if (swap_only_has_cache(si, offset, nr)) + ci =3D swap_cluster_lock(si, offset); + if (swap_only_has_cache(si, offset, nr)) { swap_entries_free(si, ci, entry, nr); - else { + } else { for (int i =3D 0; i < nr; i++, entry.val++) swap_entry_put_locked(si, ci, entry, SWAP_HAS_CACHE); } - unlock_cluster(ci); + swap_cluster_unlock(ci); } =20 static bool swap_entries_put_map(struct swap_info_struct *si, @@ -1503,7 +1465,7 @@ static bool swap_entries_put_map(struct swap_info_str= uct *si, if (count !=3D 1 && count !=3D SWAP_MAP_SHMEM) goto fallback; =20 - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); if (!swap_is_last_map(si, offset, nr, &has_cache)) { goto locked_fallback; } @@ -1512,21 +1474,20 @@ static bool swap_entries_put_map(struct swap_info_s= truct *si, else for (i =3D 0; i < nr; i++) WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE); - unlock_cluster(ci); + swap_cluster_unlock(ci); =20 return has_cache; =20 fallback: - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); locked_fallback: for (i =3D 0; i < nr; i++, entry.val++) { count =3D swap_entry_put_locked(si, ci, entry, 1); if (count =3D=3D SWAP_HAS_CACHE) has_cache =3D true; } - unlock_cluster(ci); + swap_cluster_unlock(ci); return has_cache; - } =20 /* @@ -1576,7 +1537,7 @@ static void swap_entries_free(struct swap_info_struct= *si, unsigned char *map_end =3D map + nr_pages; =20 /* It should never free entries across different clusters */ - VM_BUG_ON(ci !=3D offset_to_cluster(si, offset + nr_pages - 1)); + VM_BUG_ON(ci !=3D swp_offset_cluster(si, offset + nr_pages - 1)); VM_BUG_ON(cluster_is_empty(ci)); VM_BUG_ON(ci->count < nr_pages); =20 @@ -1651,9 +1612,9 @@ bool swap_entry_swapped(struct swap_info_struct *si, = swp_entry_t entry) struct swap_cluster_info *ci; int count; =20 - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); count =3D swap_count(si->swap_map[offset]); - unlock_cluster(ci); + swap_cluster_unlock(ci); return !!count; } =20 @@ -1676,7 +1637,7 @@ int swp_swapcount(swp_entry_t entry) =20 offset =3D swp_offset(entry); =20 - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); =20 count =3D swap_count(si->swap_map[offset]); if (!(count & COUNT_CONTINUED)) @@ -1699,7 +1660,7 @@ int swp_swapcount(swp_entry_t entry) n *=3D (SWAP_CONT_MAX + 1); } while (tmp_count & COUNT_CONTINUED); out: - unlock_cluster(ci); + swap_cluster_unlock(ci); return count; } =20 @@ -1714,7 +1675,7 @@ static bool swap_page_trans_huge_swapped(struct swap_= info_struct *si, int i; bool ret =3D false; =20 - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); if (nr_pages =3D=3D 1) { if (swap_count(map[roffset])) ret =3D true; @@ -1727,7 +1688,7 @@ static bool swap_page_trans_huge_swapped(struct swap_= info_struct *si, } } unlock_out: - unlock_cluster(ci); + swap_cluster_unlock(ci); return ret; } =20 @@ -2660,8 +2621,8 @@ static void wait_for_allocation(struct swap_info_stru= ct *si) BUG_ON(si->flags & SWP_WRITEOK); =20 for (offset =3D 0; offset < end; offset +=3D SWAPFILE_CLUSTER) { - ci =3D lock_cluster(si, offset); - unlock_cluster(ci); + ci =3D swap_cluster_lock(si, offset); + swap_cluster_unlock(ci); } } =20 @@ -3577,7 +3538,7 @@ static int __swap_duplicate(swp_entry_t entry, unsign= ed char usage, int nr) offset =3D swp_offset(entry); VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); VM_WARN_ON(usage =3D=3D 1 && nr > 1); - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); =20 err =3D 0; for (i =3D 0; i < nr; i++) { @@ -3632,7 +3593,7 @@ static int __swap_duplicate(swp_entry_t entry, unsign= ed char usage, int nr) } =20 unlock_out: - unlock_cluster(ci); + swap_cluster_unlock(ci); return err; } =20 @@ -3731,7 +3692,7 @@ int add_swap_count_continuation(swp_entry_t entry, gf= p_t gfp_mask) =20 offset =3D swp_offset(entry); =20 - ci =3D lock_cluster(si, offset); + ci =3D swap_cluster_lock(si, offset); =20 count =3D swap_count(si->swap_map[offset]); =20 @@ -3791,7 +3752,7 @@ int add_swap_count_continuation(swp_entry_t entry, gf= p_t gfp_mask) out_unlock_cont: spin_unlock(&si->cont_lock); out: - unlock_cluster(ci); + swap_cluster_unlock(ci); put_swap_device(si); outer: if (page) --=20 2.51.0