From nobody Fri Sep 25 04:07:54 2026 Received: from lgeamrelo07.lge.com (lgeamrelo07.lge.com [156.147.51.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A47CA550DA0 for ; Wed, 16 Sep 2026 18:34:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=156.147.51.103 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583693; cv=none; b=dJmx9FguvUwRTgkiovdzURB7CZF32jNxlYHdIdxa1yaUCH9VoVnXYKBj7EQsAJj60Zg0M4otKXeQABPmt+8+qbJ9ehtPIofjRIabtp2A0fzVstvtQtwTYYc9e61DOQ5DBRP4hFu4wKY2bgkLvdtuGX53VfU12SIDmoPAcUiqEq4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583693; c=relaxed/simple; bh=9vQQUggjZS/CTjvK13LaKTfDvlY33IA8A65kzKIFyFA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UZ6XBRDahUJeaE6RpUzplGIgfxrTh16cO/YiPwUKfKuMNKxFUoUULDu48OUosvVzyO3YKHOROelJ/P9x4JZVsY3Bz++PF3XJ1kfqhR2qr8TagVlPIkd7E0DddtsAOytgvqFEZaE7KxGT4xI18mSYlx9IDTFB7cHY19limYRhGTk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com; spf=pass smtp.mailfrom=lge.com; arc=none smtp.client-ip=156.147.51.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=lge.com Received: from unknown (HELO yjaykim-PowerEdge-T330.lge.net) (10.177.112.156) by 156.147.51.103 with ESMTP; 17 Sep 2026 03:34:37 +0900 X-Original-SENDERIP: 10.177.112.156 X-Original-MAILFROM: youngjun.park@lge.com From: Youngjun Park To: akpm@linux-foundation.org Cc: chrisl@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, yosry@kernel.org, joshua.hahnjy@gmail.com, taejoon.song@lge.com, her0gyugyu@gmail.com, lianux.mm@gmail.com Subject: [RFC PATCH v11 1/4] mm: swap: introduce swap tier infrastructure Date: Thu, 17 Sep 2026 03:34:34 +0900 Message-Id: <20260916183437.2946306-2-youngjun.park@lge.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260916183437.2946306-1-youngjun.park@lge.com> References: <20260916183437.2946306-1-youngjun.park@lge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce the "swap tier" as an abstraction over swap devices, so that swap allocation is organized on a tier basis rather than on a flat list of devices. Swap priority already points this way. Devices with different priorities are used in priority order, and devices with the same priority are used round-robin. A priority therefore already behaves like a tier, a group of devices that share one service speed. This patch gives that group an explicit structure, without adding any user interface. A swap tier is the set of swap devices that share a priority. A tier is created when the first device with its priority is swapped on, and removed when the last one is swapped off. The active tiers are kept sorted by priority for allocation. Making the tier own its devices gives a same-priority group the data structures it needs and lays the groundwork for tier-based allocation, onto which per-cgroup swap device selection can later be fit. No tier feature and no user-visible change are introduced here. This only prepares the ground for them. Suggested-by: Chris Li Signed-off-by: Youngjun Park --- MAINTAINERS | 2 + include/linux/swap.h | 2 +- mm/Makefile | 2 +- mm/swap.h | 1 + mm/swap_tier.c | 114 +++++++++++++++++++++++++++++++++++++++++++ mm/swap_tier.h | 37 ++++++++++++++ mm/swapfile.c | 66 +++++++++++++------------ 7 files changed, 192 insertions(+), 32 deletions(-) create mode 100644 mm/swap_tier.c create mode 100644 mm/swap_tier.h diff --git a/MAINTAINERS b/MAINTAINERS index e4412c3d8d45..37f353015cae 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17410,6 +17410,8 @@ F: mm/page_io.c F: mm/swap.h F: mm/swap_table.h F: mm/swap_state.c +F: mm/swap_tier.c +F: mm/swap_tier.h F: mm/swapfile.c =20 MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE) diff --git a/include/linux/swap.h b/include/linux/swap.h index 43155e122b5c..22ccb4b5801e 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -237,7 +237,7 @@ struct swap_info_struct { struct percpu_ref users; /* indicate and keep swap device valid. */ unsigned long flags; /* SWP_USED etc: see above */ signed short prio; /* swap priority of this type */ - struct plist_node list; /* entry in swap_active_head */ + struct plist_node list; /* entry in its swap tier */ signed char type; /* strange name for an index */ unsigned int max; /* size of this swap device */ struct swap_cluster_info *cluster_info; /* array, one entry per cluster */ diff --git a/mm/Makefile b/mm/Makefile index 2a3ec53d62ee..d89a7abadc46 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -76,7 +76,7 @@ ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) +=3D madvise.o endif =20 -obj-$(CONFIG_SWAP) +=3D page_io.o swap_state.o swapfile.o +obj-$(CONFIG_SWAP) +=3D page_io.o swap_state.o swapfile.o swap_tier.o obj-$(CONFIG_ZSWAP) +=3D zswap.o obj-$(CONFIG_HAS_DMA) +=3D dmapool.o obj-$(CONFIG_HUGETLBFS) +=3D hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o diff --git a/mm/swap.h b/mm/swap.h index b3b54c28929a..4de6b9b0f261 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -36,6 +36,7 @@ struct swap_io_ctx; #define swap_entry_order(order) 0 #endif =20 +extern spinlock_t swap_lock; extern struct swap_info_struct *swap_info[]; =20 /* diff --git a/mm/swap_tier.c b/mm/swap_tier.c new file mode 100644 index 000000000000..8ed1427cee09 --- /dev/null +++ b/mm/swap_tier.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +#include "swap.h" +#include "swap_tier.h" + +#define MAX_SWAPTIER MAX_SWAPFILES + +static struct swap_tier swap_tiers[MAX_SWAPTIER]; + +/* active swap priority list, sorted in descending order */ +LIST_HEAD(swap_tier_active_list); +/* unused swap_tier object */ +static LIST_HEAD(swap_tier_inactive_list); + +#define for_each_tier(tier, idx) \ + for (idx =3D 0, tier =3D &swap_tiers[0]; idx < MAX_SWAPTIER; \ + idx++, tier =3D &swap_tiers[idx]) + +/* + * Naming Convention: + * swap_tiers_*() - Public/exported functions + * swap_tier_*() - Private/internal functions + */ + +static struct swap_tier *swap_tier_lookup(short prio) +{ + struct swap_tier *tier; + + for_each_active_tier(tier) { + if (tier->prio =3D=3D prio) + return tier; + } + + return NULL; +} + +/* Insert new tier into the active list sorted by priority. */ +static void swap_tier_activate(struct swap_tier *new) +{ + struct list_head *pos =3D &swap_tier_active_list; + struct swap_tier *tier; + + for_each_active_tier(tier) { + if (tier->prio <=3D new->prio) { + pos =3D &tier->list; + break; + } + } + + list_add_tail(&new->list, pos); +} + +static void swap_tier_inactivate(struct swap_tier *tier) +{ + list_move_tail(&tier->list, &swap_tier_inactive_list); +} + +void swap_tiers_init(void) +{ + struct swap_tier *tier; + int idx; + + BUILD_BUG_ON(BITS_PER_TYPE(int) < MAX_SWAPTIER); + + for_each_tier(tier, idx) { + plist_head_init(&tier->active_head); + INIT_LIST_HEAD(&tier->list); + swap_tier_inactivate(tier); + } +} + +static struct swap_tier *swap_tier_prepare(short prio) +{ + struct swap_tier *tier; + + lockdep_assert_held(&swap_lock); + + /* A tier holds at least one device, so one is always unused. */ + tier =3D list_first_entry(&swap_tier_inactive_list, + struct swap_tier, list); + + list_del_init(&tier->list); + tier->prio =3D prio; + + return tier; +} + +void swap_tiers_assign_dev(struct swap_info_struct *swp) +{ + struct swap_tier *tier; + + lockdep_assert_held(&swap_lock); + + tier =3D swap_tier_lookup(swp->prio); + if (!tier) { + tier =3D swap_tier_prepare(swp->prio); + swap_tier_activate(tier); + } + + plist_add(&swp->list, &tier->active_head); +} + +void swap_tiers_remove_dev(struct swap_info_struct *swp) +{ + struct swap_tier *tier; + + lockdep_assert_held(&swap_lock); + + tier =3D swap_tier_lookup(swp->prio); + plist_del(&swp->list, &tier->active_head); + if (plist_head_empty(&tier->active_head)) + swap_tier_inactivate(tier); +} diff --git a/mm/swap_tier.h b/mm/swap_tier.h new file mode 100644 index 000000000000..3dce716d23f6 --- /dev/null +++ b/mm/swap_tier.h @@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _SWAP_TIER_H +#define _SWAP_TIER_H + +#include +#include +#include + +/* Forward declarations */ +struct swap_info_struct; + +/* + * struct swap_tier - structure representing a swap tier. + * + * @prio: priority of the swap devices in the tier. + * @active_head: swap devices in the tier. + * @list: linkage into swap_tier_active_list or swap_tier_inactive_list. + */ +struct swap_tier { + short prio; + struct plist_head active_head; + struct list_head list; +}; + +extern struct list_head swap_tier_active_list; + +#define for_each_active_tier(tier) \ + list_for_each_entry(tier, &swap_tier_active_list, list) + +/* Initialization and application */ +void swap_tiers_init(void); + +/* Tier assignment */ +void swap_tiers_assign_dev(struct swap_info_struct *swp); +void swap_tiers_remove_dev(struct swap_info_struct *swp); + +#endif /* _SWAP_TIER_H */ diff --git a/mm/swapfile.c b/mm/swapfile.c index c0eddccfaca2..8201ae779833 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -48,6 +48,7 @@ #include "swap_table.h" #include "internal.h" #include "swap.h" +#include "swap_tier.h" =20 static void swap_range_alloc(struct swap_info_struct *si, unsigned int nr_entries); @@ -61,9 +62,9 @@ static void move_cluster(struct swap_info_struct *si, * lazily allocated & freed swap device info struts, and SWP_USED indicates * which device is used, ~SWP_USED devices and can be reused. * - * Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK fl= ag. + * Also protects the swap tiers, total_swap_pages, and the SWP_WRITEOK fla= g. */ -static DEFINE_SPINLOCK(swap_lock); +DEFINE_SPINLOCK(swap_lock); static unsigned int nr_swapfiles; atomic_long_t nr_swap_pages; /* @@ -83,17 +84,11 @@ bool swap_migration_ad_supported; static const char Bad_file[] =3D "Bad swap file entry "; static const char Bad_offset[] =3D "Bad swap offset entry "; =20 -/* - * all active swap_info_structs - * protected with swap_lock, and ordered by priority. - */ -static PLIST_HEAD(swap_active_head); - /* * all available (active, not full) swap_info_structs * protected with swap_avail_lock, ordered by priority. - * This is used by folio_alloc_swap() instead of swap_active_head - * because swap_active_head includes all swap_info_structs, + * This is used by folio_alloc_swap() instead of the active lists of + * the swap tiers because those include all swap_info_structs, * but folio_alloc_swap() doesn't need to look at full ones. * This uses its own lock instead of swap_lock because when a * swap_info_struct changes between not-full/full, it needs to @@ -1444,22 +1439,27 @@ static bool swap_sync_discard(void) { bool ret =3D false; struct swap_info_struct *si, *next; + struct swap_tier *tier; + short prio; =20 spin_lock(&swap_lock); start_over: - plist_for_each_entry_safe(si, next, &swap_active_head, list) { - spin_unlock(&swap_lock); - if (get_swap_device_info(si)) { - if (si->flags & SWP_PAGE_DISCARD) - ret =3D swap_do_scheduled_discard(si); - put_swap_device(si); - } - if (ret) - return true; + for_each_active_tier(tier) { + prio =3D tier->prio; + plist_for_each_entry_safe(si, next, &tier->active_head, list) { + spin_unlock(&swap_lock); + if (get_swap_device_info(si)) { + if (si->flags & SWP_PAGE_DISCARD) + ret =3D swap_do_scheduled_discard(si); + put_swap_device(si); + } + if (ret) + return true; =20 - spin_lock(&swap_lock); - if (plist_node_empty(&next->list)) - goto start_over; + spin_lock(&swap_lock); + if (plist_node_empty(&next->list) || tier->prio !=3D prio) + goto start_over; + } } spin_unlock(&swap_lock); =20 @@ -3088,7 +3088,7 @@ static void _enable_swap_info(struct swap_info_struct= *si) =20 assert_spin_locked(&swap_lock); =20 - plist_add(&si->list, &swap_active_head); + swap_tiers_assign_dev(si); =20 /* Add back to available list */ add_to_avail_list(si, true); @@ -3182,6 +3182,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, special= file) { struct swap_info_struct *p =3D NULL; struct swap_cluster_info *cluster_info; + struct swap_tier *tier; struct file *swap_file, *victim; struct address_space *mapping; struct inode *inode; @@ -3200,13 +3201,17 @@ SYSCALL_DEFINE1(swapoff, const char __user *, speci= alfile) =20 mapping =3D victim->f_mapping; spin_lock(&swap_lock); - plist_for_each_entry(p, &swap_active_head, list) { - if (p->flags & SWP_WRITEOK) { - if (p->swap_file->f_mapping =3D=3D mapping) { - found =3D 1; - break; + for_each_active_tier(tier) { + plist_for_each_entry(p, &tier->active_head, list) { + if (p->flags & SWP_WRITEOK) { + if (p->swap_file->f_mapping =3D=3D mapping) { + found =3D 1; + break; + } } } + if (found) + break; } if (!found) { err =3D -EINVAL; @@ -3230,7 +3235,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, special= file) } spin_lock(&p->lock); del_from_avail_list(p, true); - plist_del(&p->list, &swap_active_head); + swap_tiers_remove_dev(p); atomic_long_sub(p->pages, &nr_swap_pages); total_swap_pages -=3D p->pages; spin_unlock(&p->lock); @@ -4002,7 +4007,7 @@ int swap_dup_entry_direct(swp_entry_t entry) #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) static bool __has_usable_swap(void) { - return !plist_head_empty(&swap_active_head); + return !list_empty(&swap_tier_active_list); } =20 void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp) @@ -4055,6 +4060,7 @@ static int __init swapfile_init(void) swap_migration_ad_supported =3D true; #endif /* CONFIG_MIGRATION */ =20 + swap_tiers_init(); return 0; } subsys_initcall(swapfile_init); --=20 2.48.1 From nobody Fri Sep 25 04:07:54 2026 Received: from lgeamrelo07.lge.com (lgeamrelo07.lge.com [156.147.51.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA1174D7D59 for ; Wed, 16 Sep 2026 18:34:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=156.147.51.103 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583728; cv=none; b=PO049gyDy32LM3DDmsL2ZG/df3qM2A3p7IsDqyNjz/IMcp8QlFevuK+0D1jS2U3w5qlIFdGU3uE4GeEdI0GoaaBb/xMsRK36qyHKcERNlRwPojaAKPrOHX2iXy+eE5jVepW8VUsVIEthSeiaJ46FyRZPtLlEvS6gCoU28h7gKLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583728; c=relaxed/simple; bh=8qGs0fOihYVVVFhyxjHRRvvTBCWlCtj8nIkvzKD246M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tjMVewpeyTWecyjp2dLOUJ9Ze+bxlBkW5mDtRalaI5lm5Wg2DMyU2Iq7f1Jm/GQfCKJO2YuZ6s+izpGSNZ05OWYuS6LPoF/0SBsIpbpfcL10unBkBwE2Hncne6WgcQP4/olVxnJtqcuUCqQqlgn/6K7VppaaLhlj/YYCEk6OHi0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com; spf=pass smtp.mailfrom=lge.com; arc=none smtp.client-ip=156.147.51.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=lge.com Received: from unknown (HELO yjaykim-PowerEdge-T330.lge.net) (10.177.112.156) by 156.147.51.103 with ESMTP; 17 Sep 2026 03:34:37 +0900 X-Original-SENDERIP: 10.177.112.156 X-Original-MAILFROM: youngjun.park@lge.com From: Youngjun Park To: akpm@linux-foundation.org Cc: chrisl@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, yosry@kernel.org, joshua.hahnjy@gmail.com, taejoon.song@lge.com, her0gyugyu@gmail.com, lianux.mm@gmail.com Subject: [RFC PATCH v11 2/4] mm: swap: allocate swap slots from swap tiers Date: Thu, 17 Sep 2026 03:34:35 +0900 Message-Id: <20260916183437.2946306-3-youngjun.park@lge.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260916183437.2946306-1-youngjun.park@lge.com> References: <20260916183437.2946306-1-youngjun.park@lge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The slow allocation path walks swap_avail_head, one plist of every available swap device, ordered by priority. Give each tier its own list of available devices instead. The slow path now walks the tiers in order, and walks each tier's plist as before. __folio_throttle_swaprate() walks them the same way. swap_avail_lock still protects these lists. The tier list changes under it too, so the walk can go on after it drops the lock, as long as its tier is still there. This is the groundwork for the per-priority allocation queue series from Kairui and Lian [1]. That series can move onto the per-tier device lists one tier at a time. [1] https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-0-68d3d92557= 8c@gmail.com/ Assisted-by: Claude:claude-opus-5 Signed-off-by: Youngjun Park --- include/linux/swap.h | 2 +- mm/swap.h | 1 + mm/swap_tier.c | 17 +++++++++- mm/swap_tier.h | 3 ++ mm/swapfile.c | 78 ++++++++++++++++++++++++-------------------- 5 files changed, 64 insertions(+), 37 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 22ccb4b5801e..df69c2dd434a 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -268,7 +268,7 @@ struct swap_info_struct { struct work_struct discard_work; /* discard worker */ struct work_struct reclaim_work; /* reclaim worker */ struct list_head discard_clusters; /* discard clusters list */ - struct plist_node avail_list; /* entry in swap_avail_head */ + struct plist_node avail_list; /* entry in its tier's avail list */ const struct swap_ops *ops; }; =20 diff --git a/mm/swap.h b/mm/swap.h index 4de6b9b0f261..916c0ad128f7 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -37,6 +37,7 @@ struct swap_io_ctx; #endif =20 extern spinlock_t swap_lock; +extern spinlock_t swap_avail_lock; extern struct swap_info_struct *swap_info[]; =20 /* diff --git a/mm/swap_tier.c b/mm/swap_tier.c index 8ed1427cee09..286f319fb125 100644 --- a/mm/swap_tier.c +++ b/mm/swap_tier.c @@ -65,6 +65,7 @@ void swap_tiers_init(void) =20 for_each_tier(tier, idx) { plist_head_init(&tier->active_head); + plist_head_init(&tier->avail_head); INIT_LIST_HEAD(&tier->list); swap_tier_inactivate(tier); } @@ -92,11 +93,14 @@ void swap_tiers_assign_dev(struct swap_info_struct *swp) =20 lockdep_assert_held(&swap_lock); =20 + /* The allocator walks the tiers under swap_avail_lock. */ + spin_lock(&swap_avail_lock); tier =3D swap_tier_lookup(swp->prio); if (!tier) { tier =3D swap_tier_prepare(swp->prio); swap_tier_activate(tier); } + spin_unlock(&swap_avail_lock); =20 plist_add(&swp->list, &tier->active_head); } @@ -109,6 +113,17 @@ void swap_tiers_remove_dev(struct swap_info_struct *sw= p) =20 tier =3D swap_tier_lookup(swp->prio); plist_del(&swp->list, &tier->active_head); - if (plist_head_empty(&tier->active_head)) + if (plist_head_empty(&tier->active_head)) { + spin_lock(&swap_avail_lock); swap_tier_inactivate(tier); + spin_unlock(&swap_avail_lock); + } +} + +/* The avail list of the tier @swp belongs to. */ +struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp) +{ + lockdep_assert_held(&swap_avail_lock); + + return &swap_tier_lookup(swp->prio)->avail_head; } diff --git a/mm/swap_tier.h b/mm/swap_tier.h index 3dce716d23f6..c4347c28d8f1 100644 --- a/mm/swap_tier.h +++ b/mm/swap_tier.h @@ -14,11 +14,13 @@ struct swap_info_struct; * * @prio: priority of the swap devices in the tier. * @active_head: swap devices in the tier. + * @avail_head: available swap devices in the tier. * @list: linkage into swap_tier_active_list or swap_tier_inactive_list. */ struct swap_tier { short prio; struct plist_head active_head; + struct plist_head avail_head; struct list_head list; }; =20 @@ -33,5 +35,6 @@ void swap_tiers_init(void); /* Tier assignment */ void swap_tiers_assign_dev(struct swap_info_struct *swp); void swap_tiers_remove_dev(struct swap_info_struct *swp); +struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp); =20 #endif /* _SWAP_TIER_H */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 8201ae779833..e9d142c0655b 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -85,8 +85,8 @@ static const char Bad_file[] =3D "Bad swap file entry "; static const char Bad_offset[] =3D "Bad swap offset entry "; =20 /* - * all available (active, not full) swap_info_structs - * protected with swap_avail_lock, ordered by priority. + * all available (active, not full) swap_info_structs are on the + * avail list of their swap tier, protected with swap_avail_lock. * This is used by folio_alloc_swap() instead of the active lists of * the swap tiers because those include all swap_info_structs, * but folio_alloc_swap() doesn't need to look at full ones. @@ -96,8 +96,7 @@ static const char Bad_offset[] =3D "Bad swap offset entry= "; * is held and the locking order requires swap_lock to be taken * before any swap_info_struct->lock. */ -static PLIST_HEAD(swap_avail_head); -static DEFINE_SPINLOCK(swap_avail_lock); +DEFINE_SPINLOCK(swap_avail_lock); =20 struct swap_info_struct *swap_info[MAX_SWAPFILES]; =20 @@ -1226,7 +1225,7 @@ static void del_from_avail_list(struct swap_info_stru= ct *si, bool swapoff) goto skip; } =20 - plist_del(&si->avail_list, &swap_avail_head); + plist_del(&si->avail_list, swap_tiers_avail_head(si)); =20 skip: spin_unlock(&swap_avail_lock); @@ -1267,7 +1266,7 @@ static void add_to_avail_list(struct swap_info_struct= *si, bool swapon) goto skip; } =20 - plist_add(&si->avail_list, &swap_avail_head); + plist_add(&si->avail_list, swap_tiers_avail_head(si)); =20 skip: spin_unlock(&swap_avail_lock); @@ -1398,35 +1397,40 @@ static bool swap_alloc_fast(struct folio *folio) static void swap_alloc_slow(struct folio *folio) { struct swap_info_struct *si, *next; + struct swap_tier *tier; + short prio; =20 spin_lock(&swap_avail_lock); start_over: - plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) { - /* Rotate the device and switch to a new cluster */ - plist_requeue(&si->avail_list, &swap_avail_head); - spin_unlock(&swap_avail_lock); - if (get_swap_device_info(si)) { - cluster_alloc_swap_entry(si, folio); - put_swap_device(si); - if (folio_test_swapcache(folio)) - return; - if (folio_test_large(folio)) - return; - } + for_each_active_tier(tier) { + prio =3D tier->prio; + plist_for_each_entry_safe(si, next, &tier->avail_head, avail_list) { + /* Rotate the device and switch to a new cluster */ + plist_requeue(&si->avail_list, &tier->avail_head); + spin_unlock(&swap_avail_lock); + if (get_swap_device_info(si)) { + cluster_alloc_swap_entry(si, folio); + put_swap_device(si); + if (folio_test_swapcache(folio)) + return; + if (folio_test_large(folio)) + return; + } =20 - spin_lock(&swap_avail_lock); - /* - * if we got here, it's likely that si was almost full before, - * multiple callers probably all tried to get a page from the - * same si and it filled up before we could get one; or, the si - * filled up between us dropping swap_avail_lock. - * Since we dropped the swap_avail_lock, the swap_avail_list - * may have been modified; so if next is still in the - * swap_avail_head list then try it, otherwise start over if we - * have not gotten any slots. - */ - if (plist_node_empty(&next->avail_list)) - goto start_over; + spin_lock(&swap_avail_lock); + /* + * if we got here, it's likely that si was almost full before, + * multiple callers probably all tried to get a page from the + * same si and it filled up before we could get one; or, the si + * filled up between us dropping swap_avail_lock. + * Since we dropped the swap_avail_lock, the swap_avail_list + * may have been modified; so if next is still in the + * tier's avail list and the tier is still there then try it, + * otherwise start over if we have not gotten any slots. + */ + if (plist_node_empty(&next->avail_list) || tier->prio !=3D prio) + goto start_over; + } } spin_unlock(&swap_avail_lock); } @@ -4013,6 +4017,7 @@ static bool __has_usable_swap(void) void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp) { struct swap_info_struct *si; + struct swap_tier *tier; =20 if (!(gfp & __GFP_IO)) return; @@ -4031,12 +4036,15 @@ void __folio_throttle_swaprate(struct folio *folio,= gfp_t gfp) return; =20 spin_lock(&swap_avail_lock); - plist_for_each_entry(si, &swap_avail_head, avail_list) { - if (si->bdev) { - blkcg_schedule_throttle(si->bdev->bd_disk, true); - break; + for_each_active_tier(tier) { + plist_for_each_entry(si, &tier->avail_head, avail_list) { + if (si->bdev) { + blkcg_schedule_throttle(si->bdev->bd_disk, true); + goto out; + } } } +out: spin_unlock(&swap_avail_lock); } #endif --=20 2.48.1 From nobody Fri Sep 25 04:07:54 2026 Received: from lgeamrelo07.lge.com (lgeamrelo07.lge.com [156.147.51.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FC1251FCDF for ; Wed, 16 Sep 2026 18:34:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=156.147.51.103 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583694; cv=none; b=I/9hhS9o+gmIX/ARdBQJmafgQnGx2TuKlwftUqwxxAQzJ8dO1zcm2ERaz2+i/11Wk3AUle16jgsWrWJqwhr6QwPqBGl1wIZow5Xr6aQSyHc1AUX7rKXAT0LZ1zN93jX4LT7bO3RvB4UFtkBq/nGVG6fWw2fpmN1io6Lp8te87Rk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583694; c=relaxed/simple; bh=X61Dnjaw35FZr4d8vkWRgJwXha9DyIs+cJOK34/L4bo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BfKrw8kGMoEErFllvqAar9M09DgnW0P6bEwCQipT8wf4/Btkl87kT+VOzhk9sjXFnXC/NfgEPJI20npHEvsa0F80n9422EYhO5NbSspxDwR8NEYFrXsV48hZSJj+VZ9wKlBkK8N70iPXJ5jnZMF6/W29d6/es45b+yS5LvKWv00= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com; spf=pass smtp.mailfrom=lge.com; arc=none smtp.client-ip=156.147.51.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=lge.com Received: from unknown (HELO yjaykim-PowerEdge-T330.lge.net) (10.177.112.156) by 156.147.51.103 with ESMTP; 17 Sep 2026 03:34:37 +0900 X-Original-SENDERIP: 10.177.112.156 X-Original-MAILFROM: youngjun.park@lge.com From: Youngjun Park To: akpm@linux-foundation.org Cc: chrisl@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, yosry@kernel.org, joshua.hahnjy@gmail.com, taejoon.song@lge.com, her0gyugyu@gmail.com, lianux.mm@gmail.com Subject: [RFC PATCH v11 3/4] mm: swap: add a debugfs interface for memcg tier selection Date: Thu, 17 Sep 2026 03:34:36 +0900 Message-Id: <20260916183437.2946306-4-youngjun.park@lge.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260916183437.2946306-1-youngjun.park@lge.com> References: <20260916183437.2946306-1-youngjun.park@lge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Swap tiers do nothing until something chooses between them. Let a memory cgroup do that. Each cgroup carries a mask of the tiers it may swap to. Write a cgroup path and a hex mask to /sys/kernel/debug/swap/memcg_tiers to set it. Bit i is the tier at index i. /sys/kernel/debug/swap/tiers lists each tier's index and priority. # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers This lives in debugfs on purpose. A cgroup file is a permanent ABI, and what a swap tier should look like is not settled yet. It is built only with CONFIG_MEMCG and CONFIG_DEBUG_FS. The masks are kept in a list keyed by cgroup ID rather than in struct mem_cgroup, so memcg itself is not changed. A mask applies to the memory charged to its cgroup and is not inherited by child cgroups. The entry of a removed cgroup is dropped on the next write. Each swap device is stamped with its tier's bit at swapon. A device's tier never changes, so the stamp is written once and read locklessly by the allocator. When the last device of a tier is swapped off, the tier frees its index for reuse. It waits until swapoff can no longer fail, so a device whose swapoff fails goes back to the same index. The freed bit is set back in every cgroup mask, so a cgroup that had disabled that tier must disable it again once a new tier reuses the index. Assisted-by: Claude:claude-opus-5 Signed-off-by: Youngjun Park --- Documentation/mm/index.rst | 1 + Documentation/mm/swap-tier.rst | 55 ++++++ MAINTAINERS | 1 + include/linux/swap.h | 1 + mm/swap_tier.c | 294 ++++++++++++++++++++++++++++++++- mm/swap_tier.h | 31 ++++ mm/swapfile.c | 5 + 7 files changed, 387 insertions(+), 1 deletion(-) create mode 100644 Documentation/mm/swap-tier.rst diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst index 13a79f5d092c..6afc45cd4b3d 100644 --- a/Documentation/mm/index.rst +++ b/Documentation/mm/index.rst @@ -34,6 +34,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`. page_reclaim swap swap-table + swap-tier page_cache shmfs oom diff --git a/Documentation/mm/swap-tier.rst b/Documentation/mm/swap-tier.rst new file mode 100644 index 000000000000..4007c23f83a6 --- /dev/null +++ b/Documentation/mm/swap-tier.rst @@ -0,0 +1,55 @@ +.. SPDX-License-Identifier: GPL-2.0 + +:Author: Chris Li , + Youngjun Park + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Swap Tier +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Swap tier is a group of swap devices that share a priority. It acts as a +facilitation layer, allowing users to manage swap devices based on their +speeds. + +Users are encouraged to assign swap device priorities according to device +speed to fully utilize this feature. + +Tier Index +---------- + +A tier is created when the first swap device with its priority is swapped = on, +and removed when the last one is swapped off. Each tier is given an index = when +it is created and keeps it until it is removed, so a tier's index does not +change when another priority is swapped on or off. + +Per-cgroup Tier Selection +------------------------- + +A memory cgroup can be limited to some tiers through debugfs. This is for +evaluation, not a stable ABI. + +``/sys/kernel/debug/swap/tiers`` lists the index and priority of each tier. +``/sys/kernel/debug/swap/memcg_tiers`` takes a cgroup path and a mask in h= ex, +where bit ``i`` allows the tier at index ``i``:: + + # cat /sys/kernel/debug/swap/tiers + Idx Prio + 0 100 + 1 50 + # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers + +There is no separate delete operation. Writing a mask that allows every ti= er +clears the restriction, so the cgroup drops out of the file:: + + # echo "/batch 0xffffffff" > /sys/kernel/debug/swap/memcg_tiers + +A cgroup's mask is also dropped when the cgroup is removed. + +A mask applies to the memory charged to its own cgroup and is not inherite= d by +child cgroups. A tier keeps its index for its lifetime, so the same mask k= eeps +selecting the same tier across a swapon or swapoff. + +When a tier's last device is swapped off, its index is freed and can be re= used +by a later tier. The freed index is re-allowed in every cgroup mask, so a +cgroup that had disabled it must disable it again once a new tier reuses t= he +index. diff --git a/MAINTAINERS b/MAINTAINERS index 37f353015cae..af17b804f31b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17402,6 +17402,7 @@ L: linux-mm@kvack.org S: Maintained F: Documentation/ABI/testing/sysfs-kernel-mm-swap F: Documentation/mm/swap-table.rst +F: Documentation/mm/swap-tier.rst F: include/linux/swap.h F: include/linux/swap_ops.h F: include/linux/swapfile.h diff --git a/include/linux/swap.h b/include/linux/swap.h index df69c2dd434a..30d3c37ca530 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -237,6 +237,7 @@ struct swap_info_struct { struct percpu_ref users; /* indicate and keep swap device valid. */ unsigned long flags; /* SWP_USED etc: see above */ signed short prio; /* swap priority of this type */ + unsigned int tier_mask; /* swap tier mask */ struct plist_node list; /* entry in its swap tier */ signed char type; /* strange name for an index */ unsigned int max; /* size of this swap device */ diff --git a/mm/swap_tier.c b/mm/swap_tier.c index 286f319fb125..de183c2678e0 100644 --- a/mm/swap_tier.c +++ b/mm/swap_tier.c @@ -1,5 +1,10 @@ // SPDX-License-Identifier: GPL-2.0 #include +#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS) +#include +#include +#include +#endif =20 #include "swap.h" #include "swap_tier.h" @@ -17,6 +22,10 @@ static LIST_HEAD(swap_tier_inactive_list); for (idx =3D 0, tier =3D &swap_tiers[0]; idx < MAX_SWAPTIER; \ idx++, tier =3D &swap_tiers[idx]) =20 +/* A tier's index is its slot in the array, stable for its lifetime. */ +#define TIER_IDX(tier) ((tier) - swap_tiers) +#define TIER_MASK(tier) (1U << TIER_IDX(tier)) + /* * Naming Convention: * swap_tiers_*() - Public/exported functions @@ -56,6 +65,12 @@ static void swap_tier_inactivate(struct swap_tier *tier) list_move_tail(&tier->list, &swap_tier_inactive_list); } =20 +#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS) +static void swap_tiers_debugfs_init(void); +#else +static inline void swap_tiers_debugfs_init(void) {} +#endif + void swap_tiers_init(void) { struct swap_tier *tier; @@ -69,6 +84,8 @@ void swap_tiers_init(void) INIT_LIST_HEAD(&tier->list); swap_tier_inactivate(tier); } + + swap_tiers_debugfs_init(); } =20 static struct swap_tier *swap_tier_prepare(short prio) @@ -103,6 +120,17 @@ void swap_tiers_assign_dev(struct swap_info_struct *sw= p) spin_unlock(&swap_avail_lock); =20 plist_add(&swp->list, &tier->active_head); + + /* Put back by a failed swapoff, so already stamped and counted. */ + if (swp->tier_mask) + return; + + /* + * A device's tier never changes, so stamp it once here. Paired with + * the READ_ONCE() in the allocator, which reads this without swap_lock. + */ + tier->nr_devs++; + WRITE_ONCE(swp->tier_mask, TIER_MASK(tier)); } =20 void swap_tiers_remove_dev(struct swap_info_struct *swp) @@ -113,11 +141,31 @@ void swap_tiers_remove_dev(struct swap_info_struct *s= wp) =20 tier =3D swap_tier_lookup(swp->prio); plist_del(&swp->list, &tier->active_head); - if (plist_head_empty(&tier->active_head)) { +} + +/* + * A failed swapoff puts the device back into its tier, so the tier is giv= en + * up only here, once swapoff can no longer fail. Returns the tier's mask = if + * @swp was its last device, 0 otherwise. + */ +unsigned int swap_tiers_release_dev(struct swap_info_struct *swp) +{ + struct swap_tier *tier; + unsigned int freed =3D 0; + + lockdep_assert_held(&swap_lock); + + tier =3D swap_tier_lookup(swp->prio); + if (!--tier->nr_devs) { spin_lock(&swap_avail_lock); swap_tier_inactivate(tier); spin_unlock(&swap_avail_lock); + freed =3D TIER_MASK(tier); } + + WRITE_ONCE(swp->tier_mask, 0); + + return freed; } =20 /* The avail list of the tier @swp belongs to. */ @@ -127,3 +175,247 @@ struct plist_head *swap_tiers_avail_head(struct swap_= info_struct *swp) =20 return &swap_tier_lookup(swp->prio)->avail_head; } + +#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS) +static DEFINE_MUTEX(swap_tier_lock); + +/* + * struct swap_tier_cgroup - tier mask of a cgroup. + * + * @id: cgroup ID of the cgroup. + * @mask: tiers the cgroup may swap to. + * @list: linkage into swap_tier_cgroup_list. + * @rcu: frees the entry after a grace period. + */ +struct swap_tier_cgroup { + u64 id; + unsigned int mask; + struct list_head list; + struct rcu_head rcu; +}; + +/* + * Cgroups written to memcg_tiers. Changed under swap_tier_lock, walked by + * the allocator under RCU. A cgroup not on the list may use every tier. + */ +static LIST_HEAD(swap_tier_cgroup_list); + +static struct swap_tier_cgroup *swap_tier_cgroup_lookup(u64 id) +{ + struct swap_tier_cgroup *stc; + + list_for_each_entry_rcu(stc, &swap_tier_cgroup_list, list, + lockdep_is_held(&swap_tier_lock)) { + if (stc->id =3D=3D id) + return stc; + } + + return NULL; +} + +/* Drop the entries that allow every tier or whose cgroup is removed. */ +static void swap_tier_cgroup_prune(void) +{ + struct swap_tier_cgroup *stc, *tmp; + struct cgroup *cgrp; + + lockdep_assert_held(&swap_tier_lock); + + list_for_each_entry_safe(stc, tmp, &swap_tier_cgroup_list, list) { + if (stc->mask !=3D TIER_ALL_MASK) { + cgrp =3D __cgroup_get_from_id(stc->id); + if (!IS_ERR(cgrp)) { + cgroup_put(cgrp); + continue; + } + } + + list_del_rcu(&stc->list); + kfree_rcu(stc, rcu); + } +} + +/* One line per cgroup that dropped a tier, in the syntax a write takes. */ +static int swap_tiers_memcg_show(struct seq_file *m, void *v) +{ + struct swap_tier_cgroup *stc; + struct cgroup *cgrp; + char *path; + + path =3D kmalloc(PATH_MAX, GFP_KERNEL); + if (!path) + return -ENOMEM; + + mutex_lock(&swap_tier_lock); + list_for_each_entry(stc, &swap_tier_cgroup_list, list) { + if (stc->mask =3D=3D TIER_ALL_MASK) + continue; + + /* A removed cgroup's entry stays until the next write. */ + cgrp =3D __cgroup_get_from_id(stc->id); + if (IS_ERR(cgrp)) + continue; + + cgroup_path(cgrp, path, PATH_MAX); + cgroup_put(cgrp); + seq_printf(m, "%s 0x%x\n", path, stc->mask); + } + mutex_unlock(&swap_tier_lock); + + kfree(path); + return 0; +} + +/* + * Keep only the tiers set in @mask. @cgpath must name a cgroup that has + * the memory controller enabled. + */ +static int swap_tiers_memcg_set(const char *cgpath, unsigned int mask) +{ + struct swap_tier_cgroup *stc; + struct cgroup *cgrp; + bool enabled; + int ret =3D 0; + u64 id; + + cgrp =3D cgroup_get_from_path(cgpath); + if (IS_ERR(cgrp)) + return PTR_ERR(cgrp); + + /* + * Not cgroup_get_e_css(), which falls back to an ancestor when the + * memory controller is not enabled here. + */ + rcu_read_lock(); + enabled =3D cgroup_css(cgrp, &memory_cgrp_subsys); + rcu_read_unlock(); + id =3D cgroup_id(cgrp); + cgroup_put(cgrp); + + if (!enabled) + return -ENOENT; + + mutex_lock(&swap_tier_lock); + + stc =3D swap_tier_cgroup_lookup(id); + if (stc) { + WRITE_ONCE(stc->mask, mask); + } else if (mask !=3D TIER_ALL_MASK) { + stc =3D kmalloc_obj(*stc, GFP_KERNEL); + if (stc) { + stc->id =3D id; + stc->mask =3D mask; + list_add_rcu(&stc->list, &swap_tier_cgroup_list); + } else { + ret =3D -ENOMEM; + } + } + swap_tier_cgroup_prune(); + + mutex_unlock(&swap_tier_lock); + return ret; +} + +/* The tiers that the cgroup @folio is charged to may swap to. */ +unsigned int folio_tier_mask(struct folio *folio) +{ + struct swap_tier_cgroup *stc; + struct mem_cgroup *memcg; + unsigned int mask =3D TIER_ALL_MASK; + + rcu_read_lock(); + memcg =3D folio_memcg(folio); + if (memcg) { + stc =3D swap_tier_cgroup_lookup(cgroup_id(memcg->css.cgroup)); + if (stc) + mask =3D READ_ONCE(stc->mask); + } + rcu_read_unlock(); + + return mask; +} + +/* + * When a tier is removed, its index (bit position in the mask) becomes + * free for reassignment to a future tier. If a cgroup had previously + * disabled this tier (cleared the bit in its memcg_tiers entry), its mask + * would keep that bit clear, meaning the new tier at the same index would + * be silently unavailable, an invisible cgroup constraint left behind by a + * tier that no longer exists. + * + * To prevent this, OR the removed tier's mask bit into every cgroup's + * mask. This resets the bit so the new tier is accessible by default. + * Users who want to restrict it must explicitly disable it after the tier + * is re-created. + */ +void swap_tiers_memcg_propagate(unsigned int mask) +{ + struct swap_tier_cgroup *stc; + + mutex_lock(&swap_tier_lock); + list_for_each_entry(stc, &swap_tier_cgroup_list, list) + WRITE_ONCE(stc->mask, stc->mask | mask); + mutex_unlock(&swap_tier_lock); +} + +static int swap_tiers_memcg_open(struct inode *inode, struct file *file) +{ + return single_open(file, swap_tiers_memcg_show, NULL); +} + +static ssize_t swap_tiers_memcg_write(struct file *file, + const char __user *ubuf, + size_t count, loff_t *ppos) +{ + char *pos, *tmp, *cgpath; + unsigned int mask; + int ret; + + tmp =3D memdup_user_nul(ubuf, count); + if (IS_ERR(tmp)) + return PTR_ERR(tmp); + + pos =3D strstrip(tmp); + cgpath =3D strsep(&pos, " \t\n"); + if (!cgpath || !*cgpath || !pos || + kstrtouint(skip_spaces(pos), 16, &mask)) + ret =3D -EINVAL; + else + ret =3D swap_tiers_memcg_set(cgpath, mask); + + kfree(tmp); + return ret ? ret : count; +} + +static const struct file_operations swap_tiers_memcg_fops =3D { + .open =3D swap_tiers_memcg_open, + .read =3D seq_read, + .write =3D swap_tiers_memcg_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static int swap_tiers_show(struct seq_file *m, void *v) +{ + struct swap_tier *tier; + + seq_printf(m, "%-5s %s\n", "Idx", "Prio"); + + spin_lock(&swap_lock); + for_each_active_tier(tier) + seq_printf(m, "%-5td %d\n", TIER_IDX(tier), tier->prio); + spin_unlock(&swap_lock); + + return 0; +} +DEFINE_SHOW_ATTRIBUTE(swap_tiers); + +static void swap_tiers_debugfs_init(void) +{ + struct dentry *dir =3D debugfs_create_dir("swap", NULL); + + debugfs_create_file("tiers", 0400, dir, NULL, &swap_tiers_fops); + debugfs_create_file("memcg_tiers", 0600, dir, NULL, + &swap_tiers_memcg_fops); +} +#endif diff --git a/mm/swap_tier.h b/mm/swap_tier.h index c4347c28d8f1..9967cfbcc439 100644 --- a/mm/swap_tier.h +++ b/mm/swap_tier.h @@ -9,16 +9,20 @@ /* Forward declarations */ struct swap_info_struct; =20 +#define TIER_ALL_MASK (~0U) + /* * struct swap_tier - structure representing a swap tier. * * @prio: priority of the swap devices in the tier. + * @nr_devs: swap devices in the tier, including ones being swapped off. * @active_head: swap devices in the tier. * @avail_head: available swap devices in the tier. * @list: linkage into swap_tier_active_list or swap_tier_inactive_list. */ struct swap_tier { short prio; + int nr_devs; struct plist_head active_head; struct plist_head avail_head; struct list_head list; @@ -35,6 +39,33 @@ void swap_tiers_init(void); /* Tier assignment */ void swap_tiers_assign_dev(struct swap_info_struct *swp); void swap_tiers_remove_dev(struct swap_info_struct *swp); +unsigned int swap_tiers_release_dev(struct swap_info_struct *swp); struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp); =20 +/** + * swap_tiers_mask_test - test whether two tier masks overlap + * @tier_mask: mask to test, e.g. a swap device's tier bit + * @mask: mask to test against, e.g. a cgroup's mask + * + * Return: true if @tier_mask and @mask share at least one tier bit. + */ +static inline bool swap_tiers_mask_test(unsigned int tier_mask, + unsigned int mask) +{ + return tier_mask & mask; +} + +#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS) +/* Memcg related functions */ +void swap_tiers_memcg_propagate(unsigned int mask); +unsigned int folio_tier_mask(struct folio *folio); +#else +static inline void swap_tiers_memcg_propagate(unsigned int mask) {} + +static inline unsigned int folio_tier_mask(struct folio *folio) +{ + return TIER_ALL_MASK; +} +#endif + #endif /* _SWAP_TIER_H */ diff --git a/mm/swapfile.c b/mm/swapfile.c index e9d142c0655b..bb953dd33ca0 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3191,6 +3191,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, special= file) struct address_space *mapping; struct inode *inode; unsigned int maxpages; + unsigned int freed_tier; int err, found =3D 0; =20 if (!capable(CAP_SYS_ADMIN)) @@ -3290,7 +3291,11 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specia= lfile) p->max =3D 0; p->cluster_info =3D NULL; spin_unlock(&p->lock); + freed_tier =3D swap_tiers_release_dev(p); spin_unlock(&swap_lock); + /* Under swapon_mutex, so a swapon cannot reuse the index before this. */ + if (freed_tier) + swap_tiers_memcg_propagate(freed_tier); arch_swap_invalidate_area(p->type); zswap_swapoff(p->type); mutex_unlock(&swapon_mutex); --=20 2.48.1 From nobody Fri Sep 25 04:07:54 2026 Received: from lgeamrelo07.lge.com (lgeamrelo07.lge.com [156.147.51.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58A784DAFA3 for ; Wed, 16 Sep 2026 18:34:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=156.147.51.103 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583693; cv=none; b=WbxXodIooM1HrKet5vSvz+aoQix9+0hkZvroAp8xuE2xh0vGWzpD8RYdmhih5dCw/xVjQW/CdYlRObuyrhjWAsUuxBIydFWlk/wVr5GX4AOZu5CL6fXqXXAqRi8+dGl0osLOHktpL+hWcP4zDuKlTc4gfQ0nIZfRxk3dQAlj+sQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789583693; c=relaxed/simple; bh=VklEqvAOcw4APBVtRNV692SWORmxQoeXUBCSmhJNvbY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hKfx5IO9EYcyRVraZK5ba6c/yjBgSWPTaRIJs29mJXafPTGpdReKB/SZ+T+opuFjKYE/nsW798ZBWB81pfkufWdYpT4zGtvr5lvDmR3RNv5WquIw4bqTbUNQFJYyXXc89i1pxSFMwiBp2Z2oX5GKGaL3IyEiz3pMSWoS4fnnIBY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com; spf=pass smtp.mailfrom=lge.com; arc=none smtp.client-ip=156.147.51.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=lge.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=lge.com Received: from unknown (HELO yjaykim-PowerEdge-T330.lge.net) (10.177.112.156) by 156.147.51.103 with ESMTP; 17 Sep 2026 03:34:37 +0900 X-Original-SENDERIP: 10.177.112.156 X-Original-MAILFROM: youngjun.park@lge.com From: Youngjun Park To: akpm@linux-foundation.org Cc: chrisl@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kasong@tencent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, shikemeng@huaweicloud.com, baoquan.he@linux.dev, baohua@kernel.org, yosry@kernel.org, joshua.hahnjy@gmail.com, taejoon.song@lge.com, her0gyugyu@gmail.com, lianux.mm@gmail.com Subject: [RFC PATCH v11 4/4] mm: swap: filter swap allocation by memcg tier mask Date: Thu, 17 Sep 2026 03:34:37 +0900 Message-Id: <20260916183437.2946306-5-youngjun.park@lge.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260916183437.2946306-1-youngjun.park@lge.com> References: <20260916183437.2946306-1-youngjun.park@lge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Apply the cgroup tier mask during swap slot allocation to enforce per-cgroup swap tier restrictions. The folio's mask is looked up once and passed to the fast, slow and discard paths as a parameter, so all of them act on the same mask even if the cgroup's mask changes concurrently. The device tier_mask is read with READ_ONCE() in the two paths that do not hold swap_lock, matching how si->flags is read in the same allocator. In the fast path, check the percpu cached swap_info's tier_mask against the folio's mask. If it does not match, fall through to the slow path. In the slow path, skip swap devices whose tier_mask is not covered by the folio's mask. The discard fallback honors the mask too. Without it, a discard on a device outside the folio's tiers still returns true and drives the retry, so the allocation spins through the loop consuming another device's discard queue while it cannot succeed. This works correctly when there is only one non-rotational device in the system and no devices share the same priority. However, there are known limitations. - When non-rotational devices are distributed across multiple tiers, and different memcgs are configured to use those distinct tiers, they may constantly overwrite the shared percpu swap cache. This cache thrashing leads to frequent fast path misses. - Combined with the above issue, if same-priority devices exist among them, a percpu cache miss (overwritten by another memcg) forces the allocator to round-robin to the next device prematurely, even if the current cluster is not fully exhausted. These edge cases do not affect the primary use case of directing swap traffic per cgroup. Further optimization is planned for future work. Signed-off-by: Youngjun Park --- mm/swapfile.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index bb953dd33ca0..b246eff25c96 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1364,7 +1364,7 @@ static bool get_swap_device_info(struct swap_info_str= uct *si) * Fast path try to get swap entries with specified order from current * CPU's swap entry pool (a cluster). */ -static bool swap_alloc_fast(struct folio *folio) +static bool swap_alloc_fast(struct folio *folio, unsigned int mask) { unsigned int order =3D folio_order(folio); struct swap_cluster_info *ci; @@ -1376,8 +1376,11 @@ static bool swap_alloc_fast(struct folio *folio) * so checking it's liveness by get_swap_device_info is enough. */ si =3D this_cpu_read(percpu_swap_cluster.si[order]); + if (!si || !swap_tiers_mask_test(READ_ONCE(si->tier_mask), mask)) + return false; + offset =3D this_cpu_read(percpu_swap_cluster.offset[order]); - if (!si || !offset || !get_swap_device_info(si)) + if (!offset || !get_swap_device_info(si)) return false; =20 ci =3D swap_cluster_lock(si, offset); @@ -1394,7 +1397,7 @@ static bool swap_alloc_fast(struct folio *folio) } =20 /* Rotate the device and switch to a new cluster */ -static void swap_alloc_slow(struct folio *folio) +static void swap_alloc_slow(struct folio *folio, unsigned int mask) { struct swap_info_struct *si, *next; struct swap_tier *tier; @@ -1405,6 +1408,9 @@ static void swap_alloc_slow(struct folio *folio) for_each_active_tier(tier) { prio =3D tier->prio; plist_for_each_entry_safe(si, next, &tier->avail_head, avail_list) { + if (!swap_tiers_mask_test(READ_ONCE(si->tier_mask), mask)) + continue; + /* Rotate the device and switch to a new cluster */ plist_requeue(&si->avail_list, &tier->avail_head); spin_unlock(&swap_avail_lock); @@ -1439,7 +1445,7 @@ static void swap_alloc_slow(struct folio *folio) * Discard pending clusters in a synchronized way when under high pressure. * Return: true if any cluster is discarded. */ -static bool swap_sync_discard(void) +static bool swap_sync_discard(unsigned int mask) { bool ret =3D false; struct swap_info_struct *si, *next; @@ -1451,6 +1457,8 @@ static bool swap_sync_discard(void) for_each_active_tier(tier) { prio =3D tier->prio; plist_for_each_entry_safe(si, next, &tier->active_head, list) { + if (!swap_tiers_mask_test(si->tier_mask, mask)) + continue; spin_unlock(&swap_lock); if (get_swap_device_info(si)) { if (si->flags & SWP_PAGE_DISCARD) @@ -1749,6 +1757,7 @@ int folio_alloc_swap(struct folio *folio) { unsigned int order =3D folio_order(folio); unsigned int size =3D 1 << order; + unsigned int mask; =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio); @@ -1772,13 +1781,14 @@ int folio_alloc_swap(struct folio *folio) } =20 again: + mask =3D folio_tier_mask(folio); local_lock(&percpu_swap_cluster.lock); - if (!swap_alloc_fast(folio)) - swap_alloc_slow(folio); + if (!swap_alloc_fast(folio, mask)) + swap_alloc_slow(folio, mask); local_unlock(&percpu_swap_cluster.lock); =20 if (!order && unlikely(!folio_test_swapcache(folio))) { - if (swap_sync_discard()) + if (swap_sync_discard(mask)) goto again; } =20 --=20 2.48.1