From nobody Fri Feb 13 15:02:08 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85D401304A1 for ; Fri, 24 May 2024 17:17:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716571052; cv=none; b=t3eRPQ+IkPpQYBE6QHQoAOzucZAMpY4mdgyweF99EdPkrmXsBTHknW5MKMhyPp0dQzoGIZivUeSJc4JcIzAG86zpTyKDz/dTRChAMNZ4g/Zc8jsIHdiAejEcAp4djPaEaVk0+ie6o3L64Zxr17/tHLYLG/NFgI06lfgESSMpbR8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716571052; c=relaxed/simple; bh=18sjjtIWlFD+xHMhWe5DbZFZTqO9Q8lFviFh49UTynw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=dRE9uerx6NmMCip+fCoqNO/l4qc1bQAo7EIu+pQjRtPUZdou5hZb7liEZtnL/ndOOKATVNa+atkFOCMOO6ip1uMI+VGiU903Pei5Stf91Myo6YJHYfba5AT4f8BK2g8HM7IXzgdL+9ktg1Y0kwEaaFN2AXvUGG/LrFGc+b2rdsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=s76zRULz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="s76zRULz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40BF8C2BD11; Fri, 24 May 2024 17:17:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716571050; bh=18sjjtIWlFD+xHMhWe5DbZFZTqO9Q8lFviFh49UTynw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=s76zRULziD5wYoQqKOk6AhYoUOz1EavyGUcvrOHthwUWeBgDyFPDuH8uOavf0UpIq kFby15VpfFq76T8SKfTh1RUZpR7tOr7hcygpmKVFUBQS9PS42V91B/+ldEDEmHDkf6 MaiZbnRzywgMip1OtbzFKQq4RaS1FOp5feAbspEpFv/HMPAHE8wzv2TwpHQXIP6qLY CCn2nbjItxWYc7NPEqmYyQEhSGiZDHalCfon+2iRUq+L5tTAvH/jhaGO7ejXHp3e+9 7bBNDvKIldTpC1XKEaVjK1wOVsN1yM5pVrJcdtbNTusLRMd5Y5lslgqjQsQpH8YfBw vLW0/oQCOJgzw== From: Chris Li Date: Fri, 24 May 2024 10:17:18 -0700 Subject: [PATCH 1/2] mm: swap: swap cluster switch to double link list Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240524-swap-allocator-v1-1-47861b423b26@kernel.org> References: <20240524-swap-allocator-v1-0-47861b423b26@kernel.org> In-Reply-To: <20240524-swap-allocator-v1-0-47861b423b26@kernel.org> To: Andrew Morton Cc: Kairui Song , Ryan Roberts , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.12.4 Previously, the swap cluster used a cluster index as a pointer to construct a custom single link list type "swap_cluster_list". The next cluster pointer is shared with the cluster->count. The assumption is that only the free cluster needs to be put on the list. That assumption is not true for mTHP allocators any more. Need to track the non full cluster on the list as well. =C2=A0Move the current cluster single link list into standard double link list. Remove the cluster getter/setter for accessing the cluster struct member. =C2=A0Move the cluster locking in the caller function rather than the getter/setter function. That way the locking can protect more than one member, e.g. cluster->flag. Change cluster code to use "struct swap_cluster_info *" to reference the cluster rather than by using index. That is more consistent with the list manipulation. It avoids the repeat adding index to the cluser_info. The code is easier to understand. Remove the cluster next pointer is NULL flag, the double link list can handle the empty list pretty well. The "swap_cluster_info" struct is two pointer bigger, because 512 swap entries share one swap struct, it has very little impact on the average memory usage per swap entry. =C2=A0Other than the list conversion, there is no real function change in this patch. Reported-by: Barry Song <21cnbao@gmail.com> Tested-by: Kairui Song --- include/linux/swap.h | 14 ++-- mm/swapfile.c | 231 ++++++++++++++---------------------------------= ---- 2 files changed, 68 insertions(+), 177 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..0d3906eff3c9 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -254,11 +254,12 @@ struct swap_cluster_info { * elements correspond to the swap * cluster */ - unsigned int data:24; + unsigned int count:16; unsigned int flags:8; + struct list_head next; }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ -#define CLUSTER_FLAG_NEXT_NULL 2 /* This cluster has no next cluster */ + =20 /* * The first page in the swap file is the swap header, which is always mar= ked @@ -283,11 +284,6 @@ struct percpu_cluster { unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */ }; =20 -struct swap_cluster_list { - struct swap_cluster_info head; - struct swap_cluster_info tail; -}; - /* * The in-memory structure used to track swap areas. */ @@ -300,7 +296,7 @@ struct swap_info_struct { unsigned int max; /* extent of the swap_map */ unsigned char *swap_map; /* vmalloc'ed array of usage counts */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ - struct swap_cluster_list free_clusters; /* free clusters list */ + struct list_head free_clusters; /* free clusters list */ unsigned int lowest_bit; /* index of first free in swap_map */ unsigned int highest_bit; /* index of last free in swap_map */ unsigned int pages; /* total of usable pages of swap */ @@ -333,7 +329,7 @@ struct swap_info_struct { * list. */ struct work_struct discard_work; /* discard worker */ - struct swap_cluster_list discard_clusters; /* discard clusters list */ + struct list_head discard_clusters; /* discard clusters list */ struct plist_node avail_lists[]; /* * entries in swap_avail_heads, one * entry per node. diff --git a/mm/swapfile.c b/mm/swapfile.c index 4f0e8b2ac8aa..205a60c5f9cb 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -290,64 +290,11 @@ static void discard_swap_cluster(struct swap_info_str= uct *si, #endif #define LATENCY_LIMIT 256 =20 -static inline void cluster_set_flag(struct swap_cluster_info *info, - unsigned int flag) -{ - info->flags =3D flag; -} - -static inline unsigned int cluster_count(struct swap_cluster_info *info) -{ - return info->data; -} - -static inline void cluster_set_count(struct swap_cluster_info *info, - unsigned int c) -{ - info->data =3D c; -} - -static inline void cluster_set_count_flag(struct swap_cluster_info *info, - unsigned int c, unsigned int f) -{ - info->flags =3D f; - info->data =3D c; -} - -static inline unsigned int cluster_next(struct swap_cluster_info *info) -{ - return info->data; -} - -static inline void cluster_set_next(struct swap_cluster_info *info, - unsigned int n) -{ - info->data =3D n; -} - -static inline void cluster_set_next_flag(struct swap_cluster_info *info, - unsigned int n, unsigned int f) -{ - info->flags =3D f; - info->data =3D n; -} - static inline bool cluster_is_free(struct swap_cluster_info *info) { return info->flags & CLUSTER_FLAG_FREE; } =20 -static inline bool cluster_is_null(struct swap_cluster_info *info) -{ - return info->flags & CLUSTER_FLAG_NEXT_NULL; -} - -static inline void cluster_set_null(struct swap_cluster_info *info) -{ - info->flags =3D CLUSTER_FLAG_NEXT_NULL; - info->data =3D 0; -} - static inline struct swap_cluster_info *lock_cluster(struct swap_info_stru= ct *si, unsigned long offset) { @@ -394,65 +341,11 @@ static inline void unlock_cluster_or_swap_info(struct= swap_info_struct *si, spin_unlock(&si->lock); } =20 -static inline bool cluster_list_empty(struct swap_cluster_list *list) -{ - return cluster_is_null(&list->head); -} - -static inline unsigned int cluster_list_first(struct swap_cluster_list *li= st) -{ - return cluster_next(&list->head); -} - -static void cluster_list_init(struct swap_cluster_list *list) -{ - cluster_set_null(&list->head); - cluster_set_null(&list->tail); -} - -static void cluster_list_add_tail(struct swap_cluster_list *list, - struct swap_cluster_info *ci, - unsigned int idx) -{ - if (cluster_list_empty(list)) { - cluster_set_next_flag(&list->head, idx, 0); - cluster_set_next_flag(&list->tail, idx, 0); - } else { - struct swap_cluster_info *ci_tail; - unsigned int tail =3D cluster_next(&list->tail); - - /* - * Nested cluster lock, but both cluster locks are - * only acquired when we held swap_info_struct->lock - */ - ci_tail =3D ci + tail; - spin_lock_nested(&ci_tail->lock, SINGLE_DEPTH_NESTING); - cluster_set_next(ci_tail, idx); - spin_unlock(&ci_tail->lock); - cluster_set_next_flag(&list->tail, idx, 0); - } -} - -static unsigned int cluster_list_del_first(struct swap_cluster_list *list, - struct swap_cluster_info *ci) -{ - unsigned int idx; - - idx =3D cluster_next(&list->head); - if (cluster_next(&list->tail) =3D=3D idx) { - cluster_set_null(&list->head); - cluster_set_null(&list->tail); - } else - cluster_set_next_flag(&list->head, - cluster_next(&ci[idx]), 0); - - return idx; -} - /* Add a cluster to discard list and schedule it to do discard */ static void swap_cluster_schedule_discard(struct swap_info_struct *si, - unsigned int idx) + struct swap_cluster_info *ci) { + unsigned int idx =3D ci - si->cluster_info; /* * If scan_swap_map_slots() can't find a free cluster, it will check * si->swap_map directly. To make sure the discarding cluster isn't @@ -462,17 +355,16 @@ static void swap_cluster_schedule_discard(struct swap= _info_struct *si, memset(si->swap_map + idx * SWAPFILE_CLUSTER, SWAP_MAP_BAD, SWAPFILE_CLUSTER); =20 - cluster_list_add_tail(&si->discard_clusters, si->cluster_info, idx); - + spin_lock_nested(&ci->lock, SINGLE_DEPTH_NESTING); + list_add_tail(&ci->next, &si->discard_clusters); + spin_unlock(&ci->lock); schedule_work(&si->discard_work); } =20 -static void __free_cluster(struct swap_info_struct *si, unsigned long idx) +static void __free_cluster(struct swap_info_struct *si, struct swap_cluste= r_info *ci) { - struct swap_cluster_info *ci =3D si->cluster_info; - - cluster_set_flag(ci + idx, CLUSTER_FLAG_FREE); - cluster_list_add_tail(&si->free_clusters, ci, idx); + ci->flags =3D CLUSTER_FLAG_FREE; + list_add_tail(&ci->next, &si->free_clusters); } =20 /* @@ -481,21 +373,21 @@ static void __free_cluster(struct swap_info_struct *s= i, unsigned long idx) */ static void swap_do_scheduled_discard(struct swap_info_struct *si) { - struct swap_cluster_info *info, *ci; + struct swap_cluster_info *ci; unsigned int idx; =20 - info =3D si->cluster_info; - - while (!cluster_list_empty(&si->discard_clusters)) { - idx =3D cluster_list_del_first(&si->discard_clusters, info); + while (!list_empty(&si->discard_clusters)) { + ci =3D list_first_entry(&si->discard_clusters, struct swap_cluster_info,= next); + idx =3D ci - si->cluster_info; spin_unlock(&si->lock); =20 discard_swap_cluster(si, idx * SWAPFILE_CLUSTER, SWAPFILE_CLUSTER); =20 spin_lock(&si->lock); - ci =3D lock_cluster(si, idx * SWAPFILE_CLUSTER); - __free_cluster(si, idx); + + spin_lock(&ci->lock); + __free_cluster(si, ci); memset(si->swap_map + idx * SWAPFILE_CLUSTER, 0, SWAPFILE_CLUSTER); unlock_cluster(ci); @@ -521,20 +413,20 @@ static void swap_users_ref_free(struct percpu_ref *re= f) complete(&si->comp); } =20 -static void alloc_cluster(struct swap_info_struct *si, unsigned long idx) +static struct swap_cluster_info *alloc_cluster(struct swap_info_struct *si= , unsigned long idx) { - struct swap_cluster_info *ci =3D si->cluster_info; + struct swap_cluster_info *ci =3D list_first_entry(&si->free_clusters, str= uct swap_cluster_info, next); =20 - VM_BUG_ON(cluster_list_first(&si->free_clusters) !=3D idx); - cluster_list_del_first(&si->free_clusters, ci); - cluster_set_count_flag(ci + idx, 0, 0); + VM_BUG_ON(ci - si->cluster_info !=3D idx); + list_del(&ci->next); + ci->count =3D 0; + ci->flags =3D 0; + return ci; } =20 -static void free_cluster(struct swap_info_struct *si, unsigned long idx) +static void free_cluster(struct swap_info_struct *si, struct swap_cluster_= info *ci) { - struct swap_cluster_info *ci =3D si->cluster_info + idx; - - VM_BUG_ON(cluster_count(ci) !=3D 0); + VM_BUG_ON(ci->count !=3D 0); /* * If the swap is discardable, prepare discard the cluster * instead of free it immediately. The cluster will be freed @@ -542,11 +434,11 @@ static void free_cluster(struct swap_info_struct *si,= unsigned long idx) */ if ((si->flags & (SWP_WRITEOK | SWP_PAGE_DISCARD)) =3D=3D (SWP_WRITEOK | SWP_PAGE_DISCARD)) { - swap_cluster_schedule_discard(si, idx); + swap_cluster_schedule_discard(si, ci); return; } =20 - __free_cluster(si, idx); + __free_cluster(si, ci); } =20 /* @@ -559,15 +451,15 @@ static void add_cluster_info_page(struct swap_info_st= ruct *p, unsigned long count) { unsigned long idx =3D page_nr / SWAPFILE_CLUSTER; + struct swap_cluster_info *ci =3D cluster_info + idx; =20 if (!cluster_info) return; - if (cluster_is_free(&cluster_info[idx])) + if (cluster_is_free(ci)) alloc_cluster(p, idx); =20 - VM_BUG_ON(cluster_count(&cluster_info[idx]) + count > SWAPFILE_CLUSTER); - cluster_set_count(&cluster_info[idx], - cluster_count(&cluster_info[idx]) + count); + VM_BUG_ON(ci->count + count > SWAPFILE_CLUSTER); + ci->count +=3D count; } =20 /* @@ -581,24 +473,20 @@ static void inc_cluster_info_page(struct swap_info_st= ruct *p, } =20 /* - * The cluster corresponding to page_nr decreases one usage. If the usage - * counter becomes 0, which means no page in the cluster is in using, we c= an - * optionally discard the cluster and add it to free cluster list. + * The cluster ci decreases one usage. If the usage counter becomes 0, + * which means no page in the cluster is in using, we can optionally disca= rd + * the cluster and add it to free cluster list. */ -static void dec_cluster_info_page(struct swap_info_struct *p, - struct swap_cluster_info *cluster_info, unsigned long page_nr) +static void dec_cluster_info_page(struct swap_info_struct *p, struct swap_= cluster_info *ci) { - unsigned long idx =3D page_nr / SWAPFILE_CLUSTER; - - if (!cluster_info) + if (!p->cluster_info) return; =20 - VM_BUG_ON(cluster_count(&cluster_info[idx]) =3D=3D 0); - cluster_set_count(&cluster_info[idx], - cluster_count(&cluster_info[idx]) - 1); + VM_BUG_ON(ci->count =3D=3D 0); + ci->count--; =20 - if (cluster_count(&cluster_info[idx]) =3D=3D 0) - free_cluster(p, idx); + if (!ci->count) + free_cluster(p, ci); } =20 /* @@ -611,10 +499,10 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_s= truct *si, { struct percpu_cluster *percpu_cluster; bool conflict; - + struct swap_cluster_info *first =3D list_first_entry(&si->free_clusters, = struct swap_cluster_info, next); offset /=3D SWAPFILE_CLUSTER; - conflict =3D !cluster_list_empty(&si->free_clusters) && - offset !=3D cluster_list_first(&si->free_clusters) && + conflict =3D !list_empty(&si->free_clusters) && + offset !=3D first - si->cluster_info && cluster_is_free(&si->cluster_info[offset]); =20 if (!conflict) @@ -655,10 +543,14 @@ static bool scan_swap_map_try_ssd_cluster(struct swap= _info_struct *si, cluster =3D this_cpu_ptr(si->percpu_cluster); tmp =3D cluster->next[order]; if (tmp =3D=3D SWAP_NEXT_INVALID) { - if (!cluster_list_empty(&si->free_clusters)) { - tmp =3D cluster_next(&si->free_clusters.head) * - SWAPFILE_CLUSTER; - } else if (!cluster_list_empty(&si->discard_clusters)) { + if (!list_empty(&si->free_clusters)) { + ci =3D list_first_entry(&si->free_clusters, struct swap_cluster_info, n= ext); + list_del(&ci->next); + spin_lock(&ci->lock); + ci->flags =3D 0; + spin_unlock(&ci->lock); + tmp =3D (ci - si->cluster_info) * SWAPFILE_CLUSTER; + } else if (!list_empty(&si->discard_clusters)) { /* * we don't have free cluster but have some clusters in * discarding, do discard now and reclaim them, then @@ -670,7 +562,8 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_i= nfo_struct *si, goto new_cluster; } else return false; - } + } else + ci =3D si->cluster_info + tmp; =20 /* * Other CPUs can use our cluster if they can't find a free cluster, @@ -1062,8 +955,9 @@ static void swap_free_cluster(struct swap_info_struct = *si, unsigned long idx) =20 ci =3D lock_cluster(si, offset); memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); - cluster_set_count_flag(ci, 0, 0); - free_cluster(si, idx); + ci->count =3D 0; + ci->flags =3D 0; + free_cluster(si, ci); unlock_cluster(ci); swap_range_free(si, offset, SWAPFILE_CLUSTER); } @@ -1336,7 +1230,7 @@ static void swap_entry_free(struct swap_info_struct *= p, swp_entry_t entry) count =3D p->swap_map[offset]; VM_BUG_ON(count !=3D SWAP_HAS_CACHE); p->swap_map[offset] =3D 0; - dec_cluster_info_page(p, p->cluster_info, offset); + dec_cluster_info_page(p, ci); unlock_cluster(ci); =20 mem_cgroup_uncharge_swap(entry, 1); @@ -2985,8 +2879,8 @@ static int setup_swap_map_and_extents(struct swap_inf= o_struct *p, =20 nr_good_pages =3D maxpages - 1; /* omit header page */ =20 - cluster_list_init(&p->free_clusters); - cluster_list_init(&p->discard_clusters); + INIT_LIST_HEAD(&p->free_clusters); + INIT_LIST_HEAD(&p->discard_clusters); =20 for (i =3D 0; i < swap_header->info.nr_badpages; i++) { unsigned int page_nr =3D swap_header->info.badpages[i]; @@ -3037,14 +2931,15 @@ static int setup_swap_map_and_extents(struct swap_i= nfo_struct *p, for (k =3D 0; k < SWAP_CLUSTER_COLS; k++) { j =3D (k + col) % SWAP_CLUSTER_COLS; for (i =3D 0; i < DIV_ROUND_UP(nr_clusters, SWAP_CLUSTER_COLS); i++) { + struct swap_cluster_info *ci; idx =3D i * SWAP_CLUSTER_COLS + j; + ci =3D cluster_info + idx; if (idx >=3D nr_clusters) continue; - if (cluster_count(&cluster_info[idx])) + if (ci->count) continue; - cluster_set_flag(&cluster_info[idx], CLUSTER_FLAG_FREE); - cluster_list_add_tail(&p->free_clusters, cluster_info, - idx); + ci->flags =3D CLUSTER_FLAG_FREE; + list_add_tail(&ci->next, &p->free_clusters); } } return nr_extents; --=20 2.45.1.288.g0e0cd299f1-goog From nobody Fri Feb 13 15:02:08 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85D8E1304A3 for ; Fri, 24 May 2024 17:17:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716571052; cv=none; b=AL7G6yT8VyzrwAjCcuPfNDj8ELsDD+Mi5b+A05oOp3CAc+ELhhr0tWcK5wnFRThjqTCQMvwCsK/v6txJm1iNC19UQ0B9rOcZrxR0YyosmDkmRpMC872qqRoSrhEkcmkhpXW8GptlX4iWs7Wnme0RPgygH+9WhWaRA/J+L7dAeqk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716571052; c=relaxed/simple; bh=TDa3pqk4OBaY1GRmVEGGKzMmBUluuroNy+ytnClUbpo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=FfUS6Pi/brcyPiS1/MFf7Ow83K7+f0VxNZzfJ8cb/RlyKMrot2wF1Nn+QDanV9J41vM4RrwEEFzdfC95mCf6mKGD5mR8wUlEns4seTIE7VCp9wTep0aM9/jOr1qBP50GuiWlWnC0UXfHhALkQrCkEFweFtPtWzznNUuu7eZ1Vz0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UX/P+vhE; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UX/P+vhE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5A68C4AF08; Fri, 24 May 2024 17:17:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716571051; bh=TDa3pqk4OBaY1GRmVEGGKzMmBUluuroNy+ytnClUbpo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=UX/P+vhEw9dIOFUzn2jQQTmq6HFKnwx6c0OjKQ63wRYEZeLU3Ri5wEKCzRoIbEM7q dQnyk5EqxLW3Cd7F/X0ksDLWKKL+Qiob2T13x/XAMNa1YH9EY1z43EZhtaZzlI5W2Q jKlG8D+NPX+lhGb33tNdV8rMx+mGKVBSGgBtEW+HKb/SQfBLFr6sWKYYeQdGAK2g3T jTPsz3gQ5xDWrQVNnHQDcPOrGwU6GjkstNSeRGoCh7uF0Ag8W+EjGkX8uBdfn/Xky4 6QeggWr4n83ARyedUCY5a3yWcXWXwLB1Yb8PVBX7SXyRHS7p9gUq0TvCUA4GMV3Cqj jvov4OikiN/kw== From: Chris Li Date: Fri, 24 May 2024 10:17:19 -0700 Subject: [PATCH 2/2] mm: swap: mTHP allocate swap entries from nonfull list Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20240524-swap-allocator-v1-2-47861b423b26@kernel.org> References: <20240524-swap-allocator-v1-0-47861b423b26@kernel.org> In-Reply-To: <20240524-swap-allocator-v1-0-47861b423b26@kernel.org> To: Andrew Morton Cc: Kairui Song , Ryan Roberts , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.12.4 Track the nonfull cluster as well as the empty cluster on lists. Each order has one nonfull cluster list. The cluster will remember which order it was used during new cluster allocation. When the cluster has free entry, add to the nonfull[order] list. =C2=A0When the free cluster list is empty, also allocate from the nonempty list of that order. This improves the mTHP swap allocation success rate. There are limitations if the distribution of numbers of different orders of mTHP changes a lot. e.g. there are a lot of nonfull cluster assign to order A while later time there are a lot of order B allocation while very little allocation in order A. Currently the cluster used by order A will not reused by order B unless the cluster is 100% empty. This situation is best addressed by the longer term "swap buddy allocator", in future patches. Reported-by: Barry Song <21cnbao@gmail.com> Tested-by: Kairui Song --- include/linux/swap.h | 4 ++++ mm/swapfile.c | 25 +++++++++++++++++++++++-- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 0d3906eff3c9..1b7f0794b9bf 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -255,10 +255,12 @@ struct swap_cluster_info { * cluster */ unsigned int count:16; + unsigned int order:8; unsigned int flags:8; struct list_head next; }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ +#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */ =20 =20 /* @@ -297,6 +299,8 @@ struct swap_info_struct { unsigned char *swap_map; /* vmalloc'ed array of usage counts */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ struct list_head free_clusters; /* free clusters list */ + struct list_head nonfull_clusters[SWAP_NR_ORDERS]; + /* list of cluster that contains at least one free slot */ unsigned int lowest_bit; /* index of first free in swap_map */ unsigned int highest_bit; /* index of last free in swap_map */ unsigned int pages; /* total of usable pages of swap */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 205a60c5f9cb..51923aba500e 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -363,8 +363,11 @@ static void swap_cluster_schedule_discard(struct swap_= info_struct *si, =20 static void __free_cluster(struct swap_info_struct *si, struct swap_cluste= r_info *ci) { + if (ci->flags & CLUSTER_FLAG_NONFULL) + list_move_tail(&ci->next, &si->free_clusters); + else + list_add_tail(&ci->next, &si->free_clusters); ci->flags =3D CLUSTER_FLAG_FREE; - list_add_tail(&ci->next, &si->free_clusters); } =20 /* @@ -486,7 +489,12 @@ static void dec_cluster_info_page(struct swap_info_str= uct *p, struct swap_cluste ci->count--; =20 if (!ci->count) - free_cluster(p, ci); + return free_cluster(p, ci); + + if (!(ci->flags & CLUSTER_FLAG_NONFULL)) { + list_add_tail(&ci->next, &p->nonfull_clusters[ci->order]); + ci->flags |=3D CLUSTER_FLAG_NONFULL; + } } =20 /* @@ -547,6 +555,14 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_= info_struct *si, ci =3D list_first_entry(&si->free_clusters, struct swap_cluster_info, n= ext); list_del(&ci->next); spin_lock(&ci->lock); + ci->order =3D order; + ci->flags =3D 0; + spin_unlock(&ci->lock); + tmp =3D (ci - si->cluster_info) * SWAPFILE_CLUSTER; + } else if (!list_empty(&si->nonfull_clusters[order])) { + ci =3D list_first_entry(&si->nonfull_clusters[order], struct swap_clust= er_info, next); + list_del(&ci->next); + spin_lock(&ci->lock); ci->flags =3D 0; spin_unlock(&ci->lock); tmp =3D (ci - si->cluster_info) * SWAPFILE_CLUSTER; @@ -578,6 +594,7 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_i= nfo_struct *si, break; tmp +=3D nr_pages; } + WARN_ONCE(ci->order !=3D order, "expecting order %d got %d", order, ci->= order); unlock_cluster(ci); } if (tmp >=3D max) { @@ -956,6 +973,7 @@ static void swap_free_cluster(struct swap_info_struct *= si, unsigned long idx) ci =3D lock_cluster(si, offset); memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); ci->count =3D 0; + ci->order =3D 0; ci->flags =3D 0; free_cluster(si, ci); unlock_cluster(ci); @@ -2882,6 +2900,9 @@ static int setup_swap_map_and_extents(struct swap_inf= o_struct *p, INIT_LIST_HEAD(&p->free_clusters); INIT_LIST_HEAD(&p->discard_clusters); =20 + for (i =3D 0; i < SWAP_NR_ORDERS; i++) + INIT_LIST_HEAD(&p->nonfull_clusters[i]); + for (i =3D 0; i < swap_header->info.nr_badpages; i++) { unsigned int page_nr =3D swap_header->info.badpages[i]; if (page_nr =3D=3D 0 || page_nr > swap_header->info.last_page) --=20 2.45.1.288.g0e0cd299f1-goog