From nobody Thu Apr 2 23:55:36 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC72129ACC0; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771253887; cv=none; b=gjxNZpXz15dXBFjjhNZqKMkXiaNWJ6NIEIPvlSXilGv7HiW8pk+9/bLZIuPIdgWUvQM3e1w8aBo5CQrfmnT6EnYy+VOYZDC2w06jSBIQt6YxaJCLAEa2iYju5RFo2DwXfHjiXTIe/4p2d3pzQR27ChbHmtl0R6YzMwqZk7SlaHE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771253887; c=relaxed/simple; bh=ijCT+Mg7d3790XUMIRWeaMLVEKai5+A/PxJRt61VFl4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=F64yzK/M/xHL+OlVg1YzKaOEeCbj+MU5+5j/0etYxMI0O4puNdfiCcm8RASeyo1batB0DxS50wNIwi0CKjazl55O4SDOGSwUkpKVjurZdyXUtFjkwTvaSxTu2H4UyFD5XMbc7aCU+ko2GNm2BSEVT08+hlWoTTkO5IW3OT1G0vU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bscfflmo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bscfflmo" Received: by smtp.kernel.org (Postfix) with ESMTPS id 94F5AC19423; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1771253887; bh=ijCT+Mg7d3790XUMIRWeaMLVEKai5+A/PxJRt61VFl4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=bscfflmoBukdJNZyPNYP0zJ95O8NpJxdXS5KVMOFS1tSazf4yN9d8BgInRxT1k7vj 0GU0hgjestKOZ7Xyx2RDNObFGJRZ3ZoUYhrdu9w1qZnkSLjGSYP2twTYsmWnDpmVtS /+l9UrhxdXye2SNCyGrfJSmNDjyNzDUYZpMTMjwRADlr5+QkrmSN8J5ABJHvW3mmfD d7d3kd/FHIadjDIITNDvIGmwnk17ZQ/jTQyLH4+jFvzsQbKCQMX46CkdechMzwS4nw EWUwIpqBcfpWIHdDYLNll9Orj7AkyD+EqQfpQzT4tMOXmDs1klktA+ykYQKBZYex2y 00UGdyR9cLu6g== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 832D6E81A2F; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 16 Feb 2026 22:58:02 +0800 Subject: [PATCH v4 1/3] mm, swap: speed up hibernation allocation and writeout Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260216-hibernate-perf-v4-1-1ba9f0bf1ec9@tencent.com> References: <20260216-hibernate-perf-v4-0-1ba9f0bf1ec9@tencent.com> In-Reply-To: <20260216-hibernate-perf-v4-0-1ba9f0bf1ec9@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Kemeng Shi , Nhat Pham , Baoquan He , Barry Song , Carsten Grohmann , "Rafael J. Wysocki" , linux-kernel@vger.kernel.org, "open list:SUSPEND TO RAM" , Carsten Grohmann , Kairui Song , stable@vger.kernel.org X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1771253885; l=2673; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=AP2Agry4RlYj89KzjPUleAMF+xnfApgEpjtkm4VqQJg=; b=bYE9CeOOUGzRx+yNxVfIU5aCtzWSmeScGretFXZJqot6KApMpREnHU1VqdZoQCKDTf2OtmG22 hfw1sagCcsYBeIpg3RySdPKCrxkRVZmWNBmjGE/dx7rS7MzapMvhc7C X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Since commit 0ff67f990bd4 ("mm, swap: remove swap slot cache"), hibernation has been using the swap slot slow allocation path for simplification, which turns out might cause regression for some devices because the allocator now rotates clusters too often, leading to slower allocation and more random distribution of data. Fast allocation is not complex, so implement hibernation support as well. Test result with Samsung SSD 830 Series (SATA II, 3.0 Gbps) shows the performance is several times better [1]: 6.19: 324 seconds After this series: 35 seconds Fixes: 0ff67f990bd4 ("mm, swap: remove swap slot cache") Reported-by: Carsten Grohmann Closes: https://lore.kernel.org/linux-mm/20260206121151.dea3633d1f0ded7bbf4= 9c22e@linux-foundation.org/ Link: https://lore.kernel.org/linux-mm/8b4bdcfa-ce3f-4e23-839f-31367df7c18f= @gmx.de/ [1] Cc: stable@vger.kernel.org Signed-off-by: Kairui Song Tested-by: Carsten Grohmann --- mm/swapfile.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index c6863ff7152c..32e0e7545ab8 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1926,8 +1926,9 @@ void swap_put_entries_direct(swp_entry_t entry, int n= r) /* Allocate a slot for hibernation */ swp_entry_t swap_alloc_hibernation_slot(int type) { - struct swap_info_struct *si =3D swap_type_to_info(type); - unsigned long offset; + struct swap_info_struct *pcp_si, *si =3D swap_type_to_info(type); + unsigned long pcp_offset, offset =3D SWAP_ENTRY_INVALID; + struct swap_cluster_info *ci; swp_entry_t entry =3D {0}; =20 if (!si) @@ -1937,11 +1938,21 @@ swp_entry_t swap_alloc_hibernation_slot(int type) if (get_swap_device_info(si)) { if (si->flags & SWP_WRITEOK) { /* - * Grab the local lock to be compliant - * with swap table allocation. + * Try the local cluster first if it matches the device. If + * not, try grab a new cluster and override local cluster. */ local_lock(&percpu_swap_cluster.lock); - offset =3D cluster_alloc_swap_entry(si, NULL); + pcp_si =3D this_cpu_read(percpu_swap_cluster.si[0]); + pcp_offset =3D this_cpu_read(percpu_swap_cluster.offset[0]); + if (pcp_si =3D=3D si && pcp_offset) { + ci =3D swap_cluster_lock(si, pcp_offset); + if (cluster_is_usable(ci, 0)) + offset =3D alloc_swap_scan_cluster(si, ci, NULL, pcp_offset); + else + swap_cluster_unlock(ci); + } + if (!offset) + offset =3D cluster_alloc_swap_entry(si, NULL); local_unlock(&percpu_swap_cluster.lock); if (offset) entry =3D swp_entry(si->type, offset); --=20 2.52.0 From nobody Thu Apr 2 23:55:36 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFBFC313524; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771253888; cv=none; b=qdsUO3YlwuEvY8fTQgfjojZ+Q5tH7Jse5EMgf/lN/ZGxzXKcQIUUSYJ9JKUDgEurBeCLF3+BP8Tz6sc3bB56yGMUSLiFQNWjSgODmGuEJ/zA+He6HtLz9d4sMpa9ceXoo88Rk5mvGzT5yxYc8gBm4joZ+exhYoOdMuR5t0Ink0k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771253888; c=relaxed/simple; bh=myZyOnHzp3lPkDV1e/C2uP7wr+40p+PY2MQZBb28QNM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=IAht66NYJZBQ/JuqFO/aTlRGFpwggXyqxJSZOWIM8VPn5z49v9UwEu+CUaGn5Bcd3Ef/+/D8j6nECqOwU/xaq40Wg7lgaEV1fh4Tc2PhBjjgyQgsvMxFUd+sy/fUv4BiCbyA24r/8g9n8aVUBdxelTvTPaDb2egm4q2ebTH2edE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=l+Gzv+/s; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="l+Gzv+/s" Received: by smtp.kernel.org (Postfix) with ESMTPS id BB19EC2BC86; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1771253887; bh=myZyOnHzp3lPkDV1e/C2uP7wr+40p+PY2MQZBb28QNM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=l+Gzv+/sQQ4gp2ucvDPntyFzntvJ9KJ0ebymysuwtN+qX7m7zMEXm3MMn/AalPv+E jSEzV2n9MICzI1NIzmO8o78n297/bVF7g40z1thhSU9uTSz8wEluVywhNKGPPO5nLu +KctywZ7ENltWgIDQXhQOB+taPHiBPtSZ+u6TqrY+egBC/xfSohDZIAuQXrJQH1EIW wTCqr2oMh0vRwdhZ2cnoou0aHVW4h4dlvVHBBIZYIHhrn9lyzLmx/N2tcpyEfAS1BZ rC3zOMX3m/ctsyCQfph+WAuANsaHTb3fEWJwS7nJx0llHjelyCJ+1x9AMT8UCIOtRT tschDypbqiasA== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A823BE81A28; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 16 Feb 2026 22:58:03 +0800 Subject: [PATCH v4 2/3] mm, swap: reduce indention for hibernate allocation helper Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260216-hibernate-perf-v4-2-1ba9f0bf1ec9@tencent.com> References: <20260216-hibernate-perf-v4-0-1ba9f0bf1ec9@tencent.com> In-Reply-To: <20260216-hibernate-perf-v4-0-1ba9f0bf1ec9@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Kemeng Shi , Nhat Pham , Baoquan He , Barry Song , Carsten Grohmann , "Rafael J. Wysocki" , linux-kernel@vger.kernel.org, "open list:SUSPEND TO RAM" , Carsten Grohmann , Kairui Song X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1771253885; l=2613; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=NAvqyk1BCM82J8HW6Gnh4CX79o/hdmbrD/B6Y3rHXiw=; b=k/nFxKLOf037gv1sMhovQ1z7Lzzq5b7SFtNWk9LCk4plm6mzv2JlgbNEHqsWpL6ecmjVkQy7s lmvDhThxszvBx4B7SxU8s36kT/fNDml41igu0SWj6EFXOnsA1XxehU2 X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song It doesn't have to check the device flag, as the allocator will also check the device flag and refuse to allocate if the device is not writable. This might cause a trivial waste of CPU cycles of hibernate allocation raced with swapoff, but that is very unlikely to happen. Removing the check on the common path should be more helpful. Signed-off-by: Kairui Song Tested-by: Carsten Grohmann --- mm/swapfile.c | 51 +++++++++++++++++++++++---------------------------- 1 file changed, 23 insertions(+), 28 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 32e0e7545ab8..ea63885f344a 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1931,35 +1931,30 @@ swp_entry_t swap_alloc_hibernation_slot(int type) struct swap_cluster_info *ci; swp_entry_t entry =3D {0}; =20 - if (!si) - goto fail; - - /* This is called for allocating swap entry, not cache */ - if (get_swap_device_info(si)) { - if (si->flags & SWP_WRITEOK) { - /* - * Try the local cluster first if it matches the device. If - * not, try grab a new cluster and override local cluster. - */ - local_lock(&percpu_swap_cluster.lock); - pcp_si =3D this_cpu_read(percpu_swap_cluster.si[0]); - pcp_offset =3D this_cpu_read(percpu_swap_cluster.offset[0]); - if (pcp_si =3D=3D si && pcp_offset) { - ci =3D swap_cluster_lock(si, pcp_offset); - if (cluster_is_usable(ci, 0)) - offset =3D alloc_swap_scan_cluster(si, ci, NULL, pcp_offset); - else - swap_cluster_unlock(ci); - } - if (!offset) - offset =3D cluster_alloc_swap_entry(si, NULL); - local_unlock(&percpu_swap_cluster.lock); - if (offset) - entry =3D swp_entry(si->type, offset); - } - put_swap_device(si); + /* Return empty entry if device is not usable (swapoff or full) */ + if (!si || !get_swap_device_info(si)) + return entry; + /* + * Try the local cluster first if it matches the device. If + * not, try grab a new cluster and override local cluster. + */ + local_lock(&percpu_swap_cluster.lock); + pcp_si =3D this_cpu_read(percpu_swap_cluster.si[0]); + pcp_offset =3D this_cpu_read(percpu_swap_cluster.offset[0]); + if (pcp_si =3D=3D si && pcp_offset) { + ci =3D swap_cluster_lock(si, pcp_offset); + if (cluster_is_usable(ci, 0)) + offset =3D alloc_swap_scan_cluster(si, ci, NULL, pcp_offset); + else + swap_cluster_unlock(ci); } -fail: + if (offset =3D=3D SWAP_ENTRY_INVALID) + offset =3D cluster_alloc_swap_entry(si, NULL); + local_unlock(&percpu_swap_cluster.lock); + if (offset) + entry =3D swp_entry(si->type, offset); + put_swap_device(si); + return entry; } =20 --=20 2.52.0 From nobody Thu Apr 2 23:55:36 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F376331353B; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771253888; cv=none; b=V2E8RKSIvvmwqWs9Fx4tgbxYgZ6ghwJEsNFnN22709XWN2RamEGrcctpkgZ13a9U3QkDVHwi3Xay1Fbwi2bI4aW5JdI17Nq/85qjxAnCn/KrlXE/nfrN2TP5JEfKT1Za2jYqqRgiptCtXEWYBBVOVUHANpWcP72KsgY1dd6lVtY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771253888; c=relaxed/simple; bh=WYW37/BfhOOjNfUbgzmr0b/6G+uzVVZ+LippTRuq7l8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=NnQYWrBS+mphY6aEuIuzsZ71Fz+/HbXRvtxzi6AbXEgpTb9EBFgDYrNzKAGdcXdKWesdvblV3OdgxjWB1enIB+I0pH1j17TimzByMZg6xhHfZz5anknEqhuIFv9PjH9iL5/J86L/zGguehVX9q0h4pRYZVcrxROegFy/Y58ZIIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LZ1MWcoW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LZ1MWcoW" Received: by smtp.kernel.org (Postfix) with ESMTPS id D3A96C116C6; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1771253887; bh=WYW37/BfhOOjNfUbgzmr0b/6G+uzVVZ+LippTRuq7l8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=LZ1MWcoWLWOo8VNMgjp1CUNUSjNO4mLPJIUXIY8WVZaRAj37yWd0ELYIErKcDYEGk IAH/orXK8IXxhlwbISW5/F32Eu0hHlaPuSqT/UvEmVsd9CE8IbrIukvAtPvNYaBSJT yFlvY2UG9iEqFlyfgSDHnNFeqo3++trMu5VLTeLSmTsurpfXTQ1SBPf8ka7E2obJwh vCL+VBaDdNCQBZJlR62n3Q60pILX9xDKk+ZZibBuN5GRNbWS2WGwQXvh4y+QwbfofO Dn0yMujM3OXGbRxR4jJAn//kSgQoBunG3uCASyEjzYlBc97WMEl9FSzrmowsQxi9fB 8vQSsgsLWnXuA== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C15F2E81A32; Mon, 16 Feb 2026 14:58:07 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 16 Feb 2026 22:58:04 +0800 Subject: [PATCH v4 3/3] mm, swap: merge common convention and simplify allocation helper Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260216-hibernate-perf-v4-3-1ba9f0bf1ec9@tencent.com> References: <20260216-hibernate-perf-v4-0-1ba9f0bf1ec9@tencent.com> In-Reply-To: <20260216-hibernate-perf-v4-0-1ba9f0bf1ec9@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Kemeng Shi , Nhat Pham , Baoquan He , Barry Song , Carsten Grohmann , "Rafael J. Wysocki" , linux-kernel@vger.kernel.org, "open list:SUSPEND TO RAM" , Carsten Grohmann , Kairui Song X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1771253885; l=4406; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=XE85Cs7eMoW2MzbPsMRN8glTTJ2qQfea8kHvJhs9XRM=; b=XjOjKB6C9RKF289BaDOi2fed12TaT5b+w77m1kwzOZ5PBWUKf9cA9Lwh5/Vmc8idUkOkv49cW CGGS8aMFVcQCG2luwdfgnGmOyIk+xyV50eFl6DypcOEVr3JsMK69xGJ X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Almost all callers of the cluster scan helper require the: lock -> check usefulness/emptiness check -> allocate -> unlock routine. So merge them into the same helper to simplify the code. While at it, add some kerneldoc too. Signed-off-by: Kairui Song Tested-by: Carsten Grohmann --- mm/swapfile.c | 54 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 31 insertions(+), 23 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index ea63885f344a..a6276c5ead8e 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -910,7 +910,21 @@ static bool cluster_alloc_range(struct swap_info_struc= t *si, return true; } =20 -/* Try use a new cluster for current CPU and allocate from it. */ +/* + * alloc_swap_scan_cluster - Scan and allocate swap entries from one clust= er. + * @si: the swap device of the cluster. + * @ci: the cluster, must be locked. + * @folio: the folio to allocate for, could be NULL. + * @offset: scan start offset, must be a swap device offset pointing insid= e @ci. + * + * Scan the swap slots inside @ci, starting from @offset, and allocate + * contiguous entries that point to these slots. If @folio is not NULL, fo= lio + * size number of entries are allocated, and the starting entry is stored = to + * folio->swap. If @folio is NULL, one entry will be allocated and passed = to + * the caller as the return value. In both cases, the offset is returned. + * + * This helper also updates the percpu cached cluster. + */ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci, struct folio *folio, unsigned long offset) @@ -923,11 +937,14 @@ static unsigned int alloc_swap_scan_cluster(struct sw= ap_info_struct *si, bool need_reclaim, ret, usable; =20 lockdep_assert_held(&ci->lock); - VM_WARN_ON(!cluster_is_usable(ci, order)); =20 - if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER) + if (!cluster_is_usable(ci, order) || end < nr_pages || + ci->count + nr_pages > SWAPFILE_CLUSTER) goto out; =20 + if (cluster_is_empty(ci)) + offset =3D cluster_offset(si, ci); + for (end -=3D nr_pages; offset <=3D end; offset +=3D nr_pages) { need_reclaim =3D false; if (!cluster_scan_range(si, ci, offset, nr_pages, &need_reclaim)) @@ -951,6 +968,14 @@ static unsigned int alloc_swap_scan_cluster(struct swa= p_info_struct *si, break; } out: + /* + * Whether the allocation succeeded or failed, relocate the cluster + * and update percpu offset cache. On success this is necessary to + * mark the cluster as cached fast path. On failure, this invalidates + * the percpu cache to indicate an allocation failure and next scan + * should use a new cluster, and move the failed cluster to where it + * should be. + */ relocate_cluster(si, ci); swap_cluster_unlock(ci); if (si->flags & SWP_SOLIDSTATE) { @@ -1060,14 +1085,7 @@ static unsigned long cluster_alloc_swap_entry(struct= swap_info_struct *si, goto new_cluster; =20 ci =3D swap_cluster_lock(si, offset); - /* Cluster could have been used by another order */ - if (cluster_is_usable(ci, order)) { - if (cluster_is_empty(ci)) - offset =3D cluster_offset(si, ci); - found =3D alloc_swap_scan_cluster(si, ci, folio, offset); - } else { - swap_cluster_unlock(ci); - } + found =3D alloc_swap_scan_cluster(si, ci, folio, offset); if (found) goto done; } @@ -1332,14 +1350,7 @@ static bool swap_alloc_fast(struct folio *folio) return false; =20 ci =3D swap_cluster_lock(si, offset); - if (cluster_is_usable(ci, order)) { - if (cluster_is_empty(ci)) - offset =3D cluster_offset(si, ci); - alloc_swap_scan_cluster(si, ci, folio, offset); - } else { - swap_cluster_unlock(ci); - } - + alloc_swap_scan_cluster(si, ci, folio, offset); put_swap_device(si); return folio_test_swapcache(folio); } @@ -1943,10 +1954,7 @@ swp_entry_t swap_alloc_hibernation_slot(int type) pcp_offset =3D this_cpu_read(percpu_swap_cluster.offset[0]); if (pcp_si =3D=3D si && pcp_offset) { ci =3D swap_cluster_lock(si, pcp_offset); - if (cluster_is_usable(ci, 0)) - offset =3D alloc_swap_scan_cluster(si, ci, NULL, pcp_offset); - else - swap_cluster_unlock(ci); + offset =3D alloc_swap_scan_cluster(si, ci, NULL, pcp_offset); } if (offset =3D=3D SWAP_ENTRY_INVALID) offset =3D cluster_alloc_swap_entry(si, NULL); --=20 2.52.0