From nobody Sun Feb 8 11:48:29 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E614D286AE for ; Wed, 8 May 2024 22:41:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208112; cv=none; b=IPpST/WpV6QQS3J3nYolcvkv+f46+hXVNjrbu/VCxskf4gYYBRFCghz8z6X2+dY9F4T8rAJVqtjvuL07Tpqza/Tp238bE7fNGVrstty2bg6sTlGGAZqld8Z3muLwiDM+OkZ/Jii44hhJip80mBc2FIyEIhQYnYmvbFWByFzu3Ds= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208112; c=relaxed/simple; bh=nrAk9y30+c6gajgSfc4mOHwujo2W6S4FOBjLI/TfuUY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FLm7IvszcmbmM8IHcabSvCgiG6V/kTnWOqjs3N1fbNQZJw3KXm2R1Qb/+uuevHseb7kpxuD0zjRcWR/KX+EWkdOnVV1QHU9vDsd6IAsD1JuL487gSDketZmEGNtiipjzwaB8tU6aOfcYpDcGqyZ8xB27dM24x81AEG0FUzX23iE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=O/aKUiKW; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="O/aKUiKW" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2b3646494a8so241119a91.2 for ; Wed, 08 May 2024 15:41:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208110; x=1715812910; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DOx+XLJ6NG229/wS5wOB5lVEjUmMQDyjSrW3ljEjCDo=; b=O/aKUiKWhsaKrkTDqwOqlNKY4Ch3rw/6FIPZLSiC8zybtq28idaiYgnAhp+rBugAjA nelqlXFAtxGqsXAAfiKKvYnM2OM0Srn4IknZmnMc6tgOy2kE/eqhhE0r4N5zuYuJR4sr rDmDABBWyx6cy8/FZthDBdmfAAYW8ylbZAt9hH7C6dsOoAbXPLvT7+43Xid4tVpPGYcQ JjjiBk2Lmb5UMnd+5DvpAJI7dtUQJJizlLy8NyfCHzYUkZ5jal/9J3MaDBzcYxWd1G8W X2yL2gc79x/tGFDmO7QJkCUeG6YIwHXtBjLyKGEw47Kf0lD2QrDZbf4FzXSoyS8DAlSp EMyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208110; x=1715812910; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DOx+XLJ6NG229/wS5wOB5lVEjUmMQDyjSrW3ljEjCDo=; b=f+EvEIDqy35CBwuoeXR83ZwFSF26PAhFsapUNHLcXMHWoOI/ZN0CEDbsprDQqR2xnr w9IQxTzJcX6yN6VSjS9MawbCeLIoEdt+91MkLQl3N3p17ElSmQiEkun/goC4ekhxf251 989UkgE6gMBz21g9Hbk/6/w4P/CeNwWrbBlOXzGcqSU/+oC+MP/SmHei8drR5M/BWoM9 Olik5y4fbLwrRyL1DQT74uWZDvKqsCl8fKhISAAKJSfoAT0PSi4bJVgMvb0D6qwUfvGU 2WSCMaaF0bXEQaubdZWRFMsWoCBtXa/6N5ENOHcRQSi9iqH4nILbOp0HHxYb8Gb/fAsm ALmQ== X-Forwarded-Encrypted: i=1; AJvYcCV5w5a7aPQH8YucFQS3yCz8ok6i/yIUIG6Lbn0U7uDJvauj5YPhEsobx8eIDQ0jvH6OoWFyFKCRva8e83iORoOLAKnCAzWMxyjbhQaK X-Gm-Message-State: AOJu0YyHPrrJ7lzxGarUQqjuY7w9bkVwu/ltEzyLPTiE9HJK8uvxE37A rzfYRlLYkIjeYt06ZaQknP/GLaSRybFHjZxaBedmzKTo+4vJJY25 X-Google-Smtp-Source: AGHT+IEFp0bWFb71EA7kiwqcgB9bekLg3MgzbqqZFw9IyUlHL4NxSW9RrXf+dnY9e3fmsofVV/uA3A== X-Received: by 2002:a17:90a:fa8c:b0:2b5:1f7d:7eae with SMTP id 98e67ed59e1d1-2b6165c4fbbmr3664457a91.24.1715208110215; Wed, 08 May 2024 15:41:50 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.41.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:41:49 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 1/6] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Thu, 9 May 2024 10:40:35 +1200 Message-Id: <20240508224040.190469-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Furthermore, this new function, swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Acked-by: Chris Li Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..d1d35e92d7e9 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } =20 +static inline void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index f6ca215fb92f..ec12f2b9d229 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } =20 +static void cluster_swap_free_nr(struct swap_info_struct *sis, + unsigned long offset, int nr_pages) +{ + struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) =3D { 0 }; + int i, nr; + + ci =3D lock_cluster_or_swap_info(sis, offset); + while (nr_pages) { + nr =3D min(BITS_PER_LONG, nr_pages); + for (i =3D 0; i < nr; i++) { + if (!__swap_entry_free_locked(sis, offset + i, 1)) + bitmap_set(to_free, i, 1); + } + if (!bitmap_empty(to_free, BITS_PER_LONG)) { + unlock_cluster_or_swap_info(sis, ci); + for_each_set_bit(i, to_free, BITS_PER_LONG) + free_swap_slot(swp_entry(sis->type, offset + i)); + if (nr =3D=3D nr_pages) + return; + bitmap_clear(to_free, 0, BITS_PER_LONG); + ci =3D lock_cluster_or_swap_info(sis, offset); + } + offset +=3D nr; + nr_pages -=3D nr; + } + unlock_cluster_or_swap_info(sis, ci); +} + +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int nr; + struct swap_info_struct *sis; + unsigned long offset =3D swp_offset(entry); + + sis =3D _swap_info_get(entry); + if (!sis) + return; + + while (nr_pages) { + nr =3D min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER= ); + cluster_swap_free_nr(sis, offset, nr); + offset +=3D nr; + nr_pages -=3D nr; + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ --=20 2.34.1 From nobody Sun Feb 8 11:48:29 2026 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D422924B33 for ; Wed, 8 May 2024 22:41:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208120; cv=none; b=Cx0jw1LvTQBO3jYtGIx1QsyB22t2aLjJ0X4wsbcD1k54HCEnw27MduNd2Mvu0xAkHziizWmO3nkwny78isrgxjBKU8vt/fQTnoBEdHiJanvjZTjMQ6xq1LZFyfxOgpVR+GBcOIpxjI8kMudubK7zVU/k1TFlf0PegoUJWZYLIeE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208120; c=relaxed/simple; bh=VqBZ46CyXKDAAuUJum5h9nOVdshvg4zc3yJsiQ7tErU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lK8z6uXn+qU5/YTTbU5OOVAtU0Ovv68NNus1NwnF7PV8L6zYJ2rdACynj2Ca0wUdxKnbqnY7iHlIi5TM3c2psYAwMOER5I5eVWZrfPNZqg8cnkUXj5FmAeJ1bCE+J1FB+PnW6e1/mWhepHxRJsYSRdYeYKPzZhDA5nIPzlLH214= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nOLABrv7; arc=none smtp.client-ip=209.85.215.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nOLABrv7" Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-61aef9901deso225686a12.1 for ; Wed, 08 May 2024 15:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208118; x=1715812918; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BKP4frf3SJsYj3bpFkLEvp3HW+vS2YimBRDSFS3kDwg=; b=nOLABrv7cIMijse0YMjRuOFPYV+7TNOCb0LeV8J7kU4mozc6JEStyp6GyQL+byNN9b andMjioCoCe7I7Yn9qwFVZtFAxjoyFfKWw9znSPOqb6bEdkjcDH2lqjaqr4CkhEmb++a JYJ9Mqxu9cnqCtMhGaah22OjZEq89QDiibvnQ0EVLQddpclwdnK4DqxaKrrG4Q5pGjXc uS5crHbv87Bwiu1ZeFQVMAdjDvG5GqPQPpTMSaZSwkjc7PboJEtALg6Kg66vY/qNtnh4 CW8YYsokRPw8yNsQgpcVac3pKLXdUc02OIrynwjd6PJHxQYuO2rTl8xCPg0b9EJlC+HF Avxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208118; x=1715812918; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BKP4frf3SJsYj3bpFkLEvp3HW+vS2YimBRDSFS3kDwg=; b=W39BNNrOmRBXI5B89tcjxNcpKZKYo+iB5i8LzDZgX957XItJcJyX9QZ2qsxYGoj6M2 ecD4jIqpyTOwNQEVbbilMfJAcgwVrvnRNbcJmvxSyvxbfcJc7fTkgw7xpGeZUZDpsQx1 toEkmg7wo6fag8yANZBvzLEt+0kcR2Kc2p9HzUM5y2vSGxPWH2keX5pAZr+guuvOczHH CZ88yOrVF0hXK1t/Y+QkST1v5skPfiPsoGI7XxcYNOl/igrsT+phRFG0SB8PZRqTN4HW zPU63X7gSqEgBIS80cvy3hd2RmV+NLEwtEc7ei0/346fIs22jg3s4ZqJ9oDSAIE5BdlR tC+w== X-Forwarded-Encrypted: i=1; AJvYcCXTRY3cLylJ/HNHYqKXz+8BeOzR8qAITkGTRo1ZWu9jPfT1vQyVyv9m4aScKKoiEQXweyXHXczY3WomLAeZtJOELMuRXNjD3FFOWplr X-Gm-Message-State: AOJu0Yy0MUYRydUJJi1Q1GTV3g+XMCEnZGO8ug4gUI5ai86zxkSDfRHZ GysO2j6DuUKv+VeqTr2NUShFK6MIA+mQBhyOGSlwwI6UzZbnl3wY X-Google-Smtp-Source: AGHT+IFi9srqYFLAGkCKh6k9BgPnyvD1B3mYyYQyZ3x2gIfLtku4oUuO4QjlFLxaj5GaZF8qFdK07w== X-Received: by 2002:a17:90b:894:b0:2b6:2138:e407 with SMTP id 98e67ed59e1d1-2b62138e4c9mr3786419a91.9.1715208118171; Wed, 08 May 2024 15:41:58 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:41:57 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, "Rafael J. Wysocki" , Pavel Machek , Len Brown , Christoph Hellwig Subject: [PATCH v4 2/6] mm: remove the implementation of swap_free() and always use swap_free_nr() Date: Thu, 9 May 2024 10:40:36 +1200 Message-Id: <20240508224040.190469-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song To streamline maintenance efforts, we propose removing the implementation of swap_free(). Instead, we can simply invoke swap_free_nr() with nr set to 1. swap_free_nr() is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1. A prime candidate for leveraging swap_free_nr() lies within kernel/power/swap.c. Implementing this change facilitates the adoption of batch processing for hibernation. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: "Huang, Ying" Acked-by: Chris Li Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Len Brown Cc: Hugh Dickins Cc: Christoph Hellwig Reviewed-by: Ryan Roberts --- include/linux/swap.h | 10 +++++----- kernel/power/swap.c | 5 ++--- mm/swapfile.c | 17 ++++------------- 3 files changed, 11 insertions(+), 21 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index d1d35e92d7e9..48131b869a4d 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp= _t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); -extern void swap_free(swp_entry_t); extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); @@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp) return 0; } =20 -static inline void swap_free(swp_entry_t swp) -{ -} - static inline void swap_free_nr(swp_entry_t entry, int nr_pages) { } @@ -613,6 +608,11 @@ static inline void free_swap_and_cache(swp_entry_t ent= ry) free_swap_and_cache_nr(entry, 1); } =20 +static inline void swap_free(swp_entry_t entry) +{ + swap_free_nr(entry, 1); +} + #ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) { diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 5bc04bfe2db1..75bc9e3f9d59 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -200,12 +200,11 @@ void free_all_swap_pages(int swap) =20 while ((node =3D swsusp_extents.rb_node)) { struct swsusp_extent *ext; - unsigned long offset; =20 ext =3D rb_entry(node, struct swsusp_extent, node); rb_erase(node, &swsusp_extents); - for (offset =3D ext->start; offset <=3D ext->end; offset++) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, ext->start), + ext->end - ext->start + 1); =20 kfree(ext); } diff --git a/mm/swapfile.c b/mm/swapfile.c index ec12f2b9d229..99e701620562 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct = *p, swp_entry_t entry) swap_range_free(p, offset, 1); } =20 -/* - * Caller has made sure that the swap device corresponding to entry - * is still around or has not been recycled. - */ -void swap_free(swp_entry_t entry) -{ - struct swap_info_struct *p; - - p =3D _swap_info_get(entry); - if (p) - __swap_entry_free(p, entry); -} - static void cluster_swap_free_nr(struct swap_info_struct *sis, unsigned long offset, int nr_pages) { @@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_st= ruct *sis, unlock_cluster_or_swap_info(sis, ci); } =20 +/* + * Caller has made sure that the swap device corresponding to entry + * is still around or has not been recycled. + */ void swap_free_nr(swp_entry_t entry, int nr_pages) { int nr; --=20 2.34.1 From nobody Sun Feb 8 11:48:29 2026 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A61444500F for ; Wed, 8 May 2024 22:42:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208127; cv=none; b=h1VgJSjqFvgF9DcCqet1tfxcovbq7aT/huaGRKDQgOXXz1Fer1VB/M6+qgmmupP4h3j0ydTg0UCqvFAhFIg246spBk7NBEy9cVsnvN6ZFHvJMopnIHqHillMNtE3PbmScoFkRXljshfsneBYUKd8pu5HmRdedQCjA5fxSyWAXfg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208127; c=relaxed/simple; bh=oNACM1lFXAcyx4Hd2MDWCE00/3JOUNQ8ObZMpTgnYRA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=F0UgRPrXPxNTICIfvwMfuIHkwou+fiEOXKoRSiRPB62Bz8iRNUZBzTxeqiAX4R32XR5VCmIyGV4PiTVCOs0PPSDAJRP+D+RP9NqZ15fjIejOI2duLEKDbS1FL9+uX0iDfo6/N1AJ0KDCAyuwLVpwwRi3n/81ORWvRU2h4blrgs8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WZx4xXMB; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WZx4xXMB" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5dcc4076c13so220808a12.0 for ; Wed, 08 May 2024 15:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208125; x=1715812925; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UYGLEQ5dG8GZxiWnoQtIB+ho0dBRJLc2akdsz5aCuO8=; b=WZx4xXMB9w6B1ICK5rlxEXgmoexuZ/uSOlV/trqrOgaK04mni8sIei96ej2vgWEP97 E2za1hOidU0ExvnLrUFUyeOZ/eyHVAtLONZEyRuGceQK3Wwo/9DmclY8q8TXvdBvFqzV ymJn16Pc2zXQGzAUu6KILqVcQWxIydw6+cntef+JMr+lQq+uTxe1NCcDs/Q6i7I6dSgP tVRkLpKJv2yDebXRzdVpF1zT6OjbxGyeTxf+Zia4OhnLa2Nzu8WtS2Luw2SRLP1H5K1X vCDreMPKBN3TFr0393L8YC4aa/NGBxnO0hzeTUvm92kD3vOKK6+vi0HQYQjpTeLAb77b GQfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208125; x=1715812925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UYGLEQ5dG8GZxiWnoQtIB+ho0dBRJLc2akdsz5aCuO8=; b=u8Ki7fgTsYqUBr4Ap6bMaeU6DiB9vW4dmNk9+0DwbTsG0WnJsyzXYonhJ/Sqq8UfKq BHWbnmK0Ihfo98HBNsUKeseTlfsm4UMwNGfcnvHGQgcYG/AL2Pe7SiHQSVN63IMnvBhA C2HvMHdMNZ/3EaecWPhcI1sDVHmtlbpKgizWf+/QhVa6KKHWmUtrIsRwx49aVSJdGXL6 W0LT5Na+wYUt9tOKDW7V/poa42o7/31Tl+p8zXiit99VCBTJcn5cbP7H8Rb8aL8iJ+a6 ECk1dqj+9sg6Nvqwvo2kfoFbRYhDw+m0yBu21YPTJR5QHLo/L9RsOdYxNEX/L2u4Akmo yMig== X-Forwarded-Encrypted: i=1; AJvYcCUHz6wZTsKVkU0+qoNa+pTjhtDobyQmGDdxtefBbYJ9LX1R1CWYmsXQg8DTn47/f0kbvebqeaMfiCoZ8QeXL4IRo6GCn5ACULCQ1cM9 X-Gm-Message-State: AOJu0YwTlHSrcf/PcYOmCV3LGpXXK1aa62NGqazJQb84dCidDalqfCYn BtFBWmmU2CuonuUs6HXgqKsqYWao/lwVlxEs3nqmXCurhhN3iFl1 X-Google-Smtp-Source: AGHT+IFAtnVi6xLor2XHGazgK0Nm3nnCaOAaL5JyBksp2vgFvFWPvYOXqRq2Ix3zEh88ZA/XgC19yw== X-Received: by 2002:a17:90a:db4c:b0:2b4:32be:4442 with SMTP id 98e67ed59e1d1-2b65fa1c4a4mr1504710a91.9.1715208124905; Wed, 08 May 2024 15:42:04 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.41.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:04 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally Date: Thu, 9 May 2024 10:40:37 +1200 Message-Id: <20240508224040.190469-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song There could arise a necessity to obtain the first pte_t from a swap pte_t located in the middle. For instance, this may occur within the context of do_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introduces pte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently, pte_next_swp_offset() will directly invoke it with delta =3D 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b2c75b12014e..17b0a1824948 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio= , unsigned long addr, } =20 /** - * pte_next_swp_offset - Increment the swap entry offset field of a swap p= te. + * pte_move_swp_offset - Move the swap entry offset field of a swap pte + * forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. + * @delta: The direction and the offset we are moving; forward if delta + * is positive; backward if delta is negative * - * Increments the swap offset, while maintaining all other fields, includi= ng + * Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset(pte_t pte) +static inline pte_t pte_move_swp_offset(pte_t pte, long delta) { swp_entry_t entry =3D pte_to_swp_entry(pte); pte_t new =3D __swp_entry_to_pte(__swp_entry(swp_type(entry), - (swp_offset(entry) + 1))); + (swp_offset(entry) + delta))); =20 if (pte_swp_soft_dirty(pte)) new =3D pte_swp_mksoft_dirty(new); @@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) return new; } =20 + +/** + * pte_next_swp_offset - Increment the swap entry offset field of a swap p= te. + * @pte: The initial pte state; is_swap_pte(pte) must be true and + * non_swap_entry() must be false. + * + * Increments the swap offset, while maintaining all other fields, includi= ng + * swap type, and any swp pte bits. The resulting pte is returned. + */ +static inline pte_t pte_next_swp_offset(pte_t pte) +{ + return pte_move_swp_offset(pte, 1); +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. --=20 2.34.1 From nobody Sun Feb 8 11:48:29 2026 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AA2324B2F for ; Wed, 8 May 2024 22:42:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208134; cv=none; b=lhRVTWusD5QMCt5xLuu9/QMlFlqkkr3jrx8RD2KYzsu/cgAyBoH/VqHgpQmDT7myw7urLNEvJNqq3BENSogI0Tob2pKBWhCMprySE5zXWedJB6t1CQ1IAOskJovonpPVydbrqB6gmebyxMjWtfaVINuV0iCY8w7mHB8fPbmGXkw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208134; c=relaxed/simple; bh=xdbI3HgqGVbYBMlkoEOKyGSFZrOh4mVYelo6HDS93bI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mRLbJ7dpdE7MVRrlG/m4xeMoex8UNRm7bMP593M3bpQ1JFpHlEYFsgAsXnBNvDQM1SFKKt0g91TDG3vB9PZG2wZBSHmT+yGQX6HBkcE62L4BoI+c4wk0LyPq+N9W2dXC0o42zQ34pWhPSMmUrkHOXtsh7xFAXbNDox2rvvpp+ns= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ah2I32fw; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ah2I32fw" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-5e4f79007ffso204254a12.2 for ; Wed, 08 May 2024 15:42:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208132; x=1715812932; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7DcpNxuaHKQ2XGPRcrgwYPClc6OYNtCFnxYQ34hBDF0=; b=Ah2I32fwodcmjO7bcCXeobZR2yL7NVrXgDawWk6FR3zEDwRUbEcmCW00Wzx7lhXnjh jQB6nMqZqQHNKmM868iHBN5OWqL36XeeFPKIgzKxuU74fF20pByN8IP0xXdR6tGtvxyS XWX17s8OuucoCL5YDRkjYylO87u8UGRF1mqp34ryvaX6hU1d/7chNhPtSoY3JeXXV80S 6G62x1ZBMIGCA/Ly6Wxfg9PnmrfhxNhsFgfMwptqPE+ds6p61R8THb9HlPTWnueJ++t+ tVryNnOZYJd7ydNHMQtEVpo5YHvN5qmluQG48fX444JOJINHxg2KcstCpUoP62TX8seh By4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208132; x=1715812932; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7DcpNxuaHKQ2XGPRcrgwYPClc6OYNtCFnxYQ34hBDF0=; b=AjJVIaLGvJs5MGDySs7DwU0y9n5yQ5YuE+GNANe9nzBWsUTGsBh1RIlZG9+uwQnS+G f2shLUPDb1viLu7arr2fDyucEsRfTGE0oqD330HiCJKcTEQHja0pbhlVYOjWOLy53Xtb 87vokk//Mu+3YL2wcUQXXVozschtTr8KfNxz2NlY2Zz+Qn/mIPWA6MXzw8JPnrxpSA8L e+MTCHyVi0v9ja8YWnZv+udWPQC2mWcgJdXBtwSuypd2HMsdRW19nSWFGYKTncXuBlz+ rRQ+7Ut0yC9Ls29vP4J3P05Su5zKZB/5sb+2f9t4JxtW5+LIO4QbEF1WYOQlbMR2zJOa u5yA== X-Forwarded-Encrypted: i=1; AJvYcCUWPTk+kqUfjjLc4c7PTeCCRAr5tknGXm8Q8ONMEJH+rn88pOLG3whH+Vzk4vp3F0Tzo6Nb03yfdt/o2sFibd1arrNnLRwQsDijBZWR X-Gm-Message-State: AOJu0YyJuMK/LOpu+RuoiptlBAPGjTeUXNRpc81+ZHt50jNyqQQCYQVz 0w/BSI5GMpcEU1jfQidZ46YtfsAK+6rkgu5cgX/oebWdwsPxWChO X-Google-Smtp-Source: AGHT+IGkFdAUAzrMlbUlBhREM/CEvwatOEIMRBL45vN2bg1LTAUKPSPifp6IeAAwfIL/STXKxZU/xg== X-Received: by 2002:a17:90a:8b06:b0:2a2:981b:2c9e with SMTP id 98e67ed59e1d1-2b6169e1a86mr4021869a91.36.1715208132570; Wed, 08 May 2024 15:42:12 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.42.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:12 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, Khalid Aziz , "David S. Miller" , Andreas Larsson Subject: [PATCH v4 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Date: Thu, 9 May 2024 10:40:38 +1200 Message-Id: <20240508224040.190469-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Should do_swap_page() have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enable THP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementing THP_SWAP might invoke this function with nr values exceeding 1, subsequent to do_swap_page() successfully mapping an entire large folio. Nonetheless, their arch_do_swap_page_nr() functions remain empty. Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: Khalid Aziz Cc: "David S. Miller" Cc: Andreas Larsson --- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 18019f037bae..463e84c3de26 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1084,6 +1084,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) }) =20 #ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be @@ -1092,12 +1101,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) * page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page(struct mm_struct *mm, - struct vm_area_struct *vma, - unsigned long addr, - pte_t pte, pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + for (int i =3D 0; i < nr; i++) { + arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE, + pte_advance_pfn(pte, i), + pte_advance_pfn(oldpte, i)); + } } #endif =20 diff --git a/mm/memory.c b/mm/memory.c index eea6e4984eae..b51c059af0b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4266,7 +4266,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte, 1); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { --=20 2.34.1 From nobody Sun Feb 8 11:48:29 2026 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0FCCB24B2F for ; Wed, 8 May 2024 22:42:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208141; cv=none; b=As3iHRcbSTZZXP8xrKj7neekEV0Zpupwk9bz/h+yThp8AIoIOVWCqlkMxGeAXCqykVX58LlyWK77qeTGCS5KmtPLt0X39EQJxCQkVt5IAetXJX9US8BezYD8UfFYWzZitidRh9fa3W30UVZ6+5/rzOWtNBCtrDm8Lu3Bb2UF7Rc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208141; c=relaxed/simple; bh=3JpFwQMSCq/cMZ/d8Z+jM59pDpM4EAcAV96a2sNlir4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ia3unzJCIV6bVxRsvzi9x9diu7nS7kc6a8ykj/neBjkqz0ezvs/kFzUYlvRvJzT8HlroGJ+OaaKyn/mxpmpgSXt+V93tdGLz6UQ4+DOsSadlqZkPA1wdNLnjivJwAt8aaDO5+p5Ld4kaWKU62iHWIPUJnI8RY2xA2ZmgECBDpr0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=l4BoFNAy; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l4BoFNAy" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2b2b42b5126so266954a91.3 for ; Wed, 08 May 2024 15:42:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208139; x=1715812939; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0eTHILZ80XouT+1m3vAPlwsDJfF9Pcan8u7a91N/lZo=; b=l4BoFNAye3TxOi3+bsrpHuTGDehBVrrZcwM8fvFLdHhV123Ufsup69/yxIK7y5lN5B St8ySyP7oHrbYYplf51BkFHgAOr+TF+Mskw199ZiKuzsMuohF+dfHA+x1Wc4M+RJW1dI xnQLyBctHI8jgQvn3neE0H9zhjnWSH5/MSOPlEbviZytMT1jMrwTmmdFYgls3VZX06Zr DA29mLmDnEvr2vTljXHicF8DpehwnygtkKg3tn1ziQml9WNQ6dT0usGI7Cz4hcIwqD9C aVQRTT0rKw1ubfELnZRSzuX4Z6wh5FOlUOzpDcnK/CW4KnMmiR437YqQzSyDGR4RDRbt z+cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208139; x=1715812939; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0eTHILZ80XouT+1m3vAPlwsDJfF9Pcan8u7a91N/lZo=; b=oPws+0W5pAY/4Yd4amvPsOfk8ABA4tUFjtRGFM39jYWPoYJggXtKR0bG0ciblC8sOs ZnL/BKMQm24d63HS2KjfUXa48cU+MwhVChKPE6nqneaUeEIXobfBAqkIRHcmfbZhknn2 VGy4yxnboe7pY6AwaAdHtkof/orfCFTWDNAxcJbGOJ6yX+KK7fXEkCG2KNs7RovONbyF 0JhdIUrefuvdXMu8Rt859p3Jyn04muD0x8qygi3uhBjKF/bXLTW00q+uYA5m5fN4I+5o 8nmCuvZwpgPbu0r1Lt0fb4/l5E/zZs8Q7QQIvzWA0O2YEJo+LphP5pE73z4vQAOI2QkO 8/1g== X-Forwarded-Encrypted: i=1; AJvYcCUMdiNAY2RMCMiY+4QcvQyzDhfbPLZERWKCsMgCwzfPH1TF2Lfbe9DT6yJVx9COFB/agsvFCGgCNhOMUbnqejBp3Gtt1e1A6qfyT5Wu X-Gm-Message-State: AOJu0YyHlk9vabxVSyQ4Zx4224NnUhnSr8LAjiVmLw4SD8ZGnZwTPLtd TVJT/4cDEhfIZ651M4U3ym4Enm/lR3lDenCjFpC7EeZpjYzNGao4 X-Google-Smtp-Source: AGHT+IHz0/iZAnfRiaUeJgoYtdpHEy5GFxUrXuxrhikbMPcC7Bwle7yCU/X9SrLM3+bghWb9T3SB5w== X-Received: by 2002:a17:90a:ce94:b0:2b1:73f8:a013 with SMTP id 98e67ed59e1d1-2b61639d6b2mr3773903a91.2.1715208139383; Wed, 08 May 2024 15:42:19 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.42.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:18 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 5/6] mm: swap: make should_try_to_free_swap() support large-folio Date: Thu, 9 May 2024 10:40:39 +1200 Message-Id: <20240508224040.190469-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages() =3D 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" Reviewed-by: David Hildenbrand --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index b51c059af0b0..d9434df24d62 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3877,7 +3877,7 @@ static inline bool should_try_to_free_swap(struct fol= io *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) =3D=3D 2; + folio_ref_count(folio) =3D=3D (1 + folio_nr_pages(folio)); } =20 static vm_fault_t pte_marker_clear(struct vm_fault *vmf) --=20 2.34.1 From nobody Sun Feb 8 11:48:29 2026 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1AA324B2F for ; Wed, 8 May 2024 22:42:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208148; cv=none; b=AMPbNLqQrQCBV8lFWDcojsWZTV+0vGhZndadmcUwDXvaPL9nL4Km9xhcD/fszlpE0U8SF3KplpXhBqYjtf6Z6adui1BDbYBD8QNOPAacuu0A1m5WyA3r67d4PTptFIK/nIjFKzbOIBUf2YgiDVEgoyXra9FO1yDQQl7aTMWIebw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715208148; c=relaxed/simple; bh=I8jcwKuouofcxZRKWJiNRwxuS2b/SayLDctNUASbQlg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WwYSdr93g+dnRBFD+JlpCrbHXICDr7PV29sM5FDl2C/MBRO5/iZEr610WE2MReWp0/Vppi/IZY3jR7JceoEYWhqBV+bGSSfbajkQElyuNHeqPmxjtJv07Cg5uZqOgzuyNm64NjLBFKwsUHSwsjadW0QjCRIZIDRa81UVIPcuXwY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WQE3uGsI; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WQE3uGsI" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-5ca29c131ebso200054a12.0 for ; Wed, 08 May 2024 15:42:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208146; x=1715812946; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=WQE3uGsIHml1bNsBtW475/i0Dc8Jib7QEkeLyk/B+x+fulOX3L+e8irr3NsbfqaI85 3GQpVDG8zTylpx97sKGtek0fYGi8LwhQBQ5Xy0Mr4W5IHz23duoS7yISot2HVDRDZKQz JymkLRYxTomAXC4kVdWW1LMyjozbxDDVmLVvQbA6y8t4t2eRsQzNNjQhAZtB17QERz5y R7GHBIMaTT0hKxHdiWcktxB79NPm98KJk1mTSjSfEpXiyCw1SXvb+TtDy1zqU3fGzHGa Qb7T+/q27WDm/u9I9/J78Vwqd4WSCnHLKBFbet0Gith2lxA1u5Bj5Qh7YvTUdVH+Wm50 4ooQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208146; x=1715812946; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=be9nw6LtWvWyvsOuBJNllB5gskOgt06aZtqgLJY5UciBAcrxJinHY9tLWN3Iv6Kzo2 EYSArkGy7khRVeeIJGREP0kSVZwm8hhp691zTRFBGnzu6sN9IDRdjPTwICFStRdSoKq4 uLF/P+zCOD2WCk3WDB2vu6cYiTjSyaVT791fJu8jliJm5pV7duglOGPqiV19PT1uu4/P yfDjWZLhwv6nCRb/ivIqeZoL9kDkEGF9XkWdmhrOwz+uwvb6jVVz2CmLetYEC9cLIMAK uZZLBCLQAEFQ7K8dad/HOPVEBZY/v4CL1BkPBOckOP1NxCAWO3SBeBtR7MhZbC7ictPd DrIw== X-Forwarded-Encrypted: i=1; AJvYcCV0U/0GKYNfwjlOtxu69SUWifcFaNU0X9y9UMqe09BXxjth4Eilry+KVmSiEuaFVJ76KAe20lS6aNeSxUipX9sfK25yPpym8JfH0/pY X-Gm-Message-State: AOJu0Ywpm28jtyrPPXMR+xDAJ0o8Vwp+dvCl1WLQGsP1ZatyDi9nvzHB ssgfN/OQWXm1So8ute2I+pe6xPpgl9IvIqfFTvzupseX/O1vAIDg X-Google-Smtp-Source: AGHT+IGq29rmz9yqnIyAWg43yTePii4lbT2fQtdkcVHel1ug/Dy/4QuBgQDgX9YahoaNoa9CKKLOjw== X-Received: by 2002:a17:90a:8811:b0:2b3:28df:92bb with SMTP id 98e67ed59e1d1-2b6165a62eemr3793409a91.13.1715208146212; Wed, 08 May 2024 15:42:26 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.42.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:25 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 6/6] mm: swap: entirely map large folios found in swapcache Date: Thu, 9 May 2024 10:40:40 +1200 Message-Id: <20240508224040.190469-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 59 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index d9434df24d62..8b9e4cab93ed 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret =3D 0; void *shadow =3D NULL; + int nr_pages; + unsigned long page_idx; + unsigned long address; + pte_t *ptep; =20 if (!pte_unmap_same(vmf)) goto out; @@ -4166,6 +4170,38 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } =20 + nr_pages =3D 1; + page_idx =3D 0; + address =3D vmf->address; + ptep =3D vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr =3D folio_nr_pages(folio); + unsigned long idx =3D folio_page_idx(folio, page); + unsigned long folio_start =3D address - idx * PAGE_SIZE; + unsigned long folio_end =3D folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end))) + goto check_folio; + + folio_ptep =3D vmf->pte - idx; + folio_pte =3D ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) !=3D nr) + goto check_folio; + + page_idx =3D idx; + address =3D folio_start; + ptep =3D folio_ptep; + nr_pages =3D nr; + entry =3D folio->swap; + page =3D &folio->page; + } + +check_folio: /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4261,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); =20 - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte =3D mk_pte(page, vma->vm_page_prot); =20 /* @@ -4247,27 +4283,28 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } rmap_flags |=3D RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + folio_ref_add(folio, nr_pages - 1); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte =3D pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte =3D pte_mkuffd_wp(pte); - vmf->orig_pte =3D pte; + vmf->orig_pte =3D pte_advance_pfn(pte, page_idx); =20 /* ksm created a completely new copy */ if (unlikely(folio !=3D swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } =20 VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { @@ -4291,7 +4328,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); --=20 2.34.1