From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7330920DD51 for ; Tue, 1 Sep 2026 05:44:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241461; cv=none; b=trOsnRGtlCaHLhFmdyL8UtXTv4YKRgqZ9ufoc4Da+Lyh4qLYKsfVwZUM+ZRWnhUsCrgdCp6m8xOnPJxq/GmI3D54BeMIPRq40jFUQvYgstILx0NF106mDnI8Np9OnuZRN04akLvXkRuUVO8US8s6OXsKriy83jPc+6NldGIW88I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241461; c=relaxed/simple; bh=ig7ZTYrfkjo7NzncPTcndZj/uDDh9JipwJASpm9qtRQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qFez2Vk+JxXZnNtUca87AnFPqjdN4PCqI6uxh8nJgvPwd1gJcHIuuPm5NqrO+iLIoj1VQ/GF9SwyxX5u+pYQ6TklSF63Syy4JHQX+8SK3QzT2RAQwAS+MxTuaVoutsdVWSFxUsTaYMlNxr6UPX9Z5DLU3BHEMYEdxpD7tXPIpL4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=TbN/jnTt; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="TbN/jnTt" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 75DBD1AED; Mon, 31 Aug 2026 22:44:14 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5E0893F85F; Mon, 31 Aug 2026 22:44:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241458; bh=ig7ZTYrfkjo7NzncPTcndZj/uDDh9JipwJASpm9qtRQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TbN/jnTt+Z5QixddOBXum3DzI2tgmG3n2627m9V+ny+gH5A+Z5kyLNEv8Llz7pZcA yYdms0gvFemF812GEqlwPInMFKiYyzF/9LfCQRigWvwhJRupHqMs6BPyu7Z3zkdyI5 XZnGhxgAubUd2LSoxxhUIREqTa3einEik3lSeLjA= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 1/8] mm/swapfile: add batched version of folio_dup_swap Date: Tue, 1 Sep 2026 05:43:48 +0000 Message-ID: <20260901054358.4049095-2-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add folio_dup_swap_pages to handle a batch of consecutive pages. Note that folio_dup_swap already can handle a subset of this: nr_pages =3D=3D 1 = and nr_pages =3D=3D folio_nr_pages(folio). Generalize this to any nr_pages. Currently we have a not-so-nice logic of passing in subpage =3D=3D NULL if we mean to exercise the logic on the entire folio, and subpage !=3D NULL if we want to exercise the logic on only that subpage. Remove this indirection: the caller invokes folio_dup_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 and folio_nr_pages()), and invokes folio_dup_swap() if it wants to operate on the entire folio. Acked-by: David Hildenbrand (Arm) Signed-off-by: Dev Jain Reviewed-by: Barry Song --- mm/rmap.c | 2 +- mm/shmem.c | 2 +- mm/swap.h | 18 ++++++++++++++++-- mm/swapfile.c | 16 ++++++++-------- 4 files changed, 26 insertions(+), 12 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 606c8ecf31302..0a4aa06b66de6 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2148,7 +2148,7 @@ static bool ttu_anon_swapbacked_folio(struct vm_area_= struct *vma, swp_entry_t entry =3D page_swap_entry(page); struct mm_struct *mm =3D vma->vm_mm; =20 - if (folio_dup_swap(folio, page) < 0) + if (folio_dup_swap_pages(folio, page, 1) < 0) return false; =20 /* diff --git a/mm/shmem.c b/mm/shmem.c index 897fa2b61346f..6cea018d69f04 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1712,7 +1712,7 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct fo= lio *folio, spin_unlock(&shmem_swaplist_lock); } =20 - folio_dup_swap(folio, NULL); + folio_dup_swap(folio); shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap)); =20 BUG_ON(folio_mapped(folio)); diff --git a/mm/swap.h b/mm/swap.h index 90a551a88df63..f52ecbedae8b8 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -244,7 +244,8 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gf= p_t gfp); * folio_put_swap(): does the opposite thing of folio_dup_swap(). */ int folio_alloc_swap(struct folio *folio); -int folio_dup_swap(struct folio *folio, struct page *page); +int folio_dup_swap_pages(struct folio *folio, struct page *page, + unsigned long nr_pages); void folio_put_swap(struct folio *folio, struct page *page); =20 /* For internal use */ @@ -368,7 +369,8 @@ static inline int folio_alloc_swap(struct folio *folio) return -EINVAL; } =20 -static inline int folio_dup_swap(struct folio *folio, struct page *page) +static inline int folio_dup_swap_pages(struct folio *folio, struct page *p= age, + unsigned long nr_pages) { return -EINVAL; } @@ -463,6 +465,18 @@ static inline void __swap_cache_replace_folio(struct s= wap_cluster_info *ci, } #endif /* CONFIG_SWAP */ =20 +/** + * folio_dup_swap() - Increase swap count of all swap entries of a folio. + * @folio: folio with swap entries bound. + * + * See folio_dup_swap_pages() for more information. + */ +static inline int folio_dup_swap(struct folio *folio) +{ + return folio_dup_swap_pages(folio, folio_page(folio, 0), + folio_nr_pages(folio)); +} + extern const struct swap_ops swap_bdev_ops; =20 int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, diff --git a/mm/swapfile.c b/mm/swapfile.c index 53bf01d5f7f11..25bb6fc562c57 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1785,9 +1785,10 @@ int folio_alloc_swap(struct folio *folio) } =20 /** - * folio_dup_swap() - Increase swap count of swap entries of a folio. + * folio_dup_swap_pages() - Increase swap count of swap entries of a folio. * @folio: folio with swap entries bounded. - * @page: if not NULL, only increase the swap count of this page. + * @page: the first page in the folio to increase the swap count for. + * @nr_pages: the number of pages in the folio to increase the swap count = for. * * Typically called when the folio is unmapped and have its swap entry to * take its place: Swap entries allocated to a folio has count =3D=3D 0 an= d pinned @@ -1801,18 +1802,17 @@ int folio_alloc_swap(struct folio *folio) * swap_put_entries_direct on its swap entry before this helper returns, or * the swap count may underflow. */ -int folio_dup_swap(struct folio *folio, struct page *page) +int folio_dup_swap_pages(struct folio *folio, struct page *page, + unsigned long nr_pages) { swp_entry_t entry =3D folio->swap; - unsigned long nr_pages =3D folio_nr_pages(folio); + unsigned long idx =3D folio_page_idx(folio, page); =20 VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio); + VM_WARN_ON_FOLIO(idx + nr_pages > folio_nr_pages(folio), folio); =20 - if (page) { - entry.val +=3D folio_page_idx(folio, page); - nr_pages =3D 1; - } + entry.val +=3D idx; =20 return swap_dup_entries_cluster(swap_entry_to_info(entry), swp_offset(entry), nr_pages); --=20 2.43.0 From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8101C38657E for ; Tue, 1 Sep 2026 05:44:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241469; cv=none; b=Rbx9ELyJfae7Bsf0y7hMb0qiCyR7Izpi2OkAgkuKiguJ4gX0jjfWqgrvRdUpfUFRmur1O9HL911UdfiM3RYa+jzk8MKyuYf27YoYvtRJrAtrCm76r2omtaEe16zrlOuo4MZ6x5JwKHQb4F7Zy5X0r9PCEQ11LKWqmynAQ/0XWUs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241469; c=relaxed/simple; bh=E2ktIYiViPu9MnUaU8AeLIAVoGUmKFh4+m0Y1CBWS4o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RdXEH/Ncs8AcrV5xRaOTEWXfEzEZUw3Rt6mW4G5IFK8okfxiSBJJsN4ZLA16GiscxO2LjA8Y9h6Sdtc7+wi6IxKMRmXCQfwStue0hdwhHvf/m4fyYkJjuhgIZpdMpVx92PWMEwTbwMyH23lriC3UGDvEITSlSP+He6aAMCxaszY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=BK9TIEMF; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="BK9TIEMF" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D70491756; Mon, 31 Aug 2026 22:44:21 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BF7153F85F; Mon, 31 Aug 2026 22:44:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241465; bh=E2ktIYiViPu9MnUaU8AeLIAVoGUmKFh4+m0Y1CBWS4o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BK9TIEMFCOEDM5tAyRTeOzg/0PPizZCgEpvn0On2eINATQ6RHoSeKLHTLRcmI4NGk VhIdCsAPgy2d2/fwOlTLKYpkwDAU69IXATxDssn/+V06oTCebY6DuU/YpX58ZU42MQ nYHTpOmHVwtSvuD0jemjfqhEb5b0A9nwHPq1z2Dk= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 2/8] mm/swapfile: add batched version of folio_put_swap Date: Tue, 1 Sep 2026 05:43:49 +0000 Message-ID: <20260901054358.4049095-3-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add folio_put_swap_pages to handle a batch of consecutive pages. Note that folio_put_swap already can handle a subset of this: nr_pages =3D=3D 1 = and nr_pages =3D=3D folio_nr_pages(folio). Generalize this to any nr_pages. Currently we have a not-so-nice logic of passing in subpage =3D=3D NULL if we mean to exercise the logic on the entire folio, and subpage !=3D NULL if we want to exercise the logic on only that subpage. Remove this indirection: the caller invokes folio_put_swap_pages() if it wants to operate on a range of pages in the folio (i.e nr_pages may be anything between 1 and folio_nr_pages()), and invokes folio_put_swap() if it wants to operate on the entire folio. Acked-by: David Hildenbrand (Arm) Signed-off-by: Dev Jain Reviewed-by: Barry Song --- mm/memory.c | 6 +++--- mm/rmap.c | 4 ++-- mm/shmem.c | 6 +++--- mm/swap.h | 17 +++++++++++++++-- mm/swapfile.c | 19 ++++++++++--------- 5 files changed, 33 insertions(+), 19 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8b0c2c735d3de..6cfea4faf0588 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5202,7 +5202,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(folio !=3D swapcache)) { folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE); folio_add_lru_vma(folio, vma); - folio_put_swap(swapcache, NULL); + folio_put_swap(swapcache); } else if (!folio_test_anon(folio)) { /* * We currently only expect !anon folios that are fully @@ -5211,12 +5211,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) VM_WARN_ON_ONCE_FOLIO(folio_nr_pages(folio) !=3D nr_pages, folio); VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio); folio_add_new_anon_rmap(folio, vma, address, rmap_flags); - folio_put_swap(folio, NULL); + folio_put_swap(folio); } else { VM_WARN_ON_ONCE(nr_pages !=3D 1 && nr_pages !=3D folio_nr_pages(folio)); folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); - folio_put_swap(folio, nr_pages =3D=3D 1 ? page : NULL); + folio_put_swap_pages(folio, page, nr_pages); } =20 VM_BUG_ON(!folio_test_anon(folio) || diff --git a/mm/rmap.c b/mm/rmap.c index 0a4aa06b66de6..1b9f07d4d1be9 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2157,13 +2157,13 @@ static bool ttu_anon_swapbacked_folio(struct vm_are= a_struct *vma, * so we'll not check/care. */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { - folio_put_swap(folio, page); + folio_put_swap_pages(folio, page, 1); return false; } =20 /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && folio_try_share_anon_rmap_pte(folio, page)) { - folio_put_swap(folio, page); + folio_put_swap_pages(folio, page, 1); return false; } =20 diff --git a/mm/shmem.c b/mm/shmem.c index 6cea018d69f04..316bf14136d63 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1733,7 +1733,7 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct fo= lio *folio, /* Swap entry might be erased by racing shmem_free_swap() */ if (!error) { shmem_recalc_inode(inode, 0, -nr_pages); - folio_put_swap(folio, NULL); + folio_put_swap(folio); } =20 /* @@ -2163,7 +2163,7 @@ static void shmem_set_folio_swapin_error(struct inode= *inode, pgoff_t index, =20 nr_pages =3D folio_nr_pages(folio); folio_wait_writeback(folio); - folio_put_swap(folio, NULL); + folio_put_swap(folio); swap_cache_del_folio(folio); /* * Don't treat swapin error folio as alloced. Otherwise inode->i_blocks @@ -2393,7 +2393,7 @@ static int shmem_swapin_folio(struct inode *inode, pg= off_t index, if (sgp =3D=3D SGP_WRITE) folio_mark_accessed(folio); =20 - folio_put_swap(folio, NULL); + folio_put_swap(folio); swap_cache_del_folio(folio); folio_mark_dirty(folio); put_swap_device(si); diff --git a/mm/swap.h b/mm/swap.h index f52ecbedae8b8..0e0d00bf4daa6 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -246,7 +246,8 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gf= p_t gfp); int folio_alloc_swap(struct folio *folio); int folio_dup_swap_pages(struct folio *folio, struct page *page, unsigned long nr_pages); -void folio_put_swap(struct folio *folio, struct page *page); +void folio_put_swap_pages(struct folio *folio, struct page *page, + unsigned long nr_pages); =20 /* For internal use */ extern void __swap_cluster_free_entries(struct swap_info_struct *si, @@ -375,7 +376,8 @@ static inline int folio_dup_swap_pages(struct folio *fo= lio, struct page *page, return -EINVAL; } =20 -static inline void folio_put_swap(struct folio *folio, struct page *page) +static inline void folio_put_swap_pages(struct folio *folio, struct page *= page, + unsigned long nr_pages) { } =20 @@ -477,6 +479,17 @@ static inline int folio_dup_swap(struct folio *folio) folio_nr_pages(folio)); } =20 +/** + * folio_put_swap() - Decrease swap count of all swap entries of a folio. + * @folio: folio with swap entries bound. + * + * See folio_put_swap_pages() for more information. + */ +static inline void folio_put_swap(struct folio *folio) +{ + folio_put_swap_pages(folio, folio_page(folio, 0), folio_nr_pages(folio)); +} + extern const struct swap_ops swap_bdev_ops; =20 int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio, diff --git a/mm/swapfile.c b/mm/swapfile.c index 25bb6fc562c57..da5e66bdcef10 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1819,27 +1819,27 @@ int folio_dup_swap_pages(struct folio *folio, struc= t page *page, } =20 /** - * folio_put_swap() - Decrease swap count of swap entries of a folio. + * folio_put_swap_pages() - Decrease swap count of swap entries of a folio. * @folio: folio with swap entries bounded, must be in swap cache and lock= ed. - * @page: if not NULL, only decrease the swap count of this page. + * @page: the first page in the folio to decrease the swap count for. + * @nr_pages: the number of pages in the folio to decrease the swap count = for. * * This won't free the swap slots even if swap count drops to zero, they a= re * still pinned by the swap cache. User may call folio_free_swap to free t= hem. * Context: Caller must ensure the folio is locked and in the swap cache. */ -void folio_put_swap(struct folio *folio, struct page *page) +void folio_put_swap_pages(struct folio *folio, struct page *page, + unsigned long nr_pages) { swp_entry_t entry =3D folio->swap; - unsigned long nr_pages =3D folio_nr_pages(folio); struct swap_info_struct *si =3D __swap_entry_to_info(entry); + unsigned long idx =3D folio_page_idx(folio, page); =20 VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio); + VM_WARN_ON_FOLIO(idx + nr_pages > folio_nr_pages(folio), folio); =20 - if (page) { - entry.val +=3D folio_page_idx(folio, page); - nr_pages =3D 1; - } + entry.val +=3D idx; =20 swap_put_entries_cluster(si, swp_offset(entry), nr_pages, false); } @@ -2518,7 +2518,8 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_= t *pmd, =20 setpte: set_pte_at(vma->vm_mm, addr, pte, new_pte); - folio_put_swap(swapcache, folio_file_page(swapcache, swp_offset(entry))); + folio_put_swap_pages(swapcache, + folio_file_page(swapcache, swp_offset(entry)), 1); out: if (pte) pte_unmap_unlock(pte, ptl); --=20 2.43.0 From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 155E9379C55 for ; Tue, 1 Sep 2026 05:44:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241476; cv=none; b=kMne53ZA4Vuli3Lz1oyXaV+XbcdcgCANneoGQyC5fAYQafGpn0lCwkbh62n7hf5Zxia7x7NGN+0HIwL8hhQFd4S9huyYYPZa7Gq9tkUBtVnnNcH+uA0Br7ZUKr1Fsig6NHl9e1iTI5gVJ8wS7/Ez0jQQqf+/+nkPF2XC/9Poa5I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241476; c=relaxed/simple; bh=4tS7tmRJt89rYLLLP6NkERN/c/Y1n2Ho9ctBU+vN0ME=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KVmEbmp8jvVBthzM+vbHZbDL8ChicL+D1/AXvdjBKWx9B4p95G+uLjHhcVJjqo3w1AoUoo0RNg3DDstJn4xvmfYgxAnI+6tz012pqzeAFJWDfmxnfwxaQzguq5x+mcv10YRDc5HEUZaOEBPNzNg1NcSS/ieXNq+xfEHyG+Ub3Ew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=oz2H0BCH; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="oz2H0BCH" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 434201BCA; Mon, 31 Aug 2026 22:44:29 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2C75C3F85F; Mon, 31 Aug 2026 22:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241473; bh=4tS7tmRJt89rYLLLP6NkERN/c/Y1n2Ho9ctBU+vN0ME=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oz2H0BCHqxgB8+EYjO1CA6z0w/3MTbwN/Bfl0F/wGW82MBejFyFiivi+atLNtZA95 HR7yQfq+oXKfD4NtrevQ8lbiL8cn7edktq/LjALdoUuLMssxPktPZQkewKoiMBiB0B zxaU7ipCnf3gZL3SU8D2mw2zk0i9x6NZJsLc3TzU= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 3/8] mm: move anon-exclusive batch helper to mm.h Date: Tue, 1 Sep 2026 05:43:50 +0000 Message-ID: <20260901054358.4049095-4-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for optimizing large folio unmapping, we need to reuse the page_anon_exclusive_batch helper in rmap.c and rmap.h and obey the existing use in mprotect.c . Therefore, move it from mprotect.c to mm.h. Gate with CONFIG_MMU since both rmap and mprotect users are built only for CONFIG_MMU. While at it, change start_idx and max_len to unsigned long type for future proofing against THP support at >=3D PUD level. Also shorten expected_anon_exclusive -> anon_exclusive. Signed-off-by: Dev Jain --- include/linux/mm.h | 19 +++++++++++++++++++ mm/mprotect.c | 17 ----------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dd09c438fa23e..63da8813bc5df 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -244,6 +244,25 @@ static inline unsigned long folio_page_idx(const struc= t folio *folio, return page - &folio->page; } =20 +#ifdef CONFIG_MMU +/* + * Get max length of consecutive PTEs pointing to PageAnonExclusive() page= s or + * !PageAnonExclusive() pages, starting from start_idx. Caller must enforce + * that the PTEs point to consecutive pages of the same anon large folio. + */ +static __always_inline int page_anon_exclusive_batch(unsigned long start_i= dx, + unsigned long max_len, struct page *first_page, bool anon_exclusive) +{ + int idx; + + for (idx =3D start_idx + 1; idx < start_idx + max_len; ++idx) { + if (anon_exclusive !=3D PageAnonExclusive(first_page + idx)) + break; + } + return idx - start_idx; +} +#endif + static inline struct folio *lru_to_folio(struct list_head *head) { return list_entry((head)->prev, struct folio, lru); diff --git a/mm/mprotect.c b/mm/mprotect.c index 2888ee638d872..75c2b1a00eb36 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -138,23 +138,6 @@ static __always_inline void prot_commit_flush_ptes(str= uct vm_area_struct *vma, tlb_flush_pte_range(tlb, addr, nr_ptes * PAGE_SIZE); } =20 -/* - * Get max length of consecutive ptes pointing to PageAnonExclusive() page= s or - * !PageAnonExclusive() pages, starting from start_idx. Caller must enforce - * that the ptes point to consecutive pages of the same anon large folio. - */ -static __always_inline int page_anon_exclusive_batch(int start_idx, int ma= x_len, - struct page *first_page, bool expected_anon_exclusive) -{ - int idx; - - for (idx =3D start_idx + 1; idx < start_idx + max_len; ++idx) { - if (expected_anon_exclusive !=3D PageAnonExclusive(first_page + idx)) - break; - } - return idx - start_idx; -} - /* * This function is a result of trying our very best to retain the * "avoid the write-fault handler" optimization. In can_change_pte_writabl= e(), --=20 2.43.0 From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 049F434214A for ; Tue, 1 Sep 2026 05:44:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241483; cv=none; b=ka/pzJmQPrgVjBdK4eHECqL2NxbK2FbqViIKVkqgvm7x093NhoXiWe9ozBZVVkxx+CBDbZHGIrLj8oxSr+8HxTD2qCmMVyhq5PzJnW8I4GyrbHOo3KwnnVQd0aSZfVYzfqnhlym3/GbhoN22zAtdFr0/Q1aKM8fG115TqBZMwT0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241483; c=relaxed/simple; bh=hmuhLEjycle0vItvKp8YgT1Zsj6d4IkLCimp6M+bfX8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jpQhF3QJ3WppPWOTzKRQV1rzDXfssn0K+2VoWtzd2P4A46fpe3MzQbpLydll8e3FEBot+KUHf9dRKMnFEvGnmfi+iNKXH4JeN5k0GtcigNBb8UNxeyrBgYk80KWrRXuQwhVOvxiI9X8+BsSop/9ZcC4/1GLLUDkwtQZqXOJvInk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=tWwefnfA; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="tWwefnfA" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A294E1756; Mon, 31 Aug 2026 22:44:36 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8DF4E3F85F; Mon, 31 Aug 2026 22:44:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241480; bh=hmuhLEjycle0vItvKp8YgT1Zsj6d4IkLCimp6M+bfX8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tWwefnfAhhBIpkRqqur17UuVtt3/sSCqdPq2EY+WY4FNf1DHC36JM1Q8qvEKzdYIh Gtxx0+HQorULV+ZQ69Khn+KIV3NV5xPZQSCirYE2riTZlphNNqZ8+sJv43mfhL+n4J lPNgmUXmQHJXdWSxXuSl0IY2Y38ckOUv1mhy+xQo= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 4/8] mm/rmap: Add batched version of folio_try_share_anon_rmap_pte Date: Tue, 1 Sep 2026 05:43:51 +0000 Message-ID: <20260901054358.4049095-5-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To enable batched unmapping of anonymous folios, we need to handle the sharing of exclusive pages. Hence, a batched version of folio_try_share_anon_rmap_pte is required. Currently, the sole purpose of nr_pages in __folio_try_share_anon_rmap is to do some rmap sanity checks. Now, clear the PageAnonExclusive bit on a batch of nr_pages. Refactor the function such that the clearing of the bit can be done at one place without duplication. Note that __folio_try_share_anon_rmap can receive nr_pages =3D=3D HPAGE_PMD= _NR from the PMD path, but currently we only clear the bit on the head page. Retain this behaviour by setting nr_pages =3D 1 in case the caller is folio_try_share_anon_rmap_pmd. While at it, convert nr_pages to unsigned long to future-proof from overflow in case P4D-huge mappings etc get supported down the road. I haven't made such a change in each function receiving nr_pages in try_to_unmap_one - perhaps this can be done incrementally. Add two WARN's: check that the batch is entirely exclusive (for PMD callers, need to check only head page), and that there are only PTE/PMD paths converging into __folio_try_share_anon_rmap. Signed-off-by: Dev Jain --- include/linux/rmap.h | 56 ++++++++++++++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 17 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0b332770abeed..320f9f14f6020 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -706,17 +706,23 @@ static inline int folio_try_dup_anon_rmap_pmd(struct = folio *folio, } =20 static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, - struct page *page, int nr_pages, enum pgtable_level level) + struct page *page, unsigned long nr_pages, enum pgtable_level level) { + /* device private folios cannot get pinned via GUP. */ + const bool pinnable =3D !folio_is_device_private(folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, level); =20 - /* device private folios cannot get pinned via GUP. */ - if (unlikely(folio_is_device_private(folio))) { - ClearPageAnonExclusive(page); - return 0; - } + VM_WARN_ON_ONCE(level > PGTABLE_LEVEL_PMD); + + /* We only clear anon-exclusive from head page of PMD folio. */ + if (level =3D=3D PGTABLE_LEVEL_PMD) + nr_pages =3D 1; + + VM_WARN_ON_FOLIO(page_anon_exclusive_batch(0, nr_pages, page, true) !=3D = nr_pages, folio); =20 /* * We have to make sure that when we clear PageAnonExclusive, that @@ -760,29 +766,38 @@ static __always_inline int __folio_try_share_anon_rma= p(struct folio *folio, * so we use explicit ones here. */ =20 - /* Paired with the memory barrier in try_grab_folio(). */ - if (IS_ENABLED(CONFIG_HAVE_GUP_FAST)) - smp_mb(); + if (likely(pinnable)) { + /* Paired with the memory barrier in try_grab_folio(). */ + if (IS_ENABLED(CONFIG_HAVE_GUP_FAST)) + smp_mb(); =20 - if (unlikely(folio_maybe_dma_pinned(folio))) - return -EBUSY; - ClearPageAnonExclusive(page); + if (unlikely(folio_maybe_dma_pinned(folio))) + return -EBUSY; + } + + for (;;) { + ClearPageAnonExclusive(page); + if (--nr_pages =3D=3D 0) + break; + page++; + } =20 /* * This is conceptually a smp_wmb() paired with the smp_rmb() in * gup_must_unshare(). */ - if (IS_ENABLED(CONFIG_HAVE_GUP_FAST)) + if (likely(pinnable) && IS_ENABLED(CONFIG_HAVE_GUP_FAST)) smp_mb__after_atomic(); return 0; } =20 /** - * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page - * mapped by a PTE possibly shared to prepare + * folio_try_share_anon_rmap_ptes - try marking exclusive anonymous pages + * mapped by PTEs possibly shared to prepare * for KSM or temporary unmapping * @folio: The folio to share a mapping of - * @page: The mapped exclusive page + * @page: The first mapped exclusive page of the batch in the folio + * @nr_pages: The number of pages to share in the folio (batch size) * * The caller needs to hold the page table lock and has to have the page t= able * entries cleared/invalidated. @@ -798,10 +813,17 @@ static __always_inline int __folio_try_share_anon_rma= p(struct folio *folio, * Returns 0 if marking the mapped page possibly shared succeeded. Returns * -EBUSY otherwise. */ +static inline int folio_try_share_anon_rmap_ptes(struct folio *folio, + struct page *page, unsigned long nr_pages) +{ + return __folio_try_share_anon_rmap(folio, page, nr_pages, + PGTABLE_LEVEL_PTE); +} + static inline int folio_try_share_anon_rmap_pte(struct folio *folio, struct page *page) { - return __folio_try_share_anon_rmap(folio, page, 1, PGTABLE_LEVEL_PTE); + return folio_try_share_anon_rmap_ptes(folio, page, 1); } =20 /** --=20 2.43.0 From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B56E734214A for ; Tue, 1 Sep 2026 05:44:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241490; cv=none; b=o75NkNz+YF7BPTtaIUEWfOeOeNJvs8y378fGc4s7Wbvw9AvRipT5ZvwHtqQxMWvBnA/gMKLWG0NxoC374+iSev7ErGAgGZrVm3Z7WOdAB9U2VMJDn9xaKqXl9g1F4ZUXrpXn4QdT+ns3OZslH2JGPr32C5zEIYUrz88DSfOWotk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241490; c=relaxed/simple; bh=BVSsEb8Y4xZRhSjlcLhU6nuyLKvRJakBNGn8zZZjAdc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VYBEW9fQLo+51DK/euwKm/pyRDEepJXonlIqoALgtkNfNPIpKvsYJgkEhqoHIUTQyDvll7aZUfd8FMNFN/7k2qLE5RHkIlP+DuGW+ls8k/Jv28WppaUtsIuIPh9Sn/ANUR9t2n9AhH1UxzY+1bgzteLL6RGK8LegZHXAWCkoGzI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=OAoPFr7R; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="OAoPFr7R" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0B91B1AED; Mon, 31 Aug 2026 22:44:44 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EBF573F85F; Mon, 31 Aug 2026 22:44:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241487; bh=BVSsEb8Y4xZRhSjlcLhU6nuyLKvRJakBNGn8zZZjAdc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OAoPFr7RxqkStsgeCzWgVlGw7svdApDqRN/OJtlKUF6uq62xSS8tpChKZ9r72yNBM JNM+uw/lf8lkFIZnS/YHdWkWq5kkT8UNU57mRx0n10nqWOm8xAJ5fgQELW4tf4mPP1 j01pd9nAh0f1UPXHsUU3/HoTniu443zYt6IzEM0o= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 5/8] mm/internal: rename swap offset helpers to softleaf offset Date: Tue, 1 Sep 2026 05:43:52 +0000 Message-ID: <20260901054358.4049095-6-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for adding a helper to set softleaf ptes in one go, generalize the swap entry helpers shifting the swap offset by delta, for softleaves. Note that the soft-dirty bit, exclusive bit and uffd bit preservation will still work for non-swap softleaves, since a softleaf entry is constructed out of a type and offset, and those bits are ahead of the soft-dirty, exclusive and uffd bits. For example, for a migration entry, pte_swp_exclusive() will return false, as the exclusivity is encoded in the type itself (SOFTLEAF_MIGRATION_READ_EXCLUSIVE). Signed-off-by: Dev Jain Reviewed-by: Barry Song --- mm/internal.h | 29 +++++++++++++++-------------- mm/memory.c | 4 ++-- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c94..cfea888366d58 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -468,16 +468,16 @@ unsigned int folio_pte_batch(struct folio *folio, pte= _t *ptep, pte_t pte, unsigned int max_nr); =20 /** - * pte_move_swp_offset - Move the swap entry offset field of a swap pte - * forward or backward by delta - * @pte: The initial pte state; must be a swap entry + * pte_move_softleaf_offset - Move the softleaf entry offset field of a + * softleaf pte forward or backward by delta + * @pte: The initial pte state; must be a softleaf entry * @delta: The direction and the offset we are moving; forward if delta * is positive; backward if delta is negative * - * Moves the swap offset, while maintaining all other fields, including - * swap type, and any swp pte bits. The resulting pte is returned. + * Moves the softleaf offset, while maintaining all other fields, including + * softleaf type, and any softleaf pte bits. The resulting pte is returned. */ -static inline pte_t pte_move_swp_offset(pte_t pte, long delta) +static inline pte_t pte_move_softleaf_offset(pte_t pte, long delta) { const softleaf_t entry =3D softleaf_from_pte(pte); pte_t new =3D __swp_entry_to_pte(__swp_entry(swp_type(entry), @@ -495,15 +495,16 @@ static inline pte_t pte_move_swp_offset(pte_t pte, lo= ng delta) =20 =20 /** - * pte_next_swp_offset - Increment the swap entry offset field of a swap p= te. - * @pte: The initial pte state; must be a swap entry. + * pte_next_softleaf_offset - Increment the softleaf entry offset field of= a + * non-present pte. + * @pte: The initial pte state; must be a softleaf entry. * - * Increments the swap offset, while maintaining all other fields, includi= ng - * swap type, and any swp pte bits. The resulting pte is returned. + * Increments the softleaf offset, while maintaining all other fields, inc= luding + * softleaf type, and any softleaf pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset(pte_t pte) +static inline pte_t pte_next_softleaf_offset(pte_t pte) { - return pte_move_swp_offset(pte, 1); + return pte_move_softleaf_offset(pte, 1); } =20 /** @@ -523,7 +524,7 @@ static inline pte_t pte_next_swp_offset(pte_t pte) */ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) { - pte_t expected_pte =3D pte_next_swp_offset(pte); + pte_t expected_pte =3D pte_next_softleaf_offset(pte); const pte_t *end_ptep =3D start_ptep + max_nr; pte_t *ptep =3D start_ptep + 1; =20 @@ -535,7 +536,7 @@ static inline int swap_pte_batch(pte_t *start_ptep, int= max_nr, pte_t pte) =20 if (!pte_same(pte, expected_pte)) break; - expected_pte =3D pte_next_swp_offset(expected_pte); + expected_pte =3D pte_next_softleaf_offset(expected_pte); ptep++; } =20 diff --git a/mm/memory.c b/mm/memory.c index 6cfea4faf0588..7ce8413a8117d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4757,7 +4757,7 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_= t *ptep, int nr_pages) idx =3D (vmf->address - addr) / PAGE_SIZE; pte =3D ptep_get(ptep); =20 - if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx))) + if (!pte_same(pte, pte_move_softleaf_offset(vmf->orig_pte, -idx))) return false; /* * swap_read_folio() can't handle the case a large folio is hybridly @@ -5066,7 +5066,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) =20 folio_ptep =3D vmf->pte - idx; folio_pte =3D ptep_get(folio_ptep); - if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + if (!pte_same(folio_pte, pte_move_softleaf_offset(vmf->orig_pte, -idx)) = || swap_pte_batch(folio_ptep, nr, folio_pte) !=3D nr) goto check_folio; =20 --=20 2.43.0 From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F126136D9FE for ; Tue, 1 Sep 2026 05:44:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241497; cv=none; b=SDivkDowxsiHKjdvJpjpjWgOd2u2oRxSRh7Ce1rJ4woeJFjD5bjToI9dUb/A41+uQxx60O90s5llDhJmqlnpHl9DpCqzL/0yUcpxqBed4W4zETuu0+4glJWMPFxC5ZgkfQFMCsWu1E2MlEnBhuauryC/IRB/ex1uTuHv2rcd074= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241497; c=relaxed/simple; bh=MUViQSYFM/j8MeFADam1qMeEw2VDwtqRRIgpPPWeUZQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dvNHN3V5FMR/h6fHOZBAUC9Z0kLnEcC8qFKvOyw4Cnmi/o7sfvDKMsrx9l5PtdnLNEJKYEySj6M4kJM8sgthiryFeN2Ke7em3lk+DXEdDNs8OrCVgIQ5jMICInTkYpn9xgMrMbF1CkrVMqFvnCd1PMjF8kyfC6luQYMgAzOobBg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=nBLhpdqq; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="nBLhpdqq" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6DA081756; Mon, 31 Aug 2026 22:44:51 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 559553F85F; Mon, 31 Aug 2026 22:44:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241495; bh=MUViQSYFM/j8MeFADam1qMeEw2VDwtqRRIgpPPWeUZQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nBLhpdqqTeiLBsehoAgcnw5ytfBWyQ0xm+6HJxkFcOWigjKxlOo00ljikMwNN901B cAROJ+KOhN61lBJivqsFVGsiHbEwUTULpVkJC3sQZ+/EcGS2ohMeraBMHRgGU5LuPz eXikSx8j70m4MiV72hE6xi7jYl9SlkGSd/YPikoI= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 6/8] mm/internal: add set_softleaf_ptes Date: Tue, 1 Sep 2026 05:43:53 +0000 Message-ID: <20260901054358.4049095-7-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently we have a helper called set_ptes() which is used to set consecutive present ptes in the pgtables. To do the same operation but to set "consecutive" nonpresent (softleaf) ptes, add set_softleaf_ptes(). The softleaves which have a notion of consecutivity is swap softleaf, and those grouped by softeaf_has_pfn(). The latter is trivial; future code can convert present ptes pointing to the same large folio to swap softleaves with consecutive offsets using this helper. The other case is softleaf markers. They do not have a notion of a swap offset or PFN, so future code can use set_softleaf_ptes() to store multiple (same) markers on the ptes. Signed-off-by: Dev Jain Reviewed-by: Barry Song --- mm/internal.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/mm/internal.h b/mm/internal.h index cfea888366d58..41cf8b9607e31 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -507,6 +507,39 @@ static inline pte_t pte_next_softleaf_offset(pte_t pte) return pte_move_softleaf_offset(pte, 1); } =20 +/** + * set_softleaf_ptes - Set consecutive softleaf PTEs. + * @mm: Address space the PTEs belong to. + * @addr: Address of the first PTE. + * @ptep: Page table pointer for the first PTE. + * @pte: PTE to set for the first entry. + * @nr: Number of PTEs to set. + * + * Install @nr softleaf PTEs, advancing @pte when its softleaf entry + * represents consecutive offsets. Swap entries advance through swap offse= ts, + * PFN softleaf entries advance through PFNs (encoded by swap offset), and + * marker entries are repeated unchanged. + */ +static inline void set_softleaf_ptes(struct mm_struct *mm, unsigned long a= ddr, + pte_t *ptep, pte_t pte, unsigned long nr) +{ + softleaf_t entry; + bool advance; + + entry =3D softleaf_from_pte(pte); + advance =3D softleaf_is_swap(entry) || softleaf_has_pfn(entry); + + for (;;) { + set_pte_at(mm, addr, ptep, pte); + if (--nr =3D=3D 0) + break; + if (advance) + pte =3D pte_next_softleaf_offset(pte); + ptep++; + addr +=3D PAGE_SIZE; + } +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. --=20 2.43.0 From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5729937F8DD for ; Tue, 1 Sep 2026 05:45:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241504; cv=none; b=EHgDKoxOTfMu5b4+DLWZcRx1tIMcUGB773unYCMnuMuYC1pPuNM1yGbbmn17S6vdCyIv2l69UaZUad79ohPoVy0VkOONEaWaYb2IScHuqKkTbF13//vs9jSoeDOgY2bn9HK7/tLZYqejnFFQK3d73Ih7aGxwiLquJmluaA7Jwl8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241504; c=relaxed/simple; bh=tmJo3rjw0Sr3OV0lC41Li6x3+cmR1t0LVTih0JfnMfU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Re3g/+vamH8UJeOt4d/2U0t3uyKx7gKKO1psFX2PaS/11q1Qwv2LaKY67f+UvO0uZx1CCAzZ/jA5GHZdi2M8XQNSTbobICY4ZkOCMzqG0pk37EF2dvbjDZJKRrjbZOAASKXGX4fVcMRVk+y5HALO7Xtdrc3AxR1HnHvyYpv6bHA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=mxQNKj4W; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="mxQNKj4W" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D512F1AED; Mon, 31 Aug 2026 22:44:58 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B6B9A3F85F; Mon, 31 Aug 2026 22:44:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241502; bh=tmJo3rjw0Sr3OV0lC41Li6x3+cmR1t0LVTih0JfnMfU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mxQNKj4Wb0AEhmFu9i0suaBsQYFruZECm4SWTseJP7rcY2W3tIcJc2YI9lEycAGbN ZsPZbxPAQokrm2WN4mK1Ud3PO6iU7lcQ+kcXpB4dNkc5H19R67JsXBZnnUMZTQD+38 NeY2MAeNqKc1GX+dQ6Zfqt+VNWjlIMJLzoHmT2n0= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 7/8] mm/memory: use set_softleaf_ptes for uffd-wp markers Date: Tue, 1 Sep 2026 05:43:54 +0000 Message-ID: <20260901054358.4049095-8-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use set_softleaf_ptes() to store multiple uffd-wp marker entries instead of open coding the batched setting. Signed-off-by: Dev Jain Reviewed-by: Barry Song --- mm/memory.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7ce8413a8117d..8c45fb25fd8dd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1737,14 +1737,8 @@ bool cond_install_uffd_wp_ptes(struct vm_area_struct= *vma, if (likely(!arm_uffd_pte)) return false; =20 - for (;;) { - set_pte_at(vma->vm_mm, addr, ptep, - make_pte_marker(PTE_MARKER_UFFD_WP)); - if (--nr_ptes =3D=3D 0) - break; - ptep++; - addr +=3D PAGE_SIZE; - } + set_softleaf_ptes(vma->vm_mm, addr, ptep, + make_pte_marker(PTE_MARKER_UFFD_WP), nr_ptes); =20 return true; } --=20 2.43.0 From nobody Sat Sep 26 13:46:49 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B587839F17F for ; Tue, 1 Sep 2026 05:45:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241513; cv=none; b=KDpT5arnOTydG59hQn4AOO3h48Y7KY3tIEcoGqD1rqavZCBAnMHRqX3ziEEI4+qwRPYMqGvwN1FA2MivMtKcXRVYGAOtVGeoUhkgAIv6yVNdax9+jZ13gK5t8cL1huFV83XxHyKjhu0RrQsg/orZ55z11wtIJ/r/qJpRk0kKNM8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788241513; c=relaxed/simple; bh=ExzsBBQQzRLXmN5iM7ua3O1UxwNf7wasB8hjbJf/zWo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WW9x7QyK/TVGHwkngibkiOjypoFp9+MPwe5ZD7VzaJ42FwPM8qsMj8fKpvvr6zSFbMyYJHVYyAO+WsJ0PfTgrII2X+cZl6fUoiWqYR1geVe3WI8ZivrmdB7AJSMjhbFEedpTz0Mh3LQ2fQKl/d2sjjv2ePoRoZTyYJZBGJcXrio= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=lIqXMwdd; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="lIqXMwdd" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 445F61756; Mon, 31 Aug 2026 22:45:06 -0700 (PDT) Received: from cesw-amp-gbt-1s-m12830-01.blr.arm.com (cesw-amp-gbt-1s-m12830-01.blr.arm.com [10.164.195.33]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2B6443F85F; Mon, 31 Aug 2026 22:45:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1788241510; bh=ExzsBBQQzRLXmN5iM7ua3O1UxwNf7wasB8hjbJf/zWo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lIqXMwddZi+uOm9ZJisAjVtSPICoXG4TgBYeL3OwVXiBe3YuDLo86lqjl4CnIXSQu hAAQNs6LM9tL+uABixrKnhZhHejKRN55E0CZS3bV2ebPGFWeTIcFC7IPuVMOaHGnxD vX99fqxU1f1+wtDyVO5ioGry1vQqF96Jso7B3QCw= From: Dev Jain To: akpm@linux-foundation.org, david@kernel.org, ljs@kernel.org, hughd@google.com, chrisl@kernel.org, kasong@tencent.com Cc: Dev Jain , riel@surriel.com, liam@infradead.org, vbabka@kernel.org, harry@kernel.org, jannh@google.com, lance.yang@linux.dev, baolin.wang@linux.alibaba.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, baoquan.he@linux.dev, baohua@kernel.org, youngjun.park@lge.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, pfalcato@suse.de, ryan.roberts@arm.com, anshuman.khandual@arm.com Subject: [PATCH v2 8/8] mm/rmap: batch unmap anonymous swap-backed large folios Date: Tue, 1 Sep 2026 05:43:55 +0000 Message-ID: <20260901054358.4049095-9-dev.jain@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260901054358.4049095-1-dev.jain@arm.com> References: <20260901054358.4049095-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable batch clearing of ptes, and batch swap setting of ptes for anon swap-backed folio unmapping. Processing all ptes of a large folio in one go helps us batch across atomics (add_mm_counter etc), barriers (in the function __folio_try_share_anon_rmap), repeated calls to page_vma_mapped_walk(), to name a few. In general, batching helps us to execute similar code together, making the execution of the program more memory and CPU friendly. On arm64-contpte, batching also helps us avoid redundant ptep_get() calls and TLB flushes while breaking the contpte mapping. The handling of anon-exclusivity is very similar to commit cac1db8c3aad ("mm: optimize mprotect() by PTE batching"). Since folio_unmap_pte_batch() won't look at the bits of the underlying page, we need to process sub-batches of ptes pointing to pages which are same w.r.t exclusivity, and batch set only those ptes to swap ptes in one go. arch_unmap_one() is only defined for sparc64; I am not comfortable regarding the nuances between retrieving the pfn from pte_pfn() or from (paddr =3D pte_val(oldpte) & _PAGE_PADDR_4V). (And, pte_next_pfn() can't even be called from arch_unmap_one() because that file does not include pgtable.h) So just disable the "sparc64-anon-swapbacked" case for now. We need to take care of rmap accounting (folio_remove_rmap_ptes) and reference accounting (folio_put_refs) when anon folio unmap succeeds. In case we partially batch the large folio and fail, we need to correctly do the accounting for pages which were successfully unmapped. So, put this accounting code (which is finish_folio_unmap()) in __ttu_anon_swapbacked_folio() itself, instead of doing some horrible goto jumping at the callsite of ttu_anon_folio(). Similarly, do the finish_folio_unmap() in ttu_anon_folio itself for the non-swapbacked (lazyfree) case. If the batch length is less than the number of pages in the folio, then we must skip over this batch. The page_vma_mapped_walk API ensures this - check_pte() will return true only if any of [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the pte. There is no pfn underlying a swap pte, so check_pte returns false and we keep skipping until we hit a present pte, which is where we want to start unmapping from next. Remove the label finish_unmap since no goto callers are left now. Signed-off-by: Dev Jain --- mm/rmap.c | 110 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 81 insertions(+), 29 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1b9f07d4d1be9..68e0201ffd003 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1964,12 +1964,14 @@ static inline unsigned int folio_unmap_pte_batch(st= ruct folio *folio, end_addr =3D pmd_addr_end(addr, vma->vm_end); max_nr =3D (end_addr - addr) >> PAGE_SHIFT; =20 - /* We only support lazyfree or file folios batching for now ... */ - if (folio_test_anon(folio) && folio_test_swapbacked(folio)) + if (pte_unused(pte)) return 1; =20 - if (pte_unused(pte)) +#ifdef __HAVE_ARCH_UNMAP_ONE + /* Add batching support to arch_unmap_one() to remove this */ + if (folio_test_anon(folio) && folio_test_swapbacked(folio)) return 1; +#endif =20 /* * If unmap fails, we need to restore the ptes. To avoid accidentally @@ -2139,16 +2141,25 @@ static pte_t swp_pte_prepare(swp_entry_t entry, pte= _t old_pte, return swp_pte; } =20 -static bool ttu_anon_swapbacked_folio(struct vm_area_struct *vma, +static void finish_folio_unmap(struct vm_area_struct *vma, + struct folio *folio, struct page *page, unsigned long nr_pages) +{ + folio_remove_rmap_ptes(folio, page, nr_pages, vma); + if (vma->vm_flags & VM_LOCKED) + mlock_drain_local(); + folio_put_refs(folio, nr_pages); +} + +static bool __ttu_anon_swapbacked_folio(struct vm_area_struct *vma, struct folio *folio, struct page *page, unsigned long address, - pte_t *ptep, pte_t pteval) + pte_t *ptep, pte_t pteval, unsigned long nr_pages, + bool anon_exclusive) { - const bool anon_exclusive =3D folio_test_anon(folio) && - PageAnonExclusive(page); swp_entry_t entry =3D page_swap_entry(page); struct mm_struct *mm =3D vma->vm_mm; + pte_t swp_pte; =20 - if (folio_dup_swap_pages(folio, page, 1) < 0) + if (folio_dup_swap_pages(folio, page, nr_pages) < 0) return false; =20 /* @@ -2157,21 +2168,57 @@ static bool ttu_anon_swapbacked_folio(struct vm_are= a_struct *vma, * so we'll not check/care. */ if (arch_unmap_one(mm, vma, address, pteval) < 0) { - folio_put_swap_pages(folio, page, 1); + VM_WARN_ON(nr_pages !=3D 1); + folio_put_swap_pages(folio, page, nr_pages); return false; } =20 /* See folio_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && folio_try_share_anon_rmap_pte(folio, page)) { - folio_put_swap_pages(folio, page, 1); + if (anon_exclusive && + folio_try_share_anon_rmap_ptes(folio, page, nr_pages)) { + folio_put_swap_pages(folio, page, nr_pages); return false; } =20 mm_prepare_for_swap_entries(mm); - dec_mm_counter(mm, MM_ANONPAGES); - inc_mm_counter(mm, MM_SWAPENTS); - set_pte_at(mm, address, ptep, - swp_pte_prepare(entry, pteval, anon_exclusive)); + add_mm_counter(mm, MM_ANONPAGES, -nr_pages); + add_mm_counter(mm, MM_SWAPENTS, nr_pages); + swp_pte =3D swp_pte_prepare(entry, pteval, anon_exclusive); + set_softleaf_ptes(mm, address, ptep, swp_pte, nr_pages); + finish_folio_unmap(vma, folio, page, nr_pages); + return true; +} + +static bool ttu_anon_swapbacked_folio(struct vm_area_struct *vma, + struct folio *folio, struct page *first_page, + unsigned long address, pte_t *ptep, pte_t pteval, + unsigned long nr_pages) +{ + unsigned long batch_idx =3D 0; + + while (nr_pages) { + bool anon_exclusive =3D PageAnonExclusive(first_page + batch_idx); + unsigned long len =3D page_anon_exclusive_batch(batch_idx, + nr_pages, first_page, anon_exclusive); + + if (!__ttu_anon_swapbacked_folio(vma, folio, + first_page + batch_idx, address, ptep, pteval, + len, anon_exclusive)) { + /* Restore the remaining PTEs that were cleared. */ + set_ptes(vma->vm_mm, address, ptep, pteval, nr_pages); + return false; + } + + nr_pages -=3D len; + if (!nr_pages) + break; + + pteval =3D pte_advance_pfn(pteval, len); + address +=3D len * PAGE_SIZE; + batch_idx +=3D len; + ptep +=3D len; + } + return true; } =20 @@ -2184,15 +2231,22 @@ static bool ttu_anon_folio(struct vm_area_struct *v= ma, struct folio *folio, * See handle_pte_fault() ... */ if (WARN_ON_ONCE(folio_test_swapbacked(folio) !=3D - folio_test_swapcache(folio))) + folio_test_swapcache(folio))) { + set_ptes(vma->vm_mm, address, ptep, pteval, nr_pages); return false; + } =20 - if (!folio_test_swapbacked(folio)) - return ttu_anon_lazyfree_folio(vma, folio, nr_pages); + if (!folio_test_swapbacked(folio)) { + if (!ttu_anon_lazyfree_folio(vma, folio, nr_pages)) { + set_ptes(vma->vm_mm, address, ptep, pteval, nr_pages); + return false; + } + finish_folio_unmap(vma, folio, page, nr_pages); + return true; + } =20 - /* nr_pages > 1 not supported yet */ return ttu_anon_swapbacked_folio(vma, folio, page, address, ptep, - pteval); + pteval, nr_pages); } =20 /* @@ -2370,13 +2424,14 @@ static bool try_to_unmap_one(struct folio *folio, s= truct vm_area_struct *vma, */ dec_mm_counter(mm, mm_counter(folio)); } else if (folio_test_anon(folio)) { + /* finish_folio_unmap handled internally */ if (!ttu_anon_folio(vma, folio, page, address, - pvmw.pte, pteval, nr_pages)) { - set_ptes(mm, address, pvmw.pte, pteval, nr_pages); + pvmw.pte, pteval, nr_pages)) goto walk_abort; - } =20 - goto finish_unmap; + if (nr_pages =3D=3D folio_nr_pages(folio)) + goto walk_done; + continue; } else { /* * This is a locked file-backed folio, @@ -2391,11 +2446,8 @@ static bool try_to_unmap_one(struct folio *folio, st= ruct vm_area_struct *vma, */ add_mm_counter(mm, mm_counter_file(folio), -nr_pages); } -finish_unmap: - folio_remove_rmap_ptes(folio, page, nr_pages, vma); - if (vma->vm_flags & VM_LOCKED) - mlock_drain_local(); - folio_put_refs(folio, nr_pages); + + finish_folio_unmap(vma, folio, page, nr_pages); =20 /* * If we are sure that we batched the entire folio and cleared --=20 2.43.0