From nobody Sat Feb 7 17:04:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03D74EB64DC for ; Mon, 3 Jul 2023 13:53:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230030AbjGCNx4 (ORCPT ); Mon, 3 Jul 2023 09:53:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230252AbjGCNxw (ORCPT ); Mon, 3 Jul 2023 09:53:52 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 0716010C8 for ; Mon, 3 Jul 2023 06:53:45 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 71661C14; Mon, 3 Jul 2023 06:54:27 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EDAF83F73F; Mon, 3 Jul 2023 06:53:42 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 1/5] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap() Date: Mon, 3 Jul 2023 14:53:26 +0100 Message-Id: <20230703135330.1865927-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In preparation for FLEXIBLE_THP support, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme. Signed-off-by: Ryan Roberts Reviewed-by: Yin, Fengwei Reviewed-by: Yu Zhao Suggested-by. --- mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1d8369549424..82ef5ba363d1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct v= m_area_struct *vma, * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * - * If the folio is large, it is accounted as a THP. As the folio + * If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *v= ma, unsigned long address) { - int nr; + int nr =3D folio_nr_pages(folio); + int i; + struct page *page; =20 - VM_BUG_ON_VMA(address < vma->vm_start || address >=3D vma->vm_end, vma); + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); =20 - if (likely(!folio_test_pmd_mappable(folio))) { + if (!folio_test_large(folio)) { /* increment count (starts at -1) */ atomic_set(&folio->_mapcount, 0); - nr =3D 1; + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); + } else if (!folio_test_pmd_mappable(folio)) { + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); + + page =3D &folio->page; + for (i =3D 0; i < nr; i++, page++, address +=3D PAGE_SIZE) { + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, address, 1); + } } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); - nr =3D folio_nr_pages(folio); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } =20 __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); - __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } =20 /** --=20 2.25.1 From nobody Sat Feb 7 17:04:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36CDDEB64DC for ; Mon, 3 Jul 2023 13:54:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231509AbjGCNyA (ORCPT ); Mon, 3 Jul 2023 09:54:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229849AbjGCNx4 (ORCPT ); Mon, 3 Jul 2023 09:53:56 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EEBF310D9 for ; Mon, 3 Jul 2023 06:53:46 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6A42A1424; Mon, 3 Jul 2023 06:54:29 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E6A393F73F; Mon, 3 Jul 2023 06:53:44 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios Date: Mon, 3 Jul 2023 14:53:27 +0100 Message-Id: <20230703135330.1865927-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei Suggested-by. --- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index 82ef5ba363d1..bbcb2308a1c5 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_ar= ea_struct *vma, * page of the folio is unmapped and at least one page * is still mapped. */ - if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) + if (folio_test_large(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) deferred_split_folio(folio); } --=20 2.25.1 From nobody Sat Feb 7 17:04:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CACCEB64DC for ; Mon, 3 Jul 2023 13:54:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229981AbjGCNyJ (ORCPT ); Mon, 3 Jul 2023 09:54:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230037AbjGCNyF (ORCPT ); Mon, 3 Jul 2023 09:54:05 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7B94910FA for ; Mon, 3 Jul 2023 06:53:49 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6399A11FB; Mon, 3 Jul 2023 06:54:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DF28C3F73F; Mon, 3 Jul 2023 06:53:46 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order() Date: Mon, 3 Jul 2023 14:53:28 +0100 Message-Id: <20230703135330.1865927-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" arch_wants_pte_order() can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous. The first user for this hint will be FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs. Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K. Signed-off-by: Ryan Roberts Suggested-by. --- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a661a17173fa..f7e38598f20b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -13,6 +13,7 @@ #include #include #include +#include =20 #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) = - \ defined(__PAGETABLE_PMD_FOLDED) !=3D CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) } #endif =20 +#ifndef arch_wants_pte_order +/* + * Returns preferred folio order for pte-mapped memory. Must be in range [= 0, + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large = folios + * to be at least order-2. + */ +static inline int arch_wants_pte_order(struct vm_area_struct *vma) +{ + return ilog2(SZ_64K >> PAGE_SHIFT); +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, --=20 2.25.1 From nobody Sat Feb 7 17:04:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68AA1EB64DC for ; Mon, 3 Jul 2023 13:54:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231157AbjGCNyO (ORCPT ); Mon, 3 Jul 2023 09:54:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230262AbjGCNyL (ORCPT ); Mon, 3 Jul 2023 09:54:11 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5971E1706 for ; Mon, 3 Jul 2023 06:53:51 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 76F33143D; Mon, 3 Jul 2023 06:54:33 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D88B03F73F; Mon, 3 Jul 2023 06:53:48 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance Date: Mon, 3 Jul 2023 14:53:29 +0100 Message-Id: <20230703135330.1865927-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Introduce FLEXIBLE_THP feature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio. The new behaviour is hidden behind the new FLEXIBLE_THP Kconfig, which defaults to disabled for now; there is a long list of todos to make FLEXIBLE_THP robust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches. When enabled, the preferred folio order is as returned by arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required. If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order. Signed-off-by: Ryan Roberts Suggested-by. --- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 7672a22647b4..1c06b2c0a24e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. =20 +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE =20 # diff --git a/mm/memory.c b/mm/memory.c index fb30f7523550..abe2ea94f3f5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm= _fault *vmf) return 0; } =20 +#ifdef CONFIG_FLEXIBLE_THP +/* + * Allocates, zeros and returns a folio of the requested order for use as + * anonymous memory. + */ +static struct folio *alloc_anon_folio(struct vm_area_struct *vma, + unsigned long addr, int order) +{ + gfp_t gfp; + struct folio *folio; + + if (order =3D=3D 0) + return vma_alloc_zeroed_movable_folio(vma, addr); + + gfp =3D vma_thp_gfp_mask(vma); + folio =3D vma_alloc_folio(gfp, order, vma, addr, true); + if (folio) + clear_huge_page(&folio->page, addr, folio_nr_pages(folio)); + + return folio; +} + +/* + * Preferred folio order to allocate for anonymous memory. + */ +#define max_anon_folio_order(vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio(vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order(vma) 0 +#endif + +/* + * Returns index of first pte that is not none, or nr if all are none. + */ +static inline int check_ptes_none(pte_t *pte, int nr) +{ + int i; + + for (i =3D 0; i < nr; i++) { + if (!pte_none(ptep_get(pte++))) + return i; + } + + return nr; +} + +static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order) +{ + /* + * The aim here is to determine what size of folio we should allocate + * for this fault. Factors include: + * - Order must not be higher than `order` upon entry + * - Folio must be naturally aligned within VA space + * - Folio must be fully contained inside one pmd entry + * - Folio must not breach boundaries of vma + * - Folio must not overlap any non-none ptes + * + * Additionally, we do not allow order-1 since this breaks assumptions + * elsewhere in the mm; THP pages must be at least order-2 (since they + * store state up to the 3rd struct page subpage), and these pages must + * be THP in order to correctly use pre-existing THP infrastructure such + * as folio_split(). + * + * Note that the caller may or may not choose to lock the pte. If + * unlocked, the result is racy and the user must re-check any overlap + * with non-none ptes under the lock. + */ + + struct vm_area_struct *vma =3D vmf->vma; + int nr; + unsigned long addr; + pte_t *pte; + pte_t *first_set =3D NULL; + int ret; + + order =3D min(order, PMD_SHIFT - PAGE_SHIFT); + + for (; order > 1; order--) { + nr =3D 1 << order; + addr =3D ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT); + pte =3D vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT); + + /* Check vma bounds. */ + if (addr < vma->vm_start || + addr + (nr << PAGE_SHIFT) > vma->vm_end) + continue; + + /* Ptes covered by order already known to be none. */ + if (pte + nr <=3D first_set) + break; + + /* Already found set pte in range covered by order. */ + if (pte <=3D first_set) + continue; + + /* Need to check if all the ptes are none. */ + ret =3D check_ptes_none(pte, nr); + if (ret =3D=3D nr) + break; + + first_set =3D pte + ret; + } + + if (order =3D=3D 1) + order =3D 0; + + return order; +} + /* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom; =20 if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { - new_folio =3D vma_alloc_zeroed_movable_folio(vma, vmf->address); + new_folio =3D alloc_anon_folio(vma, vmf->address, 0); if (!new_folio) goto oom; } else { @@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *= vmf) struct folio *folio; vm_fault_t ret =3D 0; pte_t entry; + int order; + int pgcount; + unsigned long addr; =20 /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) @@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault= *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_MISSING); } - goto setpte; + if (uffd_wp) + entry =3D pte_mkuffd_wp(entry); + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, vmf->address, vmf->pte); + goto unlock; + } + + /* + * If allocating a large folio, determine the biggest suitable order for + * the VMA (e.g. it must not exceed the VMA's bounds, it must not + * overlap with any populated PTEs, etc). We are not under the ptl here + * so we will need to re-check that we are not overlapping any populated + * PTEs once we have the lock. + */ + order =3D uffd_wp ? 0 : max_anon_folio_order(vma); + if (order > 0) { + vmf->pte =3D pte_offset_map(vmf->pmd, vmf->address); + order =3D calc_anon_folio_order_alloc(vmf, order); + pte_unmap(vmf->pte); } =20 - /* Allocate our own private page. */ + /* Allocate our own private folio. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio =3D vma_alloc_zeroed_movable_folio(vma, vmf->address); + folio =3D alloc_anon_folio(vma, vmf->address, order); + if (!folio && order > 0) { + order =3D 0; + folio =3D alloc_anon_folio(vma, vmf->address, order); + } if (!folio) goto oom; =20 + pgcount =3D 1 << order; + addr =3D ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT); + if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) goto oom_free_page; folio_throttle_swaprate(folio, GFP_KERNEL); =20 /* * The memory barrier inside __folio_mark_uptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. + * preceding stores to the folio contents become visible before + * the set_ptes() write. */ __folio_mark_uptodate(folio); =20 @@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault= *vmf) if (vma->vm_flags & VM_WRITE) entry =3D pte_mkwrite(pte_mkdirty(entry)); =20 - vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); if (vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto release; + } else if (order > 0 && check_ptes_none(vmf->pte, pgcount) !=3D pgcount) { + goto release; } =20 ret =3D check_stable_address_space(vma->vm_mm); @@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault= *vmf) return handle_userfault(vmf, VM_UFFD_MISSING); } =20 - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_ref_add(folio, pgcount - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount); + folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); -setpte: + if (uffd_wp) entry =3D pte_mkuffd_wp(entry); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount); =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vma, addr, vmf->pte, pgcount); unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; --=20 2.25.1 From nobody Sat Feb 7 17:04:40 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2A35EB64DC for ; Mon, 3 Jul 2023 13:54:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230262AbjGCNyY (ORCPT ); Mon, 3 Jul 2023 09:54:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230094AbjGCNyQ (ORCPT ); Mon, 3 Jul 2023 09:54:16 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 38C171716 for ; Mon, 3 Jul 2023 06:53:53 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6FD552F4; Mon, 3 Jul 2023 06:54:35 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EBD643F73F; Mon, 3 Jul 2023 06:53:50 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 5/5] arm64: mm: Override arch_wants_pte_order() Date: Mon, 3 Jul 2023 14:53:30 +0100 Message-Id: <20230703135330.1865927-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Define an arch-specific override of arch_wants_pte_order() so that when FLEXIBLE_THP is enabled, large folios will be allocated for anonymous memory with an order that is compatible with arm64's contpte mappings. arch_wants_pte_order() returns an order according to the following policy: For the unhinted case, when THP is not requested for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is requested for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K. Additionally, the THP and NOTHP order constants are defined using Kconfig so it is possible to override them at build time. Signed-off-by: Ryan Roberts Suggested-by. --- arch/arm64/Kconfig | 12 ++++++++++++ arch/arm64/include/asm/pgtable.h | 4 ++++ arch/arm64/mm/mmu.c | 8 ++++++++ 3 files changed, 24 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 343e1e1cae10..689c5bf13dc1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -281,6 +281,18 @@ config ARM64_CONT_PMD_SHIFT default 5 if ARM64_16K_PAGES default 4 =20 +config ARM64_PTE_ORDER_NOTHP + int + default 0 if ARM64_64K_PAGES # 64K (1 page) + default 2 if ARM64_16K_PAGES # 64K (4 pages; benefits from HPA where HW s= upports it) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + +config ARM64_PTE_ORDER_THP + int + default 5 if ARM64_64K_PAGES # 2M (32 pages; eligible for contpte-mappin= g) + default 7 if ARM64_16K_PAGES # 2M (128 pages; eligible for contpte-mappi= ng) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + config ARCH_MMAP_RND_BITS_MIN default 14 if ARM64_64K_PAGES default 16 if ARM64_16K_PAGES diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 6fd012663a01..8463d5f9f307 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1117,6 +1117,10 @@ extern pte_t ptep_modify_prot_start(struct vm_area_s= truct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +#define arch_wants_pte_order arch_wants_pte_order +extern int arch_wants_pte_order(struct vm_area_struct *vma); + #endif /* !__ASSEMBLY__ */ =20 #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index af6bc8403ee4..8556c4a9b507 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1481,3 +1481,11 @@ void ptep_modify_prot_commit(struct vm_area_struct *= vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +int arch_wants_pte_order(struct vm_area_struct *vma) +{ + if (hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + return CONFIG_ARM64_PTE_ORDER_THP; + else + return CONFIG_ARM64_PTE_ORDER_NOTHP; +} --=20 2.25.1