From nobody Mon Oct 6 10:14:39 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 136731E8320 for ; Tue, 22 Jul 2025 15:06:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753196783; cv=none; b=IrDqeGrPbiZvppo7D7n6upqKSY03Fl9L2w0vvqY+SCppp10SXcH3GgWyc9744dBSTYKVDBx5IORkf2QyGCPRxdp6xpOxqXDFjiRrLXCz5qxQag4VbwebfR9PXyE1IAgrU/z/U9kehA28760u8fZpnqiGRouSzsO5Jhvl76YC844= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753196783; c=relaxed/simple; bh=vsN6vLgauspchfLTpWwqdbhKptawSix1hRqJ2Vks3iA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oDjOSX6F7RKSlQdpW7XrUKDQtdBL0FnbS9O0AbJjF9zRXW1SRQ7Y71s7V5Vz116Og1taMFAPVTaAF8qWfm7/n6O5pypLiyJAGPWaXRkOx07vzIpOw91My95bGrCoZmbhA7y+CiaPn0zALZrIoW4ysKR6ieXgPNK1LqLSk9pXclA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BD631152B; Tue, 22 Jul 2025 08:06:14 -0700 (PDT) Received: from localhost.localdomain (unknown [10.163.92.223]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 46B723F6A8; Tue, 22 Jul 2025 08:06:16 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com Cc: ziy@nvidia.com, baolin.wang@linux.alibaba.com, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, baohua@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [PATCH v3 1/3] mm: add get_and_clear_ptes() and clear_ptes() Date: Tue, 22 Jul 2025 20:35:57 +0530 Message-Id: <20250722150559.96465-2-dev.jain@arm.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250722150559.96465-1-dev.jain@arm.com> References: <20250722150559.96465-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: David Hildenbrand From: David Hildenbrand Let's add variants to be used where "full" does not apply -- which will be the majority of cases in the future. "full" really only applies if we are about to tear down a full MM. Use get_and_clear_ptes() in existing code, clear_ptes() users will be added next. Should we make these inline functions instead and add separate docs? Probably not worth it for now. Signed-off-by: David Hildenbrand Signed-off-by: Dev Jain Reviewed-by: Baolin Wang --- arch/arm64/mm/mmu.c | 2 +- include/linux/pgtable.h | 6 ++++++ mm/mremap.c | 2 +- mm/rmap.c | 2 +- 4 files changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index abd9725796e9..20a89ab97dc5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1528,7 +1528,7 @@ early_initcall(prevent_bootmem_remove_init); pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long add= r, pte_t *ptep, unsigned int nr) { - pte_t pte =3D get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, /* full= =3D */ 0); + pte_t pte =3D get_and_clear_ptes(vma->vm_mm, addr, ptep, nr); =20 if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) { /* diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e3b99920be05..e45986b54277 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -736,6 +736,9 @@ static inline pte_t get_and_clear_full_ptes(struct mm_s= truct *mm, } #endif =20 +#define get_and_clear_ptes(_mm, _addr, _ptep, _nr) \ + get_and_clear_full_ptes(_mm, _addr, _ptep, _nr, 0) + #ifndef clear_full_ptes /** * clear_full_ptes - Clear present PTEs that map consecutive pages of the = same @@ -768,6 +771,9 @@ static inline void clear_full_ptes(struct mm_struct *mm= , unsigned long addr, } #endif =20 +#define clear_ptes(_mm, _addr, _ptep, _nr) \ + clear_full_ptes(_mm, _addr, _ptep, _nr, 0) + /* * If two threads concurrently fault at the same page, the thread that * won the race updates the PTE and its local TLB/Cache. The other thread diff --git a/mm/mremap.c b/mm/mremap.c index ac39845e9718..677a4d744df9 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -280,7 +280,7 @@ static int move_ptes(struct pagetable_move_control *pmc, old_pte, max_nr_ptes); force_flush =3D true; } - pte =3D get_and_clear_full_ptes(mm, old_addr, old_ptep, nr_ptes, 0); + pte =3D get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes); pte =3D move_pte(pte, old_addr, new_addr); pte =3D move_soft_dirty_pte(pte); =20 diff --git a/mm/rmap.c b/mm/rmap.c index f93ce27132ab..568198e9efc2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2036,7 +2036,7 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, flush_cache_range(vma, address, end_addr); =20 /* Nuke the page table entry. */ - pteval =3D get_and_clear_full_ptes(mm, address, pvmw.pte, nr_pages, 0); + pteval =3D get_and_clear_ptes(mm, address, pvmw.pte, nr_pages); /* * We clear the PTE but do not flush so potentially * a remote CPU could still be writing to the folio. --=20 2.30.2 From nobody Mon Oct 6 10:14:39 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AE7FA1E8320 for ; Tue, 22 Jul 2025 15:06:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753196787; cv=none; b=Jg5bws1Z34KN37Pv3xTuYPz/H+Chbg11msuHaRUgfGqIa1YVUEF0tC01rgfQwlUptCO72F+qMehkdrjGjPbnCBVOIrCY4g21TBRJwTsm1wz9Fg5hw39gEExYMlqVj7iPdbP7vdjCDxP85AhZE7Vu/gFzjCzDvazAf1V3uB3xaFA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753196787; c=relaxed/simple; bh=t9/tPmYresC045E3z38JghIzYZjCr3lnQYnP61HpYWU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QTupzy2j/Nc7MQRcIYP1vn9vW1822qTJuo3eL47WS/eWOvIgGs/Fv6fPMpcYIfABTD3acVpRya2d5IDOy5BWRrrvItzb1+6NIm8E9KnwcDHJIX8ufAB7y7u2ePtkc2Ab7ZH9j/bkrQJNtvDhTqhi/SS6N/fYOsWDm/hkgYZdxWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 73E59152B; Tue, 22 Jul 2025 08:06:19 -0700 (PDT) Received: from localhost.localdomain (unknown [10.163.92.223]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EFB1F3F6A8; Tue, 22 Jul 2025 08:06:20 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com Cc: ziy@nvidia.com, baolin.wang@linux.alibaba.com, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, baohua@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [PATCH v3 2/3] khugepaged: Optimize __collapse_huge_page_copy_succeeded() by PTE batching Date: Tue, 22 Jul 2025 20:35:58 +0530 Message-Id: <20250722150559.96465-3-dev.jain@arm.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250722150559.96465-1-dev.jain@arm.com> References: <20250722150559.96465-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use PTE batching to optimize __collapse_huge_page_copy_succeeded(). On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at. For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls. Signed-off-by: Dev Jain Acked-by: David Hildenbrand Reviewed-by: Baolin Wang --- mm/khugepaged.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a55fb1dcd224..63517ef7eafb 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -700,12 +700,15 @@ static void __collapse_huge_page_copy_succeeded(pte_t= *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + unsigned long end =3D address + HPAGE_PMD_SIZE; struct folio *src, *tmp; - pte_t *_pte; pte_t pteval; + pte_t *_pte; + int nr_ptes; =20 - for (_pte =3D pte; _pte < pte + HPAGE_PMD_NR; - _pte++, address +=3D PAGE_SIZE) { + for (_pte =3D pte; _pte < pte + HPAGE_PMD_NR; _pte +=3D nr_ptes, + address +=3D nr_ptes * PAGE_SIZE) { + nr_ptes =3D 1; pteval =3D ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1); @@ -722,18 +725,26 @@ static void __collapse_huge_page_copy_succeeded(pte_t= *pte, struct page *src_page =3D pte_page(pteval); =20 src =3D page_folio(src_page); - if (!folio_test_large(src)) + + if (folio_test_large(src)) { + int max_nr_ptes =3D (end - address) >> PAGE_SHIFT; + + nr_ptes =3D folio_pte_batch(src, _pte, pteval, max_nr_ptes); + } else { release_pte_folio(src); + } + /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats * inside folio_remove_rmap_pte(). */ spin_lock(ptl); - ptep_clear(vma->vm_mm, address, _pte); - folio_remove_rmap_pte(src, src_page, vma); + clear_ptes(vma->vm_mm, address, _pte, nr_ptes); + folio_remove_rmap_ptes(src, src_page, nr_ptes, vma); spin_unlock(ptl); - free_folio_and_swap_cache(src); + free_swap_cache(src); + folio_put_refs(src, nr_ptes); } } =20 --=20 2.30.2 From nobody Mon Oct 6 10:14:39 2025 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4C21A28CF47 for ; Tue, 22 Jul 2025 15:06:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753196791; cv=none; b=kJPzQglWKT4xHFj1yy4GhH2ohzhUnu/5NaaXP0q8Ib22HNNSBHAvrsli8CEtXx1m9HNFdslW/fIeuR45wrj912QQEpAfslMHGIMVEJAN8LnJ2isxJM9JaYMGXIYhXqrilcivnEK+9AF6MeSaln54uKaneHVWiBbS6sg+FozQlcs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753196791; c=relaxed/simple; bh=DDdUn0036EOEcs0aETvQn2pUt2pDL5Df0uvPUlfO6tQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GId0lg2KelQVDSUeQBa5TQjRyEP9fhMqdsErv2GofuquDCOkePTzAcn5lbvMJMLJuimAqnIr+4bo3hBvcozez7PfPUfGTGdWcLX0PrSvbmw/SlxhmAWupDWAyK8qcj9fYFbThMNGx6q+Bt7+pROdYENUCMlSVwHv10lVBzaHMcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 14530152B; Tue, 22 Jul 2025 08:06:24 -0700 (PDT) Received: from localhost.localdomain (unknown [10.163.92.223]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AFC053F6A8; Tue, 22 Jul 2025 08:06:25 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com Cc: ziy@nvidia.com, baolin.wang@linux.alibaba.com, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, baohua@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [PATCH v3 3/3] khugepaged: Optimize collapse_pte_mapped_thp() by PTE batching Date: Tue, 22 Jul 2025 20:35:59 +0530 Message-Id: <20250722150559.96465-4-dev.jain@arm.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250722150559.96465-1-dev.jain@arm.com> References: <20250722150559.96465-1-dev.jain@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use PTE batching to optimize collapse_pte_mapped_thp(). On arm64, suppose khugepaged is scanning a pte-mapped 2MB THP for collapse. Then, calling ptep_clear() for every pte will cause a TLB flush for every contpte block. Instead, clear_ptes() does a contpte_try_unfold_partial() which will flush the TLB only for the (if any) starting and ending contpte block, if they partially overlap with the range khugepaged is looking at. For all arches, there should be a benefit due to batching atomic operations on mapcounts due to folio_remove_rmap_ptes() and saving some calls. Note that we do not need to make a change to the check "if (folio_page(folio, i) !=3D page)"; if i'th page of the folio is equal to the first page of our batch, then i + 1, .... i + nr_batch_ptes - 1 pages of the folio will be equal to the corresponding pages of our batch mapping consecutive pages. Signed-off-by: Dev Jain Acked-by: David Hildenbrand Reviewed-by: Baolin Wang --- mm/khugepaged.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 63517ef7eafb..1ff0c7dd2be4 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1503,15 +1503,16 @@ static int set_huge_pmd(struct vm_area_struct *vma,= unsigned long addr, int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool install_pmd) { + int nr_mapped_ptes =3D 0, nr_batch_ptes, result =3D SCAN_FAIL; struct mmu_notifier_range range; bool notified =3D false; unsigned long haddr =3D addr & HPAGE_PMD_MASK; + unsigned long end =3D haddr + HPAGE_PMD_SIZE; struct vm_area_struct *vma =3D vma_lookup(mm, haddr); struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml =3D NULL, *ptl; - int nr_ptes =3D 0, result =3D SCAN_FAIL; int i; =20 mmap_assert_locked(mm); @@ -1625,11 +1626,15 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, goto abort; =20 /* step 2: clear page table and adjust rmap */ - for (i =3D 0, addr =3D haddr, pte =3D start_pte; - i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE, pte++) { + for (i =3D 0, addr =3D haddr, pte =3D start_pte; i < HPAGE_PMD_NR; + i +=3D nr_batch_ptes, addr +=3D nr_batch_ptes * PAGE_SIZE, + pte +=3D nr_batch_ptes) { + int max_nr_batch_ptes =3D (end - addr) >> PAGE_SHIFT; struct page *page; pte_t ptent =3D ptep_get(pte); =20 + nr_batch_ptes =3D 1; + if (pte_none(ptent)) continue; /* @@ -1643,26 +1648,29 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, goto abort; } page =3D vm_normal_page(vma, addr, ptent); + if (folio_page(folio, i) !=3D page) goto abort; =20 + nr_batch_ptes =3D folio_pte_batch(folio, pte, ptent, max_nr_batch_ptes); + /* * Must clear entry, or a racing truncate may re-remove it. * TLB flush can be left until pmdp_collapse_flush() does it. * PTE dirty? Shmem page is already dirty; file is read-only. */ - ptep_clear(mm, addr, pte); - folio_remove_rmap_pte(folio, page, vma); - nr_ptes++; + clear_ptes(mm, addr, pte, nr_batch_ptes); + folio_remove_rmap_ptes(folio, page, nr_batch_ptes, vma); + nr_mapped_ptes +=3D nr_batch_ptes; } =20 if (!pml) spin_unlock(ptl); =20 /* step 3: set proper refcount and mm_counters. */ - if (nr_ptes) { - folio_ref_sub(folio, nr_ptes); - add_mm_counter(mm, mm_counter_file(folio), -nr_ptes); + if (nr_mapped_ptes) { + folio_ref_sub(folio, nr_mapped_ptes); + add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes); } =20 /* step 4: remove empty page table */ @@ -1695,10 +1703,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, : SCAN_SUCCEED; goto drop_folio; abort: - if (nr_ptes) { + if (nr_mapped_ptes) { flush_tlb_mm(mm); - folio_ref_sub(folio, nr_ptes); - add_mm_counter(mm, mm_counter_file(folio), -nr_ptes); + folio_ref_sub(folio, nr_mapped_ptes); + add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes); } unlock: if (start_pte) --=20 2.30.2