From nobody Thu Oct 2 16:33:53 2025 Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E5E01527B4 for ; Mon, 15 Sep 2025 03:30:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.132 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757907027; cv=none; b=cYDzz8TBl44wThjVMhuI0m51QOJCQ2xfMzJtYCQUaBAz+UBFJsHFynf0yMDPjS9flrJv8QWt8ZOgpqgy4rjGrfnlkFE7L6IE3FF8aNVka6AZMiRiydH/hVllVVA2zuGTE97hsyACe3kc031M6G7ICtIih6IEftXjz5QmtDsVn60= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757907027; c=relaxed/simple; bh=Z/9i1GMtrLinZ0JgIPhedH1rmENVy8HgASCv5V/T8Cs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gpBT2fYwVmSj6DG4W26ipFy3wmYJDtaOfrUTQC5qrrVBSqCq2Nm4WQYmNqiBjx8qYyjq/P0ANbYqVrNFiQU9JbABROBsGA/PmUlq/oiChRqLLuAF898GcFl5LETXG/sGGMejJDQFqMT7ok0Q+TOPHgomCsJuHZ8zKGJK2uVUlKY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=xTUUm3bk; arc=none smtp.client-ip=115.124.30.132 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="xTUUm3bk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1757907016; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=EltDD6XaVHb4ywefi6tfTUXJyF6xjUHuaP+jJuKR6OU=; b=xTUUm3bk6sKa6+X66NCIcRlgDaFGek2cxSVqDDYjyZMNL3cfFALNeIy2jwro5OlYHf9DfaLVTcSaZ/4l1ZJ5NksDKwewgS1sDq/rErbDoGrM+ehmlV225BSfg8WInFI9RU8SMEkiDlAXaUMGAiKp0ec2sEuP4UOZd0l6rEbC9Jk= Received: from localhost.localdomain(mailfrom:ying.huang@linux.alibaba.com fp:SMTPD_---0Wnx2CMR_1757907013 cluster:ay36) by smtp.aliyun-inc.com; Mon, 15 Sep 2025 11:30:14 +0800 From: Huang Ying To: Catalin Marinas , Will Deacon , Andrew Morton , David Hildenbrand Cc: Huang Ying , Lorenzo Stoakes , Vlastimil Babka , Zi Yan , Baolin Wang , Ryan Roberts , Yang Shi , "Christoph Lameter (Ampere)" , Dev Jain , Barry Song , Anshuman Khandual , Yicong Yang , Kefeng Wang , Kevin Brodsky , Yin Fengwei , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 1/2] mm: add spurious fault fixing support for huge pmd Date: Mon, 15 Sep 2025 11:29:45 +0800 Message-Id: <20250915032946.33203-2-ying.huang@linux.alibaba.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250915032946.33203-1-ying.huang@linux.alibaba.com> References: <20250915032946.33203-1-ying.huang@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the current kernel, there is spurious fault fixing support for pte, but not for huge pmd because no architectures need it. But in the next patch in the series, we will change the write protection fault handling logic on arm64, so that some stale huge pmd entries may remain in the TLB. These entries need to be flushed via the huge pmd spurious fault fixing mechanism. Signed-off-by: Huang Ying Cc: Catalin Marinas Cc: Will Deacon Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Vlastimil Babka Cc: Zi Yan Cc: Baolin Wang Cc: Ryan Roberts Cc: Yang Shi Cc: "Christoph Lameter (Ampere)" Cc: Dev Jain Cc: Barry Song Cc: Anshuman Khandual Cc: Yicong Yang Cc: Kefeng Wang Cc: Kevin Brodsky Cc: Yin Fengwei Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/pgtable.h | 4 ++++ mm/huge_memory.c | 25 +++++++++++++++++++++---- mm/internal.h | 4 ++-- 3 files changed, 27 insertions(+), 6 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 2b80fd456c8b..d7604ad34d36 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1232,6 +1232,10 @@ static inline void arch_swap_restore(swp_entry_t ent= ry, struct folio *folio) #define flush_tlb_fix_spurious_fault(vma, address, ptep) flush_tlb_page(vm= a, address) #endif =20 +#ifndef flush_tlb_fix_spurious_fault_pmd +#define flush_tlb_fix_spurious_fault_pmd(vma, address, ptep) do { } while = (0) +#endif + /* * When walking page tables, get the address of the next boundary, * or the end address of the range if that comes earlier. Although no diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9c38a95e9f09..0e2ef6b007c2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1650,8 +1650,8 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf,= struct folio *folio, EXPORT_SYMBOL_GPL(vmf_insert_folio_pud); #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ =20 -void touch_pmd(struct vm_area_struct *vma, unsigned long addr, - pmd_t *pmd, bool write) +int touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write) { pmd_t _pmd; =20 @@ -1659,8 +1659,12 @@ void touch_pmd(struct vm_area_struct *vma, unsigned = long addr, if (write) _pmd =3D pmd_mkdirty(_pmd); if (pmdp_set_access_flags(vma, addr & HPAGE_PMD_MASK, - pmd, _pmd, write)) + pmd, _pmd, write)) { update_mmu_cache_pmd(vma, addr, pmd); + return 1; + } + + return 0; } =20 int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, @@ -1857,7 +1861,20 @@ void huge_pmd_set_accessed(struct vm_fault *vmf) if (unlikely(!pmd_same(*vmf->pmd, vmf->orig_pmd))) goto unlock; =20 - touch_pmd(vmf->vma, vmf->address, vmf->pmd, write); + if (!touch_pmd(vmf->vma, vmf->address, vmf->pmd, write)) { + /* Skip spurious TLB flush for retried page fault */ + if (vmf->flags & FAULT_FLAG_TRIED) + goto unlock; + /* + * This is needed only for protection faults but the arch code + * is not yet telling us if this is a protection fault or not. + * This still avoids useless tlb flushes for .text page faults + * with threads. + */ + if (vmf->flags & FAULT_FLAG_WRITE) + flush_tlb_fix_spurious_fault_pmd(vmf->vma, vmf->address, + vmf->pmd); + } =20 unlock: spin_unlock(vmf->ptl); diff --git a/mm/internal.h b/mm/internal.h index 45b725c3dc03..743ce97c7248 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1406,8 +1406,8 @@ int __must_check try_grab_folio(struct folio *folio, = int refs, */ void touch_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pud, bool write); -void touch_pmd(struct vm_area_struct *vma, unsigned long addr, - pmd_t *pmd, bool write); +int touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write); =20 /* * Parses a string with mem suffixes into its order. Useful to parse kernel --=20 2.39.5 From nobody Thu Oct 2 16:33:53 2025 Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 168E4CA6F for ; Mon, 15 Sep 2025 03:30:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.99 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757907027; cv=none; b=NAVaqM/BB49UTor/ER1QLSKMAemHJGzeZDDh95ZhhXcfCeZ/rjuSQ5/iNSyiUExAXD29qAmbT0Krxwbnenhh31dH45j6eHSWt+t2+8kAGh4IRgYb3mFu+avvnhaff5Q3b1CYmCiNZc7phQZciKDhbTowDed+4snfjSsunYH2M1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757907027; c=relaxed/simple; bh=wZVi2wabEOpgtUDHV27rglg9c6vnB19hOWyKfgXbmO4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iw+k/XK9qu4fSC5emdgdKsAFh1xbE4cYwWNu2g5MsSnv/lZNM4VZxGk7IcQ29u+WvGQv0tuIcDGgOAYCV3zEIkptStvIfEzj4nCepweTjDy3A6fUtRPH19uavXIjjCOxFt4zqv+Laug5muh5hXFtgyqUYY+Mww1yj3AhQxhk76E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=oE/iHbvJ; arc=none smtp.client-ip=115.124.30.99 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="oE/iHbvJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1757907017; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=aV55o1PrOMi1ewOa19IO6mFfTMCoyqv5bAgN6SrTF+U=; b=oE/iHbvJwAxW5XU8/yo2Zk/V3WnzlTNd2esVdRAmVwcIjhNvVtPByoTCrm1eFz0GHVH6AMuXicZ5j8dnU2V9sPeOFf3CI1RopSLuVxaETQAuWZG+wFtydFWOvJ/Y91BsGRCJSYzGo/HJfMv3NdJmH1rQx7NfsD/weLVYsKhAQnQ= Received: from localhost.localdomain(mailfrom:ying.huang@linux.alibaba.com fp:SMTPD_---0Wnx2CMj_1757907014 cluster:ay36) by smtp.aliyun-inc.com; Mon, 15 Sep 2025 11:30:15 +0800 From: Huang Ying To: Catalin Marinas , Will Deacon , Andrew Morton , David Hildenbrand Cc: Huang Ying , Lorenzo Stoakes , Vlastimil Babka , Zi Yan , Baolin Wang , Ryan Roberts , Yang Shi , "Christoph Lameter (Ampere)" , Dev Jain , Barry Song , Anshuman Khandual , Yicong Yang , Kefeng Wang , Kevin Brodsky , Yin Fengwei , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 2/2] arm64, tlbflush: don't TLBI broadcast if page reused in write fault Date: Mon, 15 Sep 2025 11:29:46 +0800 Message-Id: <20250915032946.33203-3-ying.huang@linux.alibaba.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250915032946.33203-1-ying.huang@linux.alibaba.com> References: <20250915032946.33203-1-ying.huang@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A multi-thread customer workload with large memory footprint uses fork()/exec() to run some external programs every tens seconds. When running the workload on an arm64 server machine, it's observed that quite some CPU cycles are spent in the TLB flushing functions. While running the workload on the x86_64 server machine, it's not. This causes the performance on arm64 to be much worse than that on x86_64. During the workload running, after fork()/exec() write-protects all pages in the parent process, memory writing in the parent process will cause a write protection fault. Then the page fault handler will make the PTE/PDE writable if the page can be reused, which is almost always true in the workload. On arm64, to avoid the write protection fault on other CPUs, the page fault handler flushes the TLB globally with TLBI broadcast after changing the PTE/PDE. However, this isn't always necessary. Firstly, it's safe to leave some stall read-only TLB entries as long as they will be flushed finally. Secondly, it's quite possible that the original read-only PTE/PDEs aren't cached in remote TLB at all if the memory footprint is large. In fact, on x86_64, the page fault handler doesn't flush the remote TLB in this situation, which benefits the performance a lot. To improve the performance on arm64, make the write protection fault handler flush the TLB locally instead of globally via TLBI broadcast after making the PTE/PDE writable. If there are stall read-only TLB entries in the remote CPUs, the page fault handler on these CPUs will regard the page fault as spurious and flush the stall TLB entries. To test the patchset, make the usemem.c from vm-scalability (https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scal= ability.git). support calling fork()/exec() periodically. To mimic the behavior of the customer workload, run usemem with 4 threads, access 100GB memory, and call fork()/exec() every 40 seconds. Test results show that with the patchset the score of usemem improves ~40.6%. The cycles% of TLB flush functions reduces from ~50.5% to ~0.3% in perf profile. Signed-off-by: Huang Ying Cc: Catalin Marinas Cc: Will Deacon Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Vlastimil Babka Cc: Zi Yan Cc: Baolin Wang Cc: Ryan Roberts Cc: Yang Shi Cc: "Christoph Lameter (Ampere)" Cc: Dev Jain Cc: Barry Song Cc: Anshuman Khandual Cc: Yicong Yang Cc: Kefeng Wang Cc: Kevin Brodsky Cc: Yin Fengwei Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- arch/arm64/include/asm/pgtable.h | 14 ++++++++----- arch/arm64/include/asm/tlbflush.h | 33 +++++++++++++++++++++++++++++++ arch/arm64/mm/fault.c | 2 +- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index abd2dee416b3..a9ed8c9d2c33 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -130,12 +130,16 @@ static inline void arch_leave_lazy_mmu_mode(void) #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ =20 /* - * Outside of a few very special situations (e.g. hibernation), we always - * use broadcast TLB invalidation instructions, therefore a spurious page - * fault on one CPU which has been handled concurrently by another CPU - * does not need to perform additional invalidation. + * We use local TLB invalidation instruction when reusing page in + * write protection fault handler to avoid TLBI broadcast in the hot + * path. This will cause spurious page faults if stall read-only TLB + * entries exist. */ -#define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0) +#define flush_tlb_fix_spurious_fault(vma, address, ptep) \ + local_flush_tlb_page_nonotify(vma, address) + +#define flush_tlb_fix_spurious_fault_pmd(vma, address, pmdp) \ + local_flush_tlb_page_nonotify(vma, address) =20 /* * ZERO_PAGE is a global shared page that is always zero: used diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlb= flush.h index 18a5dc0c9a54..607b67d8f61b 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -282,6 +282,39 @@ static inline void flush_tlb_mm(struct mm_struct *mm) mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); } =20 +static inline void __local_flush_tlb_page_nonotify_nosync( + struct mm_struct *mm, unsigned long uaddr) +{ + unsigned long addr; + + dsb(nshst); + addr =3D __TLBI_VADDR(uaddr, ASID(mm)); + __tlbi(vale1, addr); + __tlbi_user(vale1, addr); +} + +static inline void local_flush_tlb_page_nonotify( + struct vm_area_struct *vma, unsigned long uaddr) +{ + __local_flush_tlb_page_nonotify_nosync(vma->vm_mm, uaddr); + dsb(nsh); +} + +static inline void __local_flush_tlb_page_nosync( + struct mm_struct *mm, unsigned long uaddr) +{ + __local_flush_tlb_page_nonotify_nosync(mm, uaddr); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, uaddr & PAGE_MASK, + (uaddr & PAGE_MASK) + PAGE_SIZE); +} + +static inline void local_flush_tlb_page(struct vm_area_struct *vma, + unsigned long uaddr) +{ + __local_flush_tlb_page_nosync(vma->vm_mm, uaddr); + dsb(nsh); +} + static inline void __flush_tlb_page_nosync(struct mm_struct *mm, unsigned long uaddr) { diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index d816ff44faff..22f54f5afe3f 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -235,7 +235,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, =20 /* Invalidate a stale read-only entry */ if (dirty) - flush_tlb_page(vma, address); + local_flush_tlb_page(vma, address); return 1; } =20 --=20 2.39.5