From nobody Sun Feb 8 10:50:04 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27227C7EE24 for ; Fri, 2 Jun 2023 09:31:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234772AbjFBJbP (ORCPT ); Fri, 2 Jun 2023 05:31:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234686AbjFBJah (ORCPT ); Fri, 2 Jun 2023 05:30:37 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E899BE5A for ; Fri, 2 Jun 2023 02:30:05 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 19B1E12FC; Fri, 2 Jun 2023 02:30:51 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C874E3F7BD; Fri, 2 Jun 2023 02:30:03 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , SeongJae Park , Christoph Hellwig Cc: Ryan Roberts , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Lorenzo Stoakes , Uladzislau Rezki , Zi Yan , Yu Zhao , Mike Rapoport , linux-kernel@vger.kernel.org, linux-mm@kvack.org, damon@lists.linux.dev, Christoph Hellwig Subject: [PATCH v3 1/4] mm: vmalloc must set pte via arch code Date: Fri, 2 Jun 2023 10:29:46 +0100 Message-Id: <20230602092949.545577-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230602092949.545577-1-ryan.roberts@arm.com> References: <20230602092949.545577-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" It is bad practice to directly set pte entries within a pte table. Instead all modifications must go through arch-provided helpers such as set_pte_at() to give the arch code visibility and allow it to check (and potentially modify) the operation. Fixes: 3e9a9e256b1e ("mm: add a vmap_pfn function") Signed-off-by: Ryan Roberts Reviewed-by: Zi Yan Acked-by: Lorenzo Stoakes Reviewed-by: Christoph Hellwig Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Mike Rapoport (IBM) --- mm/vmalloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9683573f1225..48202ec5f79a 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2899,10 +2899,16 @@ struct vmap_pfn_data { static int vmap_pfn_apply(pte_t *pte, unsigned long addr, void *private) { struct vmap_pfn_data *data =3D private; + unsigned long pfn =3D data->pfns[data->idx]; + pte_t ptent; =20 - if (WARN_ON_ONCE(pfn_valid(data->pfns[data->idx]))) + if (WARN_ON_ONCE(pfn_valid(pfn))) return -EINVAL; - *pte =3D pte_mkspecial(pfn_pte(data->pfns[data->idx++], data->prot)); + + ptent =3D pte_mkspecial(pfn_pte(pfn, data->prot)); + set_pte_at(&init_mm, addr, pte, ptent); + + data->idx++; return 0; } =20 --=20 2.25.1 From nobody Sun Feb 8 10:50:04 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F4DDC7EE24 for ; Fri, 2 Jun 2023 09:31:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235056AbjFBJbU (ORCPT ); Fri, 2 Jun 2023 05:31:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234751AbjFBJak (ORCPT ); Fri, 2 Jun 2023 05:30:40 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id D6334E68 for ; Fri, 2 Jun 2023 02:30:07 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 13E121764; Fri, 2 Jun 2023 02:30:53 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DD7893F7BD; Fri, 2 Jun 2023 02:30:05 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , SeongJae Park , Christoph Hellwig Cc: Ryan Roberts , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Lorenzo Stoakes , Uladzislau Rezki , Zi Yan , Yu Zhao , Mike Rapoport , linux-kernel@vger.kernel.org, linux-mm@kvack.org, damon@lists.linux.dev Subject: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds Date: Fri, 2 Jun 2023 10:29:47 +0100 Message-Id: <20230602092949.545577-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230602092949.545577-1-ryan.roberts@arm.com> References: <20230602092949.545577-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation. Fixes: 3f49584b262c ("mm/damon: implement primitives for the virtual memory= address spaces"). Signed-off-by: Ryan Roberts Reviewed-by: Zi Yan Reviewed-by: SeongJae Park Reviewed-by: Mike Rapoport (IBM) --- mm/damon/ops-common.c | 16 ++++++---------- mm/damon/ops-common.h | 4 ++-- mm/damon/paddr.c | 4 ++-- mm/damon/vaddr.c | 4 ++-- 4 files changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index cc63cf953636..acc264b97903 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -37,7 +37,7 @@ struct folio *damon_get_folio(unsigned long pfn) return folio; } =20 -void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr) +void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr) { bool referenced =3D false; struct folio *folio =3D damon_get_folio(pte_pfn(*pte)); @@ -45,13 +45,11 @@ void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm,= unsigned long addr) if (!folio) return; =20 - if (pte_young(*pte)) { + if (ptep_test_and_clear_young(vma, addr, pte)) referenced =3D true; - *pte =3D pte_mkold(*pte); - } =20 #ifdef CONFIG_MMU_NOTIFIER - if (mmu_notifier_clear_young(mm, addr, addr + PAGE_SIZE)) + if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE)) referenced =3D true; #endif /* CONFIG_MMU_NOTIFIER */ =20 @@ -62,7 +60,7 @@ void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, u= nsigned long addr) folio_put(folio); } =20 -void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr) +void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE bool referenced =3D false; @@ -71,13 +69,11 @@ void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm,= unsigned long addr) if (!folio) return; =20 - if (pmd_young(*pmd)) { + if (pmdp_test_and_clear_young(vma, addr, pmd)) referenced =3D true; - *pmd =3D pmd_mkold(*pmd); - } =20 #ifdef CONFIG_MMU_NOTIFIER - if (mmu_notifier_clear_young(mm, addr, addr + HPAGE_PMD_SIZE)) + if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + HPAGE_PMD_SIZE)) referenced =3D true; #endif /* CONFIG_MMU_NOTIFIER */ =20 diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index 14f4bc69f29b..18d837d11bce 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -9,8 +9,8 @@ =20 struct folio *damon_get_folio(unsigned long pfn); =20 -void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr= ); -void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr= ); +void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr); +void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr); =20 int damon_cold_score(struct damon_ctx *c, struct damon_region *r, struct damos *s); diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 467b99166b43..5b3a3463d078 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -24,9 +24,9 @@ static bool __damon_pa_mkold(struct folio *folio, struct = vm_area_struct *vma, while (page_vma_mapped_walk(&pvmw)) { addr =3D pvmw.address; if (pvmw.pte) - damon_ptep_mkold(pvmw.pte, vma->vm_mm, addr); + damon_ptep_mkold(pvmw.pte, vma, addr); else - damon_pmdp_mkold(pvmw.pmd, vma->vm_mm, addr); + damon_pmdp_mkold(pvmw.pmd, vma, addr); } return true; } diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 1fec16d7263e..37994fb6120c 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -311,7 +311,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned l= ong addr, } =20 if (pmd_trans_huge(*pmd)) { - damon_pmdp_mkold(pmd, walk->mm, addr); + damon_pmdp_mkold(pmd, walk->vma, addr); spin_unlock(ptl); return 0; } @@ -323,7 +323,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned l= ong addr, pte =3D pte_offset_map_lock(walk->mm, pmd, addr, &ptl); if (!pte_present(*pte)) goto out; - damon_ptep_mkold(pte, walk->mm, addr); + damon_ptep_mkold(pte, walk->vma, addr); out: pte_unmap_unlock(pte, ptl); return 0; --=20 2.25.1 From nobody Sun Feb 8 10:50:04 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2205C7EE29 for ; Fri, 2 Jun 2023 09:31:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235147AbjFBJbY (ORCPT ); Fri, 2 Jun 2023 05:31:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234695AbjFBJal (ORCPT ); Fri, 2 Jun 2023 05:30:41 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id CD8B910F3 for ; Fri, 2 Jun 2023 02:30:09 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0D42D176A; Fri, 2 Jun 2023 02:30:55 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D73083F7BD; Fri, 2 Jun 2023 02:30:07 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , SeongJae Park , Christoph Hellwig Cc: Ryan Roberts , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Lorenzo Stoakes , Uladzislau Rezki , Zi Yan , Yu Zhao , Mike Rapoport , linux-kernel@vger.kernel.org, linux-mm@kvack.org, damon@lists.linux.dev Subject: [PATCH v3 3/4] mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify() Date: Fri, 2 Jun 2023 10:29:48 +0100 Message-Id: <20230602092949.545577-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230602092949.545577-1-ryan.roberts@arm.com> References: <20230602092949.545577-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" With the fix in place to atomically test and clear young on ptes and pmds, simplify the code to handle the clearing for both the primary mmu and the mmu notifier with a single API call. Signed-off-by: Ryan Roberts Acked-by: Yu Zhao Reviewed-by: SeongJae Park --- mm/damon/ops-common.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index acc264b97903..d4ab81229136 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -39,21 +39,12 @@ struct folio *damon_get_folio(unsigned long pfn) =20 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned lon= g addr) { - bool referenced =3D false; struct folio *folio =3D damon_get_folio(pte_pfn(*pte)); =20 if (!folio) return; =20 - if (ptep_test_and_clear_young(vma, addr, pte)) - referenced =3D true; - -#ifdef CONFIG_MMU_NOTIFIER - if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE)) - referenced =3D true; -#endif /* CONFIG_MMU_NOTIFIER */ - - if (referenced) + if (ptep_clear_young_notify(vma, addr, pte)) folio_set_young(folio); =20 folio_set_idle(folio); @@ -63,21 +54,12 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct= *vma, unsigned long addr void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned lon= g addr) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - bool referenced =3D false; struct folio *folio =3D damon_get_folio(pmd_pfn(*pmd)); =20 if (!folio) return; =20 - if (pmdp_test_and_clear_young(vma, addr, pmd)) - referenced =3D true; - -#ifdef CONFIG_MMU_NOTIFIER - if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + HPAGE_PMD_SIZE)) - referenced =3D true; -#endif /* CONFIG_MMU_NOTIFIER */ - - if (referenced) + if (pmdp_clear_young_notify(vma, addr, pmd)) folio_set_young(folio); =20 folio_set_idle(folio); --=20 2.25.1 From nobody Sun Feb 8 10:50:04 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 348A7C7EE29 for ; Fri, 2 Jun 2023 09:31:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235219AbjFBJb2 (ORCPT ); Fri, 2 Jun 2023 05:31:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235012AbjFBJam (ORCPT ); Fri, 2 Jun 2023 05:30:42 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 149F910F7 for ; Fri, 2 Jun 2023 02:30:12 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0728A176B; Fri, 2 Jun 2023 02:30:57 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D0E1D3F7BD; Fri, 2 Jun 2023 02:30:09 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , SeongJae Park , Christoph Hellwig Cc: Ryan Roberts , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Lorenzo Stoakes , Uladzislau Rezki , Zi Yan , Yu Zhao , Mike Rapoport , linux-kernel@vger.kernel.org, linux-mm@kvack.org, damon@lists.linux.dev Subject: [PATCH v3 4/4] mm: Fix failure to unmap pte on highmem systems Date: Fri, 2 Jun 2023 10:29:49 +0100 Message-Id: <20230602092949.545577-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230602092949.545577-1-ryan.roberts@arm.com> References: <20230602092949.545577-1-ryan.roberts@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The loser of a race to service a pte for a device private entry in the swap path previously unlocked the ptl, but failed to unmap the pte. This only affects highmem systems since unmapping a pte is a noop on non-highmem systems. Fixes: 16ce101db85d ("mm/memory.c: fix race when faulting a device private = page") Signed-off-by: Ryan Roberts Reviewed-by: Zi Yan Reviewed-by: Mike Rapoport (IBM) --- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f69fbc251198..ed429e20a1bb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3728,10 +3728,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page =3D pfn_swap_entry_to_page(entry); vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { - spin_unlock(vmf->ptl); - goto out; - } + if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) + goto unlock; =20 /* * Get a page reference while we know the page can't be --=20 2.25.1