From nobody Tue Apr 7 18:51:44 2026 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C42539A803 for ; Fri, 27 Feb 2026 09:45:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185502; cv=none; b=Ur/EfkJUoNPOKJPmb8MIHcM2Wy2XliVnFAofexUHOh/gmgK659DeFqd+x4h/2C/m4qR+So32OqJu99bnXuUzAQoFJ0ussajO3RbYsBbZFm0YuJlHnroMgWuaWc3X6WRZQQooie6OmoqN/FsCLzM+nrdR9kuiRtvqFQnZTKeMP4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185502; c=relaxed/simple; bh=t4x6GDmpzLr4rlZ/nE7xyLxijf20R7tQkFBqzX5v+0o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QGNgjrzz365jT+n3ywUfFByumSIQpl2OiCBFCek0broWoNkcYgd6KTY+2fC/vED2cCb4vHU8cdH+hDFxQQ4YdQ8FKOuODO1/6cUUB+Rq8+2egxXnciSX1NqbK+I0zSO0FSHEq3t2AuL/O+BhTH2oVT+QUmIyq44ZhC8jSw8jw00= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=UrKHb74w; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="UrKHb74w" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1772185493; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=QGeaEqvp4zEvzyk67BVba1jL/iSgAiyyvsTj8YvMr1U=; b=UrKHb74wLbO48li+JV0SeWIFhs4iRNVSvxESvKfGZRRx3VgQOXcrXHyIz9rUGZtvx3XkklHW2/laLQfzcMm4D1H+ubqDCx740ao4xieWdLMNGmguTPJ9Fj1lH1wlZ56Z2FyxiQ9+t2dS7wVQNaXRMT6aMOXxBaMQh7K6vkYhVXk= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Wztvd.X_1772185491 cluster:ay36) by smtp.aliyun-inc.com; Fri, 27 Feb 2026 17:44:51 +0800 From: Baolin Wang To: akpm@linux-foundation.org, david@kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, riel@surriel.com, harry.yoo@oracle.com, jannh@google.com, willy@infradead.org, baohua@kernel.org, dev.jain@arm.com, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, hannes@cmpxchg.org, zhengqi.arch@bytedance.com, shakeel.butt@linux.dev, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/6] mm: use inline helper functions instead of ugly macros Date: Fri, 27 Feb 2026 17:44:35 +0800 Message-ID: <4ffb2946062b9452be53d6b2fa3e359ba6446a93.1772185080.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" People have already complained that these *_clear_young_notify() related macros are very ugly, so let's use inline helpers to make them more readabl= e. In addition, we cannot implement these inline helper functions in the mmu_notifier.h file, because some arch-specific files will include the mmu_notifier.h, which introduces header compilation dependencies and causes build errors (e.g., arch/arm64/include/asm/tlbflush.h). Moreover, since these functions are only used in the mm, implementing these inline helpers in the mm/internal.h header seems reasonable. Reviewed-by: Rik van Riel Reviewed-by: Barry Song Acked-by: David Hildenbrand (Arm) Signed-off-by: Baolin Wang --- include/linux/mmu_notifier.h | 54 ------------------------------------ mm/internal.h | 52 ++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 54 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 07a2bbaf86e9..93894b90c8c1 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -515,55 +515,6 @@ static inline void mmu_notifier_range_init_owner( range->owner =3D owner; } =20 -#define clear_flush_young_ptes_notify(__vma, __address, __ptep, __nr) \ -({ \ - int __young; \ - struct vm_area_struct *___vma =3D __vma; \ - unsigned long ___address =3D __address; \ - unsigned int ___nr =3D __nr; \ - __young =3D clear_flush_young_ptes(___vma, ___address, __ptep, ___nr); \ - __young |=3D mmu_notifier_clear_flush_young(___vma->vm_mm, \ - ___address, \ - ___address + \ - ___nr * PAGE_SIZE); \ - __young; \ -}) - -#define pmdp_clear_flush_young_notify(__vma, __address, __pmdp) \ -({ \ - int __young; \ - struct vm_area_struct *___vma =3D __vma; \ - unsigned long ___address =3D __address; \ - __young =3D pmdp_clear_flush_young(___vma, ___address, __pmdp); \ - __young |=3D mmu_notifier_clear_flush_young(___vma->vm_mm, \ - ___address, \ - ___address + \ - PMD_SIZE); \ - __young; \ -}) - -#define ptep_clear_young_notify(__vma, __address, __ptep) \ -({ \ - int __young; \ - struct vm_area_struct *___vma =3D __vma; \ - unsigned long ___address =3D __address; \ - __young =3D ptep_test_and_clear_young(___vma, ___address, __ptep);\ - __young |=3D mmu_notifier_clear_young(___vma->vm_mm, ___address, \ - ___address + PAGE_SIZE); \ - __young; \ -}) - -#define pmdp_clear_young_notify(__vma, __address, __pmdp) \ -({ \ - int __young; \ - struct vm_area_struct *___vma =3D __vma; \ - unsigned long ___address =3D __address; \ - __young =3D pmdp_test_and_clear_young(___vma, ___address, __pmdp);\ - __young |=3D mmu_notifier_clear_young(___vma->vm_mm, ___address, \ - ___address + PMD_SIZE); \ - __young; \ -}) - #else /* CONFIG_MMU_NOTIFIER */ =20 struct mmu_notifier_range { @@ -651,11 +602,6 @@ static inline void mmu_notifier_subscriptions_destroy(= struct mm_struct *mm) =20 #define mmu_notifier_range_update_to_read_only(r) false =20 -#define clear_flush_young_ptes_notify clear_flush_young_ptes -#define pmdp_clear_flush_young_notify pmdp_clear_flush_young -#define ptep_clear_young_notify ptep_test_and_clear_young -#define pmdp_clear_young_notify pmdp_test_and_clear_young - static inline void mmu_notifier_synchronize(void) { } diff --git a/mm/internal.h b/mm/internal.h index f4d95e2ab799..f27cdad98c1e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -1791,4 +1792,55 @@ static inline int io_remap_pfn_range_complete(struct= vm_area_struct *vma, return remap_pfn_range_complete(vma, addr, pfn, size, prot); } =20 +#ifdef CONFIG_MMU_NOTIFIER +static inline int clear_flush_young_ptes_notify(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, unsigned int nr) +{ + int young; + + young =3D clear_flush_young_ptes(vma, addr, ptep, nr); + young |=3D mmu_notifier_clear_flush_young(vma->vm_mm, addr, + addr + nr * PAGE_SIZE); + return young; +} + +static inline int pmdp_clear_flush_young_notify(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) +{ + int young; + + young =3D pmdp_clear_flush_young(vma, addr, pmdp); + young |=3D mmu_notifier_clear_flush_young(vma->vm_mm, addr, addr + PMD_SI= ZE); + return young; +} + +static inline int ptep_clear_young_notify(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young =3D ptep_test_and_clear_young(vma, addr, ptep); + young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + return young; +} + +static inline int pmdp_clear_young_notify(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) +{ + int young; + + young =3D pmdp_test_and_clear_young(vma, addr, pmdp); + young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, addr + PMD_SIZE); + return young; +} + +#else /* CONFIG_MMU_NOTIFIER */ + +#define clear_flush_young_ptes_notify clear_flush_young_ptes +#define pmdp_clear_flush_young_notify pmdp_clear_flush_young +#define ptep_clear_young_notify ptep_test_and_clear_young +#define pmdp_clear_young_notify pmdp_test_and_clear_young + +#endif /* CONFIG_MMU_NOTIFIER */ + #endif /* __MM_INTERNAL_H */ --=20 2.47.3 From nobody Tue Apr 7 18:51:44 2026 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6A0E3D5237 for ; Fri, 27 Feb 2026 09:45:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.133 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185504; cv=none; b=Ci4ByQSgArnOJ8bid96O/ExH3X9wBCNDsXKeIFjAdeX9TCLtdLxudQAFbzuwLH6M/dFPe1I2vjxaH1Ef9X5rTkPFeaY/I/FrDWK7ie3cdxDLcC6n/stRiwBIxuAGAbIu3oBW8o1SvLR0uzHu95ai8MpUNoGKFy20FC9A7BTY3ro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185504; c=relaxed/simple; bh=TZ54zL07gog2vuFr1fxDYfY20JsPW9QRJd8xaXkHBoY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=duHMa3HwUpz1uhrLSCMzvQUOUJGQXkEMFr3R/6F2leTF8g/dhs64lGPDAogW07mNWCg/ik47TV94x8FtNuR6/rXdp9L+UxR2WjNpDwTFMV0vVGb1zmJ/GmKXw7w8ZbOGB1ErgnOHl9CJVsPAr85qsONUo3BqKIV7yA7qf6l0zbI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=BOByMg3Y; arc=none smtp.client-ip=115.124.30.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="BOByMg3Y" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1772185495; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=Gb3TnxdXPB7NGEqIRzrILLzcA1IoPG9elueOZ+c741Q=; b=BOByMg3YFsMIzm8n3q8nRvCXP0tN5uYgUk/KtNQ0VT60P2e7MuKX/PnRSD8PN+PVm855zohEPSJ/1WGiIiAAry8FCMTOgG0pkrhX2LkZxl64F477/HBHREGOyfDNvnLArgMd5JD8mSTiA13QtzLbpf+/EwN2XORlveMz71HkdqQ= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Wztvd08_1772185492 cluster:ay36) by smtp.aliyun-inc.com; Fri, 27 Feb 2026 17:44:53 +0800 From: Baolin Wang To: akpm@linux-foundation.org, david@kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, riel@surriel.com, harry.yoo@oracle.com, jannh@google.com, willy@infradead.org, baohua@kernel.org, dev.jain@arm.com, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, hannes@cmpxchg.org, zhengqi.arch@bytedance.com, shakeel.butt@linux.dev, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/6] mm: rename ptep/pmdp_clear_young_notify() to ptep/pmdp_test_and_clear_young_notify() Date: Fri, 27 Feb 2026 17:44:36 +0800 Message-ID: <283d103c5131e32879bd847263c93b768df64d15.1772185080.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename ptep/pmdp_clear_young_notify() to ptep/pmdp_test_and_clear_young_not= ify() to make the function names consistent. Suggested-by: David Hildenbrand (Arm) Signed-off-by: Baolin Wang Acked-by: David Hildenbrand (Arm) --- mm/internal.h | 8 ++++---- mm/vmscan.c | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index f27cdad98c1e..af04b177f21f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1814,7 +1814,7 @@ static inline int pmdp_clear_flush_young_notify(struc= t vm_area_struct *vma, return young; } =20 -static inline int ptep_clear_young_notify(struct vm_area_struct *vma, +static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *= vma, unsigned long addr, pte_t *ptep) { int young; @@ -1824,7 +1824,7 @@ static inline int ptep_clear_young_notify(struct vm_a= rea_struct *vma, return young; } =20 -static inline int pmdp_clear_young_notify(struct vm_area_struct *vma, +static inline int pmdp_test_and_clear_young_notify(struct vm_area_struct *= vma, unsigned long addr, pmd_t *pmdp) { int young; @@ -1838,8 +1838,8 @@ static inline int pmdp_clear_young_notify(struct vm_a= rea_struct *vma, =20 #define clear_flush_young_ptes_notify clear_flush_young_ptes #define pmdp_clear_flush_young_notify pmdp_clear_flush_young -#define ptep_clear_young_notify ptep_test_and_clear_young -#define pmdp_clear_young_notify pmdp_test_and_clear_young +#define ptep_test_and_clear_young_notify ptep_test_and_clear_young +#define pmdp_test_and_clear_young_notify pmdp_test_and_clear_young =20 #endif /* CONFIG_MMU_NOTIFIER */ =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index 6c94e4ecebb2..0a5622420987 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3508,7 +3508,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, if (!folio) continue; =20 - if (!ptep_clear_young_notify(args->vma, addr, pte + i)) + if (!ptep_test_and_clear_young_notify(args->vma, addr, pte + i)) continue; =20 if (last !=3D folio) { @@ -3599,7 +3599,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigne= d long addr, struct vm_area if (!folio) goto next; =20 - if (!pmdp_clear_young_notify(vma, addr, pmd + i)) + if (!pmdp_test_and_clear_young_notify(vma, addr, pmd + i)) goto next; =20 if (last !=3D folio) { @@ -4189,7 +4189,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); =20 - if (!ptep_clear_young_notify(vma, addr, pte)) + if (!ptep_test_and_clear_young_notify(vma, addr, pte)) return false; =20 if (spin_is_contended(pvmw->ptl)) @@ -4241,7 +4241,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) if (!folio) continue; =20 - if (!ptep_clear_young_notify(vma, addr, pte + i)) + if (!ptep_test_and_clear_young_notify(vma, addr, pte + i)) continue; =20 if (last !=3D folio) { --=20 2.47.3 From nobody Tue Apr 7 18:51:44 2026 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63ED33C1977 for ; Fri, 27 Feb 2026 09:44:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185500; cv=none; b=rqfJJUwcGIhx0V1olN4wcco2e6vJc/0EGdGw+w6nObKFQY2dEynAaLKuZKxa3AeWiw1iJ6cCwAzVLwAtyq49uMEzrg0C3sd9eOe9MmfZfnsS0OoBaFQPDSq2Dl4/iyNHez82W7ctmlrDRPS/jYjCNSPG0+7I5h3ogDzedDHgmPg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185500; c=relaxed/simple; bh=4HxKxjmwrSuGDoVoBNMI8AAdABa/ffdH9PHwu9Qn+Oo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=co8lBEip5r6P/OwamqXlB/kBIfO1Xl6+eKWjBi7PUkhBIS0bEBIJ0TKLxhpE+Mhuz6hJKuQOqm+tPtZjNFYr8PFuRa9rtkOtWkRElzAXMQmtUiRm0UVn58takNBPZSWYtu7Glyewt4s47/83C67cfyMFgQp6R0GTGWXmCPl3k5c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=MN52tlDr; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="MN52tlDr" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1772185496; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=TFvgQ+JoosgOuHvCEtm/wySIDfFeyNBVmJ3HBIM12WU=; b=MN52tlDrVxlU2YBhdh34hRUl2Or13cX+jf1/VE9Il5HznzvPMH2VeSdquDkZnElioB0KH3VcdWTCfGtgs4WWiPTUxF+uqDUasIzXgp2fnuCQsrZfZUBNN1S2+Lz6J09ghDqOgB1zZA4QhhdTmtuvZ/niybo3JyroYIwLu1fw5/Q= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Wzu0LeH_1772185494 cluster:ay36) by smtp.aliyun-inc.com; Fri, 27 Feb 2026 17:44:54 +0800 From: Baolin Wang To: akpm@linux-foundation.org, david@kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, riel@surriel.com, harry.yoo@oracle.com, jannh@google.com, willy@infradead.org, baohua@kernel.org, dev.jain@arm.com, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, hannes@cmpxchg.org, zhengqi.arch@bytedance.com, shakeel.butt@linux.dev, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/6] mm: rmap: add a ZONE_DEVICE folio warning in folio_referenced() Date: Fri, 27 Feb 2026 17:44:37 +0800 Message-ID: <6562be8c92e07f0a69c8ffd4016d1b1e55198af6.1772185080.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The folio_referenced() is used to test whether a folio was referenced during reclaim. Moreover, ZONE_DEVICE folios are controlled by their device driver, have a lifetime tied to that driver, and are never placed on the LRU list. That means we should never try to reclaim ZONE_DEVICE folios, so add a warn= ing to catch this unexpected behavior in folio_referenced() to avoid confusion, as discussed in the previous thread[1]. [1] https://lore.kernel.org/all/16fb7985-ec0f-4b56-91e7-404c5114f899@kernel= .org/ Reviewed-by: Alistair Popple Acked-by: David Hildenbrand (Arm) Signed-off-by: Baolin Wang --- mm/rmap.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/rmap.c b/mm/rmap.c index bff8f222004e..11cc6171344f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1065,6 +1065,7 @@ int folio_referenced(struct folio *folio, int is_lock= ed, .invalid_vma =3D invalid_folio_referenced_vma, }; =20 + VM_WARN_ON_ONCE_FOLIO(folio_is_zone_device(folio), folio); *vm_flags =3D 0; if (!pra.mapcount) return 0; --=20 2.47.3 From nobody Tue Apr 7 18:51:44 2026 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33C8F1F92E for ; Fri, 27 Feb 2026 09:44:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185502; cv=none; b=gUtch+uOhzNh4xGKx7XPGeHU1iqT04dDv+xEEP7G/Y8msQvfSuYcMTvDLbHl2BCxzl9EBmRXJiZCEMyEo4IhVktQkavCcwaJ77YnQOcDqnR9/6wRtqYzAEqEME1PvT6ATg3GNZ3hw1plA5TEwPenFz01vldf0LORIf7F5eIxBLg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185502; c=relaxed/simple; bh=csT0qHAE6gY9QRnQI3rHZIN8NUhGcIHGmO58ve5b6IY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FRI7gZJsm8ZWqHRDrhz0QUO+7v7nTSPyFSkUdsmOrAA89zAc0PpOmNL+G//inqdmaCjU23p4eT3Pluy3j8zHA5g9l3RcXD7KMg0cLf29jCA+Gq3ehtL5xgG0ArR8zInr/aMO8tQxLCcg+5e3YjgBNbX6lKmGxjWuoBvAghKp0PE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=uuynP2+i; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="uuynP2+i" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1772185498; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=1wgDGH7YrsjBc++BlCR5vPH+I+CcHMW2c+p9i33Zw9Q=; b=uuynP2+iI92fIwqR/h/EJ38AYEGQaHQwzk002THk3HlnC2p2OfhFJESDRaxRnGwYJN1GcIE44segWzwHZmXI1/ueLzoaJ3vZ3ZbVOaYZbBSy85n5gkFWZYbdVNIlssUxgGiUI/7q+Qq3NSV4QAE4Z0ylPVVjJA1vCqf8RjLEW/I= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Wzu12u4_1772185495 cluster:ay36) by smtp.aliyun-inc.com; Fri, 27 Feb 2026 17:44:56 +0800 From: Baolin Wang To: akpm@linux-foundation.org, david@kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, riel@surriel.com, harry.yoo@oracle.com, jannh@google.com, willy@infradead.org, baohua@kernel.org, dev.jain@arm.com, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, hannes@cmpxchg.org, zhengqi.arch@bytedance.com, shakeel.butt@linux.dev, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/6] mm: add a batched helper to clear the young flag for large folios Date: Fri, 27 Feb 2026 17:44:38 +0800 Message-ID: <589d743f4e048dc749002a7e1a1aec5d511c406b.1772185080.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, MGLRU will call ptep_test_and_clear_young_notify() to check and clear the young flag for each PTE sequentially, which is inefficient for large folios reclamation. Moreover, on Arm64 architecture, which supports contiguous PTEs, the Arm64- specific ptep_test_and_clear_young() already implements an optimization to clear the young flags for PTEs within a contiguous range. However, this is = not sufficient. Similar to the Arm64 specific clear_flush_young_ptes(), we can extend this to perform batched operations for the entire large folio (which might exceed the contiguous range: CONT_PTE_SIZE). Thus, we can introduce a new batched helper: test_and_clear_young_ptes() and its wrapper test_and_clear_young_ptes_notify() which are consistent with the existing functions, to perform batched checking of the young flags for large folios, which can help improve performance during large folio reclamation w= hen MGLRU is enabled. And it will be overridden by the architecture that implem= ents a more efficient batch operation in the following patches. Signed-off-by: Baolin Wang --- include/linux/pgtable.h | 38 ++++++++++++++++++++++++++++++++++++++ mm/internal.h | 16 +++++++++++----- 2 files changed, 49 insertions(+), 5 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 776993d4567b..29bd9fd04e1e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1103,6 +1103,44 @@ static inline int clear_flush_young_ptes(struct vm_a= rea_struct *vma, } #endif =20 +#ifndef test_and_clear_young_ptes +/** + * test_and_clear_young_ptes - Mark PTEs that map consecutive pages of the= same + * folio as old + * @vma: The virtual memory area the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to clear access bit. + * + * May be overridden by the architecture; otherwise, implemented as a simp= le + * loop over ptep_test_and_clear_young(). + * + * Note that PTE bits in the PTE range besides the PFN can differ. For exa= mple, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + * + * Returns: whether any PTE was young. + */ +static inline int test_and_clear_young_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr) +{ + int young =3D 0; + + for (;;) { + young |=3D ptep_test_and_clear_young(vma, addr, ptep); + if (--nr =3D=3D 0) + break; + ptep++; + addr +=3D PAGE_SIZE; + } + + return young; +} +#endif + /* * On some architectures hardware does not set page access bit when access= ing * memory page, it is responsibility of software setting this bit. It brin= gs diff --git a/mm/internal.h b/mm/internal.h index af04b177f21f..a5f0a264ad56 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1814,13 +1814,13 @@ static inline int pmdp_clear_flush_young_notify(str= uct vm_area_struct *vma, return young; } =20 -static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *= vma, - unsigned long addr, pte_t *ptep) +static inline int test_and_clear_young_ptes_notify(struct vm_area_struct *= vma, + unsigned long addr, pte_t *ptep, unsigned int nr) { int young; =20 - young =3D ptep_test_and_clear_young(vma, addr, ptep); - young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + young =3D test_and_clear_young_ptes(vma, addr, ptep, nr); + young |=3D mmu_notifier_clear_young(vma->vm_mm, addr, addr + nr * PAGE_SI= ZE); return young; } =20 @@ -1838,9 +1838,15 @@ static inline int pmdp_test_and_clear_young_notify(s= truct vm_area_struct *vma, =20 #define clear_flush_young_ptes_notify clear_flush_young_ptes #define pmdp_clear_flush_young_notify pmdp_clear_flush_young -#define ptep_test_and_clear_young_notify ptep_test_and_clear_young +#define test_and_clear_young_ptes_notify test_and_clear_young_ptes #define pmdp_test_and_clear_young_notify pmdp_test_and_clear_young =20 #endif /* CONFIG_MMU_NOTIFIER */ =20 +static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *= vma, + unsigned long addr, pte_t *ptep) +{ + return test_and_clear_young_ptes_notify(vma, addr, ptep, 1); +} + #endif /* __MM_INTERNAL_H */ --=20 2.47.3 From nobody Tue Apr 7 18:51:44 2026 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47ABA3921FD for ; Fri, 27 Feb 2026 09:45:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185505; cv=none; b=TAXcPzVVIxNfBXvZDPsYeUhX6niFuA0WgalFSrb/JZpihHJkTHt72C/3RUniTWn6OY8zxSHQUKRUQB2N+aFR6z4f7Ox/y14xynFEcUrCNWgI88wzoAMAEolWHKx/WL4VU4qhGBalST94G0wD90YVkKPBApxtvitUYXOxM+roN8k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185505; c=relaxed/simple; bh=9C3L9DVQJBeMLf07XC4ArfLWt/SOixfKFMj4jFauWlg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XPA56+vQCpA9R5EuBhjJJuyOtoeBPsbHarjIXfG5x2tFbIaMLiMcRmR0Xsn7cCnXoDR4JTbtmc9bV/QvWmJS8R5oD8Pzi8mlspRs2oT+oBWSWIU2LH5HyGVFA/gIQMhCrYHG/zWjA8vrupblUkw/CpDsBZfGLbmG2D66Dm22ZlA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=AzIBTTiA; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="AzIBTTiA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1772185500; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type; bh=eqA2PZyQDXj1Bd/fcvaoW0kfwGHOGzOHK70Gr5h6lbo=; b=AzIBTTiAMsWFS8Y9Q7Nx+Ptde2yuGeJ7ZpxS7Id4tCzeDTe71bLLMiLluzjKAbnRWH6TUirhlJbrmcb30++85E3x4ySihT2IAk4SYBOUqRUjm6GcUJo1IOc7V7Z15MWxKbSoEqe+i+tBkIRHMja6boTSNlGIJS+TFqmv5gYChls= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0WztzmPK_1772185497 cluster:ay36) by smtp.aliyun-inc.com; Fri, 27 Feb 2026 17:44:58 +0800 From: Baolin Wang To: akpm@linux-foundation.org, david@kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, riel@surriel.com, harry.yoo@oracle.com, jannh@google.com, willy@infradead.org, baohua@kernel.org, dev.jain@arm.com, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, hannes@cmpxchg.org, zhengqi.arch@bytedance.com, shakeel.butt@linux.dev, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/6] mm: support batched checking of the young flag for MGLRU Date: Fri, 27 Feb 2026 17:44:39 +0800 Message-ID: X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Use the batched helper test_and_clear_young_ptes_notify() to check and clear the young flag to improve the performance during large folio reclamation wh= en MGLRU is enabled. Meanwhile, we can also support batched checking the young and dirty flag when MGLRU walks the mm's pagetable to update the folios' generation counter. Since MGLRU also checks the PTE dirty bit, use folio_pte_batch_fla= gs() with FPB_MERGE_YOUNG_DIRTY set to detect batches of PTEs for a large folio. Then we can remove the ptep_test_and_clear_young_notify() since it has no users now. Note that we also update the 'young' counter and 'mm_stats[MM_LEAF_YOUNG]' = counter with the batched count in the lru_gen_look_around() and walk_pte_range(). H= owever, the batched operations may inflate these two counters, because in a large f= olio not all PTEs may have been accessed. (Additionally, tracking how many PTEs have= been accessed within a large folio is not very meaningful, since the mm core act= ually tracks access/dirty on a per-folio basis, not per page). The impact analysi= s is as follows: 1. The 'mm_stats[MM_LEAF_YOUNG]' counter has no functional impact and is ma= inly for debugging. 2. The 'young' counter is used to decide whether to place the current PMD e= ntry into the bloom filters by suitable_to_scan() (so that next time we can check whether= it has been accessed again), which may set the hash bit in the bloom filters for a PMD = entry that hasn=E2=80=99t seen much access. However, bloom filters inherently allow so= me error, so this effect appears negligible. Reviewed-by: Rik van Riel Signed-off-by: Baolin Wang --- include/linux/mmzone.h | 5 +++-- mm/internal.h | 6 ------ mm/rmap.c | 28 +++++++++++++-------------- mm/vmscan.c | 44 +++++++++++++++++++++++++++++++----------- 4 files changed, 50 insertions(+), 33 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 3e2f9c953ad4..66ad80b83baa 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -630,7 +630,7 @@ struct lru_gen_memcg { =20 void lru_gen_init_pgdat(struct pglist_data *pgdat); void lru_gen_init_lruvec(struct lruvec *lruvec); -bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int n= r); =20 void lru_gen_init_memcg(struct mem_cgroup *memcg); void lru_gen_exit_memcg(struct mem_cgroup *memcg); @@ -652,7 +652,8 @@ static inline void lru_gen_init_lruvec(struct lruvec *l= ruvec) { } =20 -static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, + unsigned int nr) { return false; } diff --git a/mm/internal.h b/mm/internal.h index a5f0a264ad56..a1b3967afe41 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1843,10 +1843,4 @@ static inline int pmdp_test_and_clear_young_notify(s= truct vm_area_struct *vma, =20 #endif /* CONFIG_MMU_NOTIFIER */ =20 -static inline int ptep_test_and_clear_young_notify(struct vm_area_struct *= vma, - unsigned long addr, pte_t *ptep) -{ - return test_and_clear_young_ptes_notify(vma, addr, ptep, 1); -} - #endif /* __MM_INTERNAL_H */ diff --git a/mm/rmap.c b/mm/rmap.c index 11cc6171344f..beb423f3e8ec 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -958,25 +958,21 @@ static bool folio_referenced_one(struct folio *folio, return false; } =20 + if (pvmw.pte && folio_test_large(folio)) { + const unsigned long end_addr =3D pmd_addr_end(address, vma->vm_end); + const unsigned int max_nr =3D (end_addr - address) >> PAGE_SHIFT; + pte_t pteval =3D ptep_get(pvmw.pte); + + nr =3D folio_pte_batch(folio, pvmw.pte, pteval, max_nr); + ptes +=3D nr; + } + if (lru_gen_enabled() && pvmw.pte) { - if (lru_gen_look_around(&pvmw)) + if (lru_gen_look_around(&pvmw, nr)) referenced++; } else if (pvmw.pte) { - if (folio_test_large(folio)) { - unsigned long end_addr =3D pmd_addr_end(address, vma->vm_end); - unsigned int max_nr =3D (end_addr - address) >> PAGE_SHIFT; - pte_t pteval =3D ptep_get(pvmw.pte); - - nr =3D folio_pte_batch(folio, pvmw.pte, - pteval, max_nr); - } - - ptes +=3D nr; if (clear_flush_young_ptes_notify(vma, address, pvmw.pte, nr)) referenced++; - /* Skip the batched PTEs */ - pvmw.pte +=3D nr - 1; - pvmw.address +=3D (nr - 1) * PAGE_SIZE; } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { if (pmdp_clear_flush_young_notify(vma, address, pvmw.pmd)) @@ -995,6 +991,10 @@ static bool folio_referenced_one(struct folio *folio, page_vma_mapped_walk_done(&pvmw); break; } + + /* Skip the batched PTEs */ + pvmw.pte +=3D nr - 1; + pvmw.address +=3D (nr - 1) * PAGE_SIZE; } =20 if (referenced) diff --git a/mm/vmscan.c b/mm/vmscan.c index 0a5622420987..7457b3c06fa3 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3474,6 +3474,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int gen =3D lru_gen_from_seq(max_seq); + unsigned int nr; pmd_t pmdval; =20 pte =3D pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval= , &ptl); @@ -3492,11 +3493,13 @@ static bool walk_pte_range(pmd_t *pmd, unsigned lon= g start, unsigned long end, =20 lazy_mmu_mode_enable(); restart: - for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { + for (i =3D pte_index(start), addr =3D start; addr !=3D end; i +=3D nr, ad= dr +=3D nr * PAGE_SIZE) { unsigned long pfn; struct folio *folio; - pte_t ptent =3D ptep_get(pte + i); + pte_t *cur_pte =3D pte + i; + pte_t ptent =3D ptep_get(cur_pte); =20 + nr =3D 1; total++; walk->mm_stats[MM_LEAF_TOTAL]++; =20 @@ -3508,7 +3511,16 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, if (!folio) continue; =20 - if (!ptep_test_and_clear_young_notify(args->vma, addr, pte + i)) + if (folio_test_large(folio)) { + const unsigned int max_nr =3D (end - addr) >> PAGE_SHIFT; + + nr =3D folio_pte_batch_flags(folio, NULL, cur_pte, &ptent, + max_nr, FPB_MERGE_YOUNG_DIRTY); + total +=3D nr - 1; + walk->mm_stats[MM_LEAF_TOTAL] +=3D nr - 1; + } + + if (!test_and_clear_young_ptes_notify(args->vma, addr, cur_pte, nr)) continue; =20 if (last !=3D folio) { @@ -3521,8 +3533,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, if (pte_dirty(ptent)) dirty =3D true; =20 - young++; - walk->mm_stats[MM_LEAF_YOUNG]++; + young +=3D nr; + walk->mm_stats[MM_LEAF_YOUNG] +=3D nr; } =20 walk_update_folio(walk, last, gen, dirty); @@ -4166,7 +4178,7 @@ static void lru_gen_age_node(struct pglist_data *pgda= t, struct scan_control *sc) * the PTE table to the Bloom filter. This forms a feedback loop between t= he * eviction and the aging. */ -bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int n= r) { int i; bool dirty; @@ -4184,12 +4196,13 @@ bool lru_gen_look_around(struct page_vma_mapped_wal= k *pvmw) struct lruvec *lruvec; struct lru_gen_mm_state *mm_state; unsigned long max_seq; + pte_t *cur_pte; int gen; =20 lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); =20 - if (!ptep_test_and_clear_young_notify(vma, addr, pte)) + if (!test_and_clear_young_ptes_notify(vma, addr, pte, nr)) return false; =20 if (spin_is_contended(pvmw->ptl)) @@ -4229,10 +4242,12 @@ bool lru_gen_look_around(struct page_vma_mapped_wal= k *pvmw) =20 pte -=3D (addr - start) / PAGE_SIZE; =20 - for (i =3D 0, addr =3D start; addr !=3D end; i++, addr +=3D PAGE_SIZE) { + for (i =3D 0, addr =3D start, cur_pte =3D pte; addr !=3D end; + i +=3D nr, cur_pte +=3D nr, addr +=3D nr * PAGE_SIZE) { unsigned long pfn; - pte_t ptent =3D ptep_get(pte + i); + pte_t ptent =3D ptep_get(cur_pte); =20 + nr =3D 1; pfn =3D get_pte_pfn(ptent, vma, addr, pgdat); if (pfn =3D=3D -1) continue; @@ -4241,7 +4256,14 @@ bool lru_gen_look_around(struct page_vma_mapped_walk= *pvmw) if (!folio) continue; =20 - if (!ptep_test_and_clear_young_notify(vma, addr, pte + i)) + if (folio_test_large(folio)) { + const unsigned int max_nr =3D (end - addr) >> PAGE_SHIFT; + + nr =3D folio_pte_batch_flags(folio, NULL, cur_pte, &ptent, + max_nr, FPB_MERGE_YOUNG_DIRTY); + } + + if (!test_and_clear_young_ptes_notify(vma, addr, cur_pte, nr)) continue; =20 if (last !=3D folio) { @@ -4254,7 +4276,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) if (pte_dirty(ptent)) dirty =3D true; =20 - young++; + young +=3D nr; } =20 walk_update_folio(walk, last, gen, dirty); --=20 2.47.3 From nobody Tue Apr 7 18:51:44 2026 Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA7313E8C72 for ; Fri, 27 Feb 2026 09:45:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.101 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185507; cv=none; b=LfiWLcU9k4zIYr9Tgu0LT9FNhUDzhBi8N5SSofMWoXCrlpxMkl31fRkDvmCs5SWdfZZKE46LgAJqoXKoyZvkTDGndsdEkD+YiJA5WpTWwQiw3M/fAJ5B5hOSqfeigLCcPgNdBEFho71miBsVvtBoRx6QkUb6ZIgEFnyJeW7O6TU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772185507; c=relaxed/simple; bh=xQSoH8Ux9Ilv+cYOKrx/tHuqnVK+e+Pu2dYDUnQwH7c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DVYJG5+bTC5YvdTB3D+9SbfeCYnRyKXQa2mnxjgdq/lbZdx5Jl/wpA8S+Uki2ATvN09gSim6kqjzWdCzp7J7EOj7D0xpZrmE+QTDb0pC1idVF69hs+/H3Vh0CHFJd/cFl2cyaNqhpYFHBlGXMpiMNw5HcpVjN/GNGyFtKYLlpI8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=gWj2RQFk; arc=none smtp.client-ip=115.124.30.101 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="gWj2RQFk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1772185502; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=Z89VjCPOsBTAMzdOkBmmkyGLdD2D9prjOEPgmVbdWLg=; b=gWj2RQFkdoGA7uoAOoGEFtWa4/+0mCvN/pZ2+w82Im7Y0CpnSU4D35BwMgibO+KbMkr6X4fhxpfMxm+sYqesY6jX6hUoZpYUxinIsd6YxTEZovA0aS/TJGXFw7Te+CgBMu6EpSVV94WIplY0l//XLbXFmmRQNjNV29NnotfyNmU= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0Wzu12vf_1772185499 cluster:ay36) by smtp.aliyun-inc.com; Fri, 27 Feb 2026 17:44:59 +0800 From: Baolin Wang To: akpm@linux-foundation.org, david@kernel.org Cc: catalin.marinas@arm.com, will@kernel.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, riel@surriel.com, harry.yoo@oracle.com, jannh@google.com, willy@infradead.org, baohua@kernel.org, dev.jain@arm.com, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, hannes@cmpxchg.org, zhengqi.arch@bytedance.com, shakeel.butt@linux.dev, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 6/6] arm64: mm: implement the architecture-specific test_and_clear_young_ptes() Date: Fri, 27 Feb 2026 17:44:40 +0800 Message-ID: <5d9298b94607b2bf4f1f92ea29a4c96217c5bcc1.1772185080.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the Arm64 architecture-specific test_and_clear_young_ptes() to en= able batched checking of young flags, improving performance during large folio reclamation when MGLRU is enabled. While we're at it, simplify ptep_test_and_clear_young() by calling test_and_clear_young_ptes(). Since callers guarantee that PTEs are present before calling these functions, we can use pte_cont() to check the CONT_PTE flag instead of pte_valid_cont(). Performance testing: Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a mem= ory cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim int= erface. I can observe 60%+ performance improvement on my Arm64 32-core server (and = about 15% improvement on my X86 machine). W/o patchset: real 0m0.470s user 0m0.000s sys 0m0.470s W/ patchset: real 0m0.180s user 0m0.001s sys 0m0.179s Reviewed-by: Rik van Riel Signed-off-by: Baolin Wang --- arch/arm64/include/asm/pgtable.h | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index aa4b13da6371..ab451d20e4c5 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1812,16 +1812,22 @@ static inline pte_t ptep_get_and_clear(struct mm_st= ruct *mm, return __ptep_get_and_clear(mm, addr, ptep); } =20 +#define test_and_clear_young_ptes test_and_clear_young_ptes +static inline int test_and_clear_young_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr) +{ + if (likely(nr =3D=3D 1 && !pte_cont(__ptep_get(ptep)))) + return __ptep_test_and_clear_young(vma, addr, ptep); + + return contpte_test_and_clear_young_ptes(vma, addr, ptep, nr); +} + #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { - pte_t orig_pte =3D __ptep_get(ptep); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_test_and_clear_young(vma, addr, ptep); - - return contpte_test_and_clear_young_ptes(vma, addr, ptep, 1); + return test_and_clear_young_ptes(vma, addr, ptep, 1); } =20 #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH --=20 2.47.3