Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable
batched checking of young flags, improving performance during large folio
reclamation when MGLRU is enabled.
While we're at it, simplify ptep_test_and_clear_young() by calling
test_and_clear_young_ptes(). Since callers guarantee that PTEs are present
before calling these functions, we can use pte_cont() to check the CONT_PTE
flag instead of pte_valid_cont().
Performance testing:
Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory
cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface.
I can observe 60%+ performance improvement on my Arm64 32-core server (and about
15% improvement on my X86 machine).
W/o patchset:
real 0m0.470s
user 0m0.000s
sys 0m0.470s
W/ patchset:
real 0m0.180s
user 0m0.001s
sys 0m0.179s
Reviewed-by: Rik van Riel <riel@surriel.com>
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
---
arch/arm64/include/asm/pgtable.h | 18 ++++++++++++------
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index aa4b13da6371..ab451d20e4c5 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1812,16 +1812,22 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
return __ptep_get_and_clear(mm, addr, ptep);
}
+#define test_and_clear_young_ptes test_and_clear_young_ptes
+static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ if (likely(nr == 1 && !pte_cont(__ptep_get(ptep))))
+ return __ptep_test_and_clear_young(vma, addr, ptep);
+
+ return contpte_test_and_clear_young_ptes(vma, addr, ptep, nr);
+}
+
#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
static inline int ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep)
{
- pte_t orig_pte = __ptep_get(ptep);
-
- if (likely(!pte_valid_cont(orig_pte)))
- return __ptep_test_and_clear_young(vma, addr, ptep);
-
- return contpte_test_and_clear_young_ptes(vma, addr, ptep, 1);
+ return test_and_clear_young_ptes(vma, addr, ptep, 1);
}
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
--
2.47.3On 3/6/26 07:43, Baolin Wang wrote: > Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable > batched checking of young flags, improving performance during large folio > reclamation when MGLRU is enabled. > > While we're at it, simplify ptep_test_and_clear_young() by calling > test_and_clear_young_ptes(). Since callers guarantee that PTEs are present > before calling these functions, we can use pte_cont() to check the CONT_PTE > flag instead of pte_valid_cont(). > > Performance testing: > Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory > cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface. > I can observe 60%+ performance improvement on my Arm64 32-core server (and about > 15% improvement on my X86 machine). > > W/o patchset: > real 0m0.470s > user 0m0.000s > sys 0m0.470s > > W/ patchset: > real 0m0.180s > user 0m0.001s > sys 0m0.179s > > Reviewed-by: Rik van Riel <riel@surriel.com> > Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com> > --- Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> -- Cheers, David
On 3/6/26 07:43, Baolin Wang wrote:
> Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable
> batched checking of young flags, improving performance during large folio
> reclamation when MGLRU is enabled.
>
> While we're at it, simplify ptep_test_and_clear_young() by calling
> test_and_clear_young_ptes(). Since callers guarantee that PTEs are present
> before calling these functions, we can use pte_cont() to check the CONT_PTE
> flag instead of pte_valid_cont().
>
> Performance testing:
> Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory
> cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface.
> I can observe 60%+ performance improvement on my Arm64 32-core server (and about
> 15% improvement on my X86 machine).
>
> W/o patchset:
> real 0m0.470s
> user 0m0.000s
> sys 0m0.470s
>
> W/ patchset:
> real 0m0.180s
> user 0m0.001s
> sys 0m0.179s
>
> Reviewed-by: Rik van Riel <riel@surriel.com>
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> ---
> arch/arm64/include/asm/pgtable.h | 18 ++++++++++++------
> 1 file changed, 12 insertions(+), 6 deletions(-)
>
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index aa4b13da6371..ab451d20e4c5 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -1812,16 +1812,22 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
> return __ptep_get_and_clear(mm, addr, ptep);
> }
>
> +#define test_and_clear_young_ptes test_and_clear_young_ptes
> +static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
> + unsigned long addr, pte_t *ptep,
> + unsigned int nr)
> +{
> + if (likely(nr == 1 && !pte_cont(__ptep_get(ptep))))
> + return __ptep_test_and_clear_young(vma, addr, ptep);
> +
> + return contpte_test_and_clear_young_ptes(vma, addr, ptep, nr);
> +}
Thinking out loud, what would happen if
(a) The range spans multiple possible cont ranges (like, 64 ptes).
(b) The first pte is !pte_cont(), but some others in there are?
--
Cheers,
David
On 3/6/26 10:47 PM, David Hildenbrand (Arm) wrote:
> On 3/6/26 07:43, Baolin Wang wrote:
>> Implement the Arm64 architecture-specific test_and_clear_young_ptes() to enable
>> batched checking of young flags, improving performance during large folio
>> reclamation when MGLRU is enabled.
>>
>> While we're at it, simplify ptep_test_and_clear_young() by calling
>> test_and_clear_young_ptes(). Since callers guarantee that PTEs are present
>> before calling these functions, we can use pte_cont() to check the CONT_PTE
>> flag instead of pte_valid_cont().
>>
>> Performance testing:
>> Enable MGLRU, then allocate 10G clean file-backed folios by mmap() in a memory
>> cgroup, and try to reclaim 8G file-backed folios via the memory.reclaim interface.
>> I can observe 60%+ performance improvement on my Arm64 32-core server (and about
>> 15% improvement on my X86 machine).
>>
>> W/o patchset:
>> real 0m0.470s
>> user 0m0.000s
>> sys 0m0.470s
>>
>> W/ patchset:
>> real 0m0.180s
>> user 0m0.001s
>> sys 0m0.179s
>>
>> Reviewed-by: Rik van Riel <riel@surriel.com>
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>> ---
>> arch/arm64/include/asm/pgtable.h | 18 ++++++++++++------
>> 1 file changed, 12 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
>> index aa4b13da6371..ab451d20e4c5 100644
>> --- a/arch/arm64/include/asm/pgtable.h
>> +++ b/arch/arm64/include/asm/pgtable.h
>> @@ -1812,16 +1812,22 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
>> return __ptep_get_and_clear(mm, addr, ptep);
>> }
>>
>> +#define test_and_clear_young_ptes test_and_clear_young_ptes
>> +static inline int test_and_clear_young_ptes(struct vm_area_struct *vma,
>> + unsigned long addr, pte_t *ptep,
>> + unsigned int nr)
>> +{
>> + if (likely(nr == 1 && !pte_cont(__ptep_get(ptep))))
>> + return __ptep_test_and_clear_young(vma, addr, ptep);
>> +
>> + return contpte_test_and_clear_young_ptes(vma, addr, ptep, nr);
>> +}
>
> Thinking out loud, what would happen if
Good questions, I think the contpte_test_and_clear_young_ptes() takes
that into account.
> (a) The range spans multiple possible cont ranges (like, 64 ptes).
The contpte_test_and_clear_young_ptes() will call
contpte_align_addr_ptep() to align the range to cont‑block boundary,
that means the range can span multiple cont blocks.
int contpte_test_and_clear_young_ptes(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
unsigned int nr)
{
unsigned long end = addr + nr * PAGE_SIZE;
int young = 0;
ptep = contpte_align_addr_ptep(&addr, &end, ptep, nr);
for (; addr != end; ptep++, addr += PAGE_SIZE)
young |= __ptep_test_and_clear_young(vma, addr, ptep);
return young;
}
>
> (b) The first pte is !pte_cont(), but some others in there are?
IMO they can’t be handled in a single batch. Since the folio_pte_batch()
will group consecutive !cont PTEs into one batch and consecutive cont
PTEs into another (assume all PTEs belong to a single large folio),
because their PTE entries have different CONT bits.
Even if the callers do so, contpte_align_addr_ptep() will check the
pte_cont() of the start and end address to align the range appropriately.
>> >> (b) The first pte is !pte_cont(), but some others in there are? > > IMO they can’t be handled in a single batch. Since the folio_pte_batch() > will group consecutive !cont PTEs into one batch and consecutive cont > PTEs into another (assume all PTEs belong to a single large folio), > because their PTE entries have different CONT bits. Interesting, thanks. I thought that folio_pte_batch() would be able to batch that. But yes, pte_batch_hint() relies on the CONT bit still being set after a ptep_get(). I wonder whether we should document somewhere that the arm implementation depends on that. This might be something to look into in the future. (make folio_pte_batch() ignore cont information when comparing and make the arm implementation be able to deal with that). Assume we unmapped the last page of a large folio. Ideally, we'd be able to process the remaining THP pieces (all ptes) in a single operation. I guess right now it would be two. -- Cheers, David
On 3/9/26 10:39 PM, David Hildenbrand (Arm) wrote: >>> >>> (b) The first pte is !pte_cont(), but some others in there are? >> >> IMO they can’t be handled in a single batch. Since the folio_pte_batch() >> will group consecutive !cont PTEs into one batch and consecutive cont >> PTEs into another (assume all PTEs belong to a single large folio), >> because their PTE entries have different CONT bits. > Interesting, thanks. I thought that folio_pte_batch() would be able to > batch that. > > But yes, pte_batch_hint() relies on the CONT bit still being set after a > ptep_get(). I wonder whether we should document somewhere that the arm > implementation depends on that. > > This might be something to look into in the future. (make > folio_pte_batch() ignore cont information when comparing and make the > arm implementation be able to deal with that). > > Assume we unmapped the last page of a large folio. Ideally, we'd be able > to process the remaining THP pieces (all ptes) in a single operation. I > guess right now it would be two. Right. Let me investigate this further and see if I can figure out how to optimize this case. Thanks for reviewing.
© 2016 - 2026 Red Hat, Inc.