This series adds multi-page clearing for hugepages, improving on the
current page-at-a-time approach in two ways:
- amortizes the per-page setup cost over a larger extent
- when using string instructions, exposes the real region size to the
processor. A processor could use that as a hint to optimize based
on the full extent size. AMD Zen uarchs, as an example, elide
allocation of cachelines for regions larger than L3-size.
Demand faulting a 64GB region shows good performance improvements:
$ perf bench mem map -p $page-size -f demand -s 64GB -l 5
mm/folio_zero_user x86/folio_zero_user change
(GB/s +- %stdev) (GB/s +- %stdev)
pg-sz=2MB 11.82 +- 0.67% 16.48 +- 0.30% + 39.4%
pg-sz=1GB 17.51 +- 1.19% 40.03 +- 7.26% [#] +129.9%
[#] Only with preempt=full|lazy because cooperatively preempted models
need regular invocations of cond_resched(). This limits the extent
sizes that can be cleared as a unit.
Raghavendra also tested on AMD Genoa and that shows similar
improvements [1].
Series structure:
Patches 1-5, 8,
"perf bench mem: Remove repetition around time measurement"
"perf bench mem: Defer type munging of size to float"
"perf bench mem: Move mem op parameters into a structure"
"perf bench mem: Pull out init/fini logic"
"perf bench mem: Switch from zalloc() to mmap()"
"perf bench mem: Refactor mem_options"
refactor, and patches 6-7, 9
"perf bench mem: Allow mapping of hugepages"
"perf bench mem: Allow chunking on a memory region"
"perf bench mem: Add mmap() workload"
add a few new perf bench mem workloads (chunking and mapping performance).
Patches 10-11,
"x86/mm: Simplify clear_page_*"
"x86/clear_page: Introduce clear_pages()"
inlines the ERMS and REP_GOOD implementations used from clear_page()
and adds clear_pages() to handle page extents.
And finally patches 12-13, allow an arch override to folio_zero_user()
and provide the x86 implementation that can do the actual multi-page
clearing.
"mm: memory: allow arch override for folio_zero_user()"
"x86/folio_zero_user: Add multi-page clearing"
Changelog:
v4:
- adds perf bench workloads to exercise mmap() populate/demand-fault (Ingo)
- inline stosb etc (PeterZ)
- handle cooperative preemption models (Ingo)
- interface and other cleanups all over (Ingo)
v3:
- get rid of preemption dependency (TIF_ALLOW_RESCHED); this version
was limited to preempt=full|lazy.
- override folio_zero_user() (Linus)
(https://lore.kernel.org/lkml/20250414034607.762653-1-ankur.a.arora@oracle.com/)
v2:
- addressed review comments from peterz, tglx.
- Removed clear_user_pages(), and CONFIG_X86_32:clear_pages()
- General code cleanup
(https://lore.kernel.org/lkml/20230830184958.2333078-1-ankur.a.arora@oracle.com/)
Comments appreciated!
Also at:
github.com/terminus/linux clear-pages.v4
[1] https://lore.kernel.org/lkml/0d6ba41c-0c90-4130-896a-26eabbd5bd24@amd.com/
Ankur Arora (13):
perf bench mem: Remove repetition around time measurement
perf bench mem: Defer type munging of size to float
perf bench mem: Move mem op parameters into a structure
perf bench mem: Pull out init/fini logic
perf bench mem: Switch from zalloc() to mmap()
perf bench mem: Allow mapping of hugepages
perf bench mem: Allow chunking on a memory region
perf bench mem: Refactor mem_options
perf bench mem: Add mmap() workload
x86/mm: Simplify clear_page_*
x86/clear_page: Introduce clear_pages()
mm: memory: allow arch override for folio_zero_user()
x86/folio_zero_user: Add multi-page clearing
arch/x86/include/asm/page_32.h | 18 +-
arch/x86/include/asm/page_64.h | 38 +-
arch/x86/lib/clear_page_64.S | 39 +-
arch/x86/mm/Makefile | 1 +
arch/x86/mm/memory.c | 97 +++++
mm/memory.c | 5 +-
tools/perf/bench/bench.h | 1 +
tools/perf/bench/mem-functions.c | 391 ++++++++++++++-----
tools/perf/bench/mem-memcpy-arch.h | 2 +-
tools/perf/bench/mem-memcpy-x86-64-asm-def.h | 4 +
tools/perf/bench/mem-memset-arch.h | 2 +-
tools/perf/bench/mem-memset-x86-64-asm-def.h | 4 +
tools/perf/builtin-bench.c | 1 +
13 files changed, 452 insertions(+), 151 deletions(-)
create mode 100644 arch/x86/mm/memory.c
--
2.43.5