After a long session of performance-cat herding, here's the first version
I am relatively ok with.
Micro-optimize the change_protection functionality and the
change_pte_range() routine. This set of functions works in an incredibly
tight loop, and even small inefficiencies are incredibly evident when spun
hundreds, thousands or hundreds of thousands of times.
There was an attempt to keep the batching functionality as much as possible,
which introduced some part of the slowness, but not all of it. Removing it
for !arm64 architectures would speed mprotect() up even further, but could
easily pessimize cases where large folios are mapped (which is not as rare
as it seems, particularly when it comes to the page cache these days).
The micro-benchmark used for the tests was [0] (usable using google/benchmark
and g++ -O2 -lbenchmark repro.cpp)
This resulted in the following (first entry is baseline):
---------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------
mprotect_bench 85967 ns 85967 ns 6935
mprotect_bench 82402 ns 82402 ns 6745
mprotect_bench 86776 ns 86776 ns 8100
mprotect_bench 86463 ns 86463 ns 8087
mprotect_bench 73374 ns 73373 ns 9602
After the patchset we can observe a 14% speedup in mprotect. Wonderful
for the elusive mprotect-based workloads!
Testing & more ideas welcome. I suspect there is plenty of improvement possible
but it would require more time than what I have on my hands right now. The
entire inlined function (which inlines into change_protection()) is gigantic
- I'm not surprised this is so finnicky.
[0]: https://gist.github.com/heatd/1450d273005aba91fa5744f44dfcd933
Link: https://lore.kernel.org/all/aY8-XuFZ7zCvXulB@luyang-thinkpadp1gen7.toromso.csb/
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Luke Yang <luyang@redhat.com>
Cc: jhladky@redhat.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Pedro Falcato (4):
mm/mprotect: encourage inlining with __always_inline
mm/mprotect: move softleaf code out of the main function
mm/mprotect: un-inline folio_pte_batch_flags()
mm/mprotect: special-case small folios when applying write permissions
mm/mprotect.c | 158 +++++++++++++++++++++++++++-----------------------
1 file changed, 85 insertions(+), 73 deletions(-)
--
2.53.0