arch/arm64/Kconfig | 12 +++++++--- arch/arm64/include/asm/mmu.h | 5 ++++ arch/arm64/include/asm/mmu_context.h | 9 +++++++- arch/arm64/include/asm/percpu.h | 37 ++++++++++++++++++++++++++++- arch/arm64/include/asm/pgalloc.h | 24 +++++++++++++++++++ arch/arm64/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++--- arch/arm64/kernel/setup.c | 3 +++ arch/arm64/kernel/smp.c | 44 +++++++++++++++++++++++++++++++++++ arch/arm64/mm/kasan_init.c | 47 +++++++++++++++++++++++-------------- arch/arm64/mm/mmu.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------- arch/arm64/mm/ptdump.c | 4 ++++ arch/riscv/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++++++++ drivers/base/arch_numa.c | 51 +--------------------------------------- include/linux/mm.h | 11 +++++++++ include/linux/percpu.h | 4 +++- include/linux/pgalloc.h | 13 +++++++++++ include/linux/vmalloc.h | 3 +++ mm/Kconfig | 9 ++++++++ mm/internal.h | 5 +++- mm/kmsan/hooks.c | 14 +++++------ mm/percpu-internal.h | 14 +++++++++++ mm/percpu-vm.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/percpu.c | 58 +++++++++++++++++++++++++++++++++++++--------- mm/sparse-vmemmap.c | 4 ++-- mm/vmalloc.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------- 25 files changed, 712 insertions(+), 144 deletions(-)
Hi,
This is v2 RFC. In v2 a lot problems found out by Sashiko were fixed and more
feature gaps were closed (please see the below changelog for the details).
Although there are still some open issues, for example, it just can support
48 bits VA (for 4K and 64K) and 47 bits VA (for 16K), KPTI support has not
been solved yet, etc, but I think the delta should be big enough and worth
a new RFC to gather comments in order to make sure I'm on the right track.
Some more benchmarks were done, for example, some latency related benchmarks
that I mentioned at LSFMM because I thought responsiveness should be improved
due to the removal of preempt_disable. Collected more PMU counters as well.
Please refer to the benchmark section for more details.
Look forward to comments.
Changelog
v2: * Added 3-level and 2-level page table support.
* Tested with 16K and 64K page size. But we just support 48 bits VA (4K
and 64K) and 47 bits VA with 16K for now. Please refer to the below
"known issue" section for the detail reason.
* Added support for memory hotplug.
* Added support for KASAN (generic).
* Treated percpu and local percpu area address as vmalloc address.
* Fixed build failure for x86.
* Fixed build failure for !CONFIG_NUMA.
* Added KASAN support for local percpu area.
* Some other misc bug fixes found out by Sashiko.
* More code refactor and cleanup.
* Regorganized the patches.
* More benchmarks, refer to benchmark section for more details.
* Rebased to v7.2-rc1.
Introduction
============
This patch series implemented the LSFMM 2026 proposal for optimizing
this_cpu_*() ops on ARM64. For the details of the proposal, Please refer to:
https://lore.kernel.org/linux-mm/CAHbLzkpcN-T8MH6=W3jCxcFj1gVZp8fRqe231yzZT-rV_E_org@mail.gmail.com/
I didn't repeat it in the cover letter because there is no change to the
proposal.
The series is based on 7.1-rc1. It is basically minimum viable patches.
There are still a few hacks in this series and it may break something,
for example, KPTI, SMT machines which shared TLB, etc. But it shoule be
good enough for now to demonstrate the core idea. The main purpose of the
RFC is to gather feedback, figure out missing parts and risks, and make sure
we are on the right track, as well as hopefully it can help the discussion
for the upcoming LSFMM.
I broke the patches down to arch-dependent and arch-independent parts so that
hopefully the interested persons can do experiments on other architectures,
for example, S390, easier.
A new kernel config is introduced, HAVE_LOCAL_PER_CPU_MAP. The architectures
which can support this feature will select it. Allocating and freeing percpu
local mapping is protected by this config so that others won't pay the cost.
Known Issues
============
1. KPTI
-------
We need determine what CPU we are on, then switch to the right page table.
Currently arm64 kernel fetches tramp_pg_dir via swapper_pg_dir - fixed_offset,
and fetches swapper_pg_dir from ttbr1. But ttbr1 may not hold swapper_pg_dir
anymore except CPU #0. So we need to figure out the other way to handle it.
Switching to tramp_pg_dir should be easy, but the reverse seems harder because
tramp_pg_dir just maps the trampoline vectors.
Maybe we can do two steps switch. Switch to swapper_pg_dir at the first step,
then switch to per cpu page table (for entry) or tramp page table (for exit).
Nobody should call this_cpu_*() at either userspace -> kernel entry stage or
kernel -> userspace exit stage.
2. SW PAN
---------
Has the similar issue as KPTI. It installs reserved_pg_dir to TTBR0 when running
in kernel space, but fetching reserved_pg_dir via swapper_pg_dir - fixed_offset.
Maybe we can save the physical address of swapper_pg_dir in a variable, then load
it from that variable instead of ttbr1.
3. Shared TLB machines
----------------------
Some machines may share TLB between CPUs, for example, SMT machines may share
TLB between the two hardware threads in one core.
The per cpu page table just can't work with it. Maybe we need a new
cpufeature to indicate whether per cpu page table is allowed or not. Then
just enable it for not-shared-TLB machines.
4. Don't support all VA bits
----------------------------
We just support 48 bits VA (4K and 64K) and 47 bits VA (16K) for now. For 4K
and 64K, supporting other VA bits is not hard, we just need to determine the
size for percpu and local percpu area.
But it is harder for supporting 48 bits VA + 16K page size. We just have two
top level kernel page table entries with this configuration, but we assume we
just need to sync up kernel page table at the top level for now. We need to
sync up kernel page table at the second level in order to support it. I'm not
sure whether it is worth it or not.
Benchmark
=========
The benchmarks are done on 160 core AmpereOne machine. The baseline is
v7.2-rc1 kernel.
1. Reduction of kernel text size
--------------------------------
The patchset can reduce at least 11 instructions for this_cpu_*() ops. Both
preempt_disable() and preempt_enable() need 4 instructions to manipulate
the preempt count, and preempt_enable() needs more instructions (compare +
READ + compare) to determine whether reschedule is needed or not.
Because this_cpu_*() ops are inlined and called in a lot of places so we
can save a lot of instructions.
The size of kernel text is reduced by ~184KB with default Fedora kernel
config. This also helps reduce kernel icache miss rate and stalled frontend
cycles as kernel build benchmark result showed.
2. Kernel Build
---------------
Run kernel build (make -j160) with the default Fedora kernel config in a
memcg.
13% - 18% sys time improvment
3% - 7% wall time improvement
5% fewer kernel icache miss, 5% fewer executed kernel instructions and
15% fewer stalled frontend cycles for kernel.
3. stress-ng vm ops
-------------------
stress-ng --vm 160 --vm-bytes 128M --vm-ops 100000000
8.5% improvement
4. stress-ng vm ops + fork
--------------------------
stress-ng --mmapfork 160 --mmapfork-bytes 128M --mmapfork-ops 500
15% improvement
5. Specjbb
----------
The specjbb test latency curves showed the patched kernel has consistently
lower p99 latency (the lower the better) than the baseline.
2.5% improvement on max-jOPS and 4% - 5% improvement on critical-jOPS.
The specjbb benchmark is quite sensitive to latency and responsiveness,
particularly critical-jOPS result. The patches are supposed to improve the
responsiveness due to the reduction of preempt-disabled critical sections.
6. MySQL
--------
1% - 2% gains on read-only test, 2% - 4% gains on write-only test. Also see
15% decrease on frontend cache stall.
Regression test
===============
1. memcg creation
-----------------
Create 10K memcgs. Each memcg creation needs to allocate multiple percpu
variables, for example, percpu refcnt, rstat and objcg percpu refcnt.
Consumed 2112K more virtual memory for percpu “local mapping” and a few
more mega bytes consumed by per cpu page tables.
No noticeable regression was found for elapsed time.
2. fork test
------------
stress-ng --fork 160 --fork-ops 10000000
fork() needs to allocate multiple percpu variables, for example, rss
counters and mm_cid_cpu.
Roughly 1% regression was found. However stress-ng fork test has quites
small address space, the real life workloads typically have much larger
address space and do more complicated works. The stress-ng mmapfork
benchmark saw 15% improvement.
The organization of patches
===========================
The refactor and prepatory patches (patch 1 - patch 4)
Percpu page table support patches (patch 5 - patch 8)
Local percpu area support patches (patch 7 - patch 15)
Use local percpu area for this_cpu ops (patch 16)
Yang Shi (16):
drivers: arch_numa: move percpu set up code to arch
arm64: kconfig: make percpu related configs not depend on NUMA
mm: pgalloc: introduce {pud|pmd}_populate_sync()
vmalloc: pass in pgd pointer for vmap{__vunmap}_range_noflush()
arm64: mm: enable percpu kernel page table
arm64: mm: defined {pud|pmd}_populate_sync()
arm64: mm: sync percpu page table for memory hotplug/unplug
arm64: kasan: sync up kasan shadow area page table
arm64: mm: define percpu virtual space area
mm: percpu: prepare to use dedicated percpu area
arm64: mm: map local percpu first chunk
mm: percpu: set up first chunk and reserve chunk
arm64: mm: introduce __per_cpu_local_off
mm: percpu: allocate and free local percpu vm area
arm64: kconfig: select HAVE_LOCAL_PER_CPU_MAP
arm64: percpu: use local percpu for this_cpu_*() APIs
arch/arm64/Kconfig | 12 +++++++---
arch/arm64/include/asm/mmu.h | 5 ++++
arch/arm64/include/asm/mmu_context.h | 9 +++++++-
arch/arm64/include/asm/percpu.h | 37 ++++++++++++++++++++++++++++-
arch/arm64/include/asm/pgalloc.h | 24 +++++++++++++++++++
arch/arm64/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++---
arch/arm64/kernel/setup.c | 3 +++
arch/arm64/kernel/smp.c | 44 +++++++++++++++++++++++++++++++++++
arch/arm64/mm/kasan_init.c | 47 +++++++++++++++++++++++--------------
arch/arm64/mm/mmu.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------
arch/arm64/mm/ptdump.c | 4 ++++
arch/riscv/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++++++++
drivers/base/arch_numa.c | 51 +---------------------------------------
include/linux/mm.h | 11 +++++++++
include/linux/percpu.h | 4 +++-
include/linux/pgalloc.h | 13 +++++++++++
include/linux/vmalloc.h | 3 +++
mm/Kconfig | 9 ++++++++
mm/internal.h | 5 +++-
mm/kmsan/hooks.c | 14 +++++------
mm/percpu-internal.h | 14 +++++++++++
mm/percpu-vm.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
mm/percpu.c | 58 +++++++++++++++++++++++++++++++++++++---------
mm/sparse-vmemmap.c | 4 ++--
mm/vmalloc.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------
25 files changed, 712 insertions(+), 144 deletions(-)
Thanks,
Yang
On 15/07/2026 19:04, Yang Shi wrote:
>
> Hi,
>
> This is v2 RFC. In v2 a lot problems found out by Sashiko were fixed and more
> feature gaps were closed (please see the below changelog for the details).
> Although there are still some open issues, for example, it just can support
> 48 bits VA (for 4K and 64K) and 47 bits VA (for 16K), KPTI support has not
> been solved yet, etc, but I think the delta should be big enough and worth
> a new RFC to gather comments in order to make sure I'm on the right track.
>
> Some more benchmarks were done, for example, some latency related benchmarks
> that I mentioned at LSFMM because I thought responsiveness should be improved
> due to the removal of preempt_disable. Collected more PMU counters as well.
> Please refer to the benchmark section for more details.
Hi Yang,
When we spoke about this at LSFMM, I said I would take the series for a spin on
other hardware to see if the performance gains generalise. I did make a start on
that but the kernel wouldn't even boot and I never found time to debug. Sorry
that I didn't let you know sooner.
But I think that there are other issues with this concept that likely block it
from being accepted upstream; see below...
>
> Look forward to comments.
>
>
> Changelog
> v2: * Added 3-level and 2-level page table support.
> * Tested with 16K and 64K page size. But we just support 48 bits VA (4K
> and 64K) and 47 bits VA with 16K for now. Please refer to the below
> "known issue" section for the detail reason.
> * Added support for memory hotplug.
> * Added support for KASAN (generic).
> * Treated percpu and local percpu area address as vmalloc address.
> * Fixed build failure for x86.
> * Fixed build failure for !CONFIG_NUMA.
> * Added KASAN support for local percpu area.
> * Some other misc bug fixes found out by Sashiko.
> * More code refactor and cleanup.
> * Regorganized the patches.
> * More benchmarks, refer to benchmark section for more details.
> * Rebased to v7.2-rc1.
>
>
> Introduction
> ============
> This patch series implemented the LSFMM 2026 proposal for optimizing
> this_cpu_*() ops on ARM64. For the details of the proposal, Please refer to:
> https://lore.kernel.org/linux-mm/CAHbLzkpcN-T8MH6=W3jCxcFj1gVZp8fRqe231yzZT-rV_E_org@mail.gmail.com/
> I didn't repeat it in the cover letter because there is no change to the
> proposal.
>
> The series is based on 7.1-rc1. It is basically minimum viable patches.
> There are still a few hacks in this series and it may break something,
> for example, KPTI, SMT machines which shared TLB, etc. But it shoule be
> good enough for now to demonstrate the core idea. The main purpose of the
> RFC is to gather feedback, figure out missing parts and risks, and make sure
> we are on the right track, as well as hopefully it can help the discussion
> for the upcoming LSFMM.
>
> I broke the patches down to arch-dependent and arch-independent parts so that
> hopefully the interested persons can do experiments on other architectures,
> for example, S390, easier.
>
> A new kernel config is introduced, HAVE_LOCAL_PER_CPU_MAP. The architectures
> which can support this feature will select it. Allocating and freeing percpu
> local mapping is protected by this config so that others won't pay the cost.
I think this approach is a bit of a sledge hammer to crack a nut. If the problem
is the cost this_cpu_* ops on arm64 due to the need to disable/enable
preemption, I think we can solve that much more simply with [1], which
introduces in-kernel restartable sequences. Using this, the operation no longer
needs to disable preemption but can instead just detect when it is preempted and
retry.
[1]
https://lore.kernel.org/all/20260223163843.GR1282955@noisy.programming.kicks-ass.net/
We have observed a few performance regressions recently, for which the root
cause is increased use of this_cpu_*. We have somebody at Arm about to start an
investigation into whether in-kernel rseq can solve the problem.
>
>
> Known Issues
> ============
> 1. KPTI
> -------
> We need determine what CPU we are on, then switch to the right page table.
> Currently arm64 kernel fetches tramp_pg_dir via swapper_pg_dir - fixed_offset,
> and fetches swapper_pg_dir from ttbr1. But ttbr1 may not hold swapper_pg_dir
> anymore except CPU #0. So we need to figure out the other way to handle it.
> Switching to tramp_pg_dir should be easy, but the reverse seems harder because
> tramp_pg_dir just maps the trampoline vectors.
> Maybe we can do two steps switch. Switch to swapper_pg_dir at the first step,
> then switch to per cpu page table (for entry) or tramp page table (for exit).
> Nobody should call this_cpu_*() at either userspace -> kernel entry stage or
> kernel -> userspace exit stage.
>
> 2. SW PAN
> ---------
> Has the similar issue as KPTI. It installs reserved_pg_dir to TTBR0 when running
> in kernel space, but fetching reserved_pg_dir via swapper_pg_dir - fixed_offset.
> Maybe we can save the physical address of swapper_pg_dir in a variable, then load
> it from that variable instead of ttbr1.
>
> 3. Shared TLB machines
> ----------------------
> Some machines may share TLB between CPUs, for example, SMT machines may share
> TLB between the two hardware threads in one core.
> The per cpu page table just can't work with it. Maybe we need a new
> cpufeature to indicate whether per cpu page table is allowed or not. Then
> just enable it for not-shared-TLB machines.
The architectural feature you're referring to here is FEAT_TTCNP (common not
private). There are many CPUs out there that use this feature (not just those
that support SMT).
FEAT_TTCNP is mandatory from Armv8.2 and the presence of the feature only tells
you whether you can legally set the CnP bit - it doesn't actually tell you if
the HW does any sharing - there is no way to detect this. So making per-cpu
pgtables mutually exclusive with CNP is a non-starter as all CPUs from v8.2
onwards advertise CNP.
Even ignoring the CNP problem, you'll end up installing the per-cpu TLB entries
tagged with whatever user ASID happens to be installed at the time. Which means
you could end up with lots of duplicated TLB entries all differing only by ASID.
It's probably not the end of the world, but it seems... inefficient.
The only way I can see to make this work fully is to rely on FEAT_ASID2, which
allows specifying separate ASIDs for TTBR0 and TTBR1. Then each CPU can have
it's own (permanently installed) ASID for local mappings in TTBR1. That would
solve the CNP issue. FEAT_ASID2 is optional from v9.4 and mandatory from v9.5.
>
> 4. Don't support all VA bits
> ----------------------------
> We just support 48 bits VA (4K and 64K) and 47 bits VA (16K) for now. For 4K
> and 64K, supporting other VA bits is not hard, we just need to determine the
> size for percpu and local percpu area.
> But it is harder for supporting 48 bits VA + 16K page size. We just have two
> top level kernel page table entries with this configuration, but we assume we
> just need to sync up kernel page table at the top level for now. We need to
> sync up kernel page table at the second level in order to support it. I'm not
> sure whether it is worth it or not.
If your proposal is to only support per-cpu pgtables for certain VA
combinations, I don't think that will fly. We'd have too much of a test burden
for these different configs.
>
>
> Benchmark
> =========
> The benchmarks are done on 160 core AmpereOne machine. The baseline is
> v7.2-rc1 kernel.
>
> 1. Reduction of kernel text size
> --------------------------------
> The patchset can reduce at least 11 instructions for this_cpu_*() ops. Both
> preempt_disable() and preempt_enable() need 4 instructions to manipulate
> the preempt count, and preempt_enable() needs more instructions (compare +
> READ + compare) to determine whether reschedule is needed or not.
> Because this_cpu_*() ops are inlined and called in a lot of places so we
> can save a lot of instructions.
>
> The size of kernel text is reduced by ~184KB with default Fedora kernel
> config. This also helps reduce kernel icache miss rate and stalled frontend
> cycles as kernel build benchmark result showed.
>
> 2. Kernel Build
> ---------------
> Run kernel build (make -j160) with the default Fedora kernel config in a
> memcg.
> 13% - 18% sys time improvment
> 3% - 7% wall time improvement
>
> 5% fewer kernel icache miss, 5% fewer executed kernel instructions and
> 15% fewer stalled frontend cycles for kernel.
>
> 3. stress-ng vm ops
> -------------------
> stress-ng --vm 160 --vm-bytes 128M --vm-ops 100000000
> 8.5% improvement
>
> 4. stress-ng vm ops + fork
> --------------------------
> stress-ng --mmapfork 160 --mmapfork-bytes 128M --mmapfork-ops 500
> 15% improvement
>
> 5. Specjbb
> ----------
> The specjbb test latency curves showed the patched kernel has consistently
> lower p99 latency (the lower the better) than the baseline.
>
> 2.5% improvement on max-jOPS and 4% - 5% improvement on critical-jOPS.
> The specjbb benchmark is quite sensitive to latency and responsiveness,
> particularly critical-jOPS result. The patches are supposed to improve the
> responsiveness due to the reduction of preempt-disabled critical sections.
>
> 6. MySQL
> --------
> 1% - 2% gains on read-only test, 2% - 4% gains on write-only test. Also see
> 15% decrease on frontend cache stall.
These gains all look great but I expect we will see similar gains using the rseq
approach, which is simpler and can be applied universally.
Thanks,
Ryan
>
>
> Regression test
> ===============
> 1. memcg creation
> -----------------
> Create 10K memcgs. Each memcg creation needs to allocate multiple percpu
> variables, for example, percpu refcnt, rstat and objcg percpu refcnt.
>
> Consumed 2112K more virtual memory for percpu “local mapping” and a few
> more mega bytes consumed by per cpu page tables.
> No noticeable regression was found for elapsed time.
>
> 2. fork test
> ------------
> stress-ng --fork 160 --fork-ops 10000000
> fork() needs to allocate multiple percpu variables, for example, rss
> counters and mm_cid_cpu.
>
> Roughly 1% regression was found. However stress-ng fork test has quites
> small address space, the real life workloads typically have much larger
> address space and do more complicated works. The stress-ng mmapfork
> benchmark saw 15% improvement.
>
>
> The organization of patches
> ===========================
> The refactor and prepatory patches (patch 1 - patch 4)
> Percpu page table support patches (patch 5 - patch 8)
> Local percpu area support patches (patch 7 - patch 15)
> Use local percpu area for this_cpu ops (patch 16)
>
>
> Yang Shi (16):
> drivers: arch_numa: move percpu set up code to arch
> arm64: kconfig: make percpu related configs not depend on NUMA
> mm: pgalloc: introduce {pud|pmd}_populate_sync()
> vmalloc: pass in pgd pointer for vmap{__vunmap}_range_noflush()
> arm64: mm: enable percpu kernel page table
> arm64: mm: defined {pud|pmd}_populate_sync()
> arm64: mm: sync percpu page table for memory hotplug/unplug
> arm64: kasan: sync up kasan shadow area page table
> arm64: mm: define percpu virtual space area
> mm: percpu: prepare to use dedicated percpu area
> arm64: mm: map local percpu first chunk
> mm: percpu: set up first chunk and reserve chunk
> arm64: mm: introduce __per_cpu_local_off
> mm: percpu: allocate and free local percpu vm area
> arm64: kconfig: select HAVE_LOCAL_PER_CPU_MAP
> arm64: percpu: use local percpu for this_cpu_*() APIs
>
> arch/arm64/Kconfig | 12 +++++++---
> arch/arm64/include/asm/mmu.h | 5 ++++
> arch/arm64/include/asm/mmu_context.h | 9 +++++++-
> arch/arm64/include/asm/percpu.h | 37 ++++++++++++++++++++++++++++-
> arch/arm64/include/asm/pgalloc.h | 24 +++++++++++++++++++
> arch/arm64/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++---
> arch/arm64/kernel/setup.c | 3 +++
> arch/arm64/kernel/smp.c | 44 +++++++++++++++++++++++++++++++++++
> arch/arm64/mm/kasan_init.c | 47 +++++++++++++++++++++++--------------
> arch/arm64/mm/mmu.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------
> arch/arm64/mm/ptdump.c | 4 ++++
> arch/riscv/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++++++++
> drivers/base/arch_numa.c | 51 +---------------------------------------
> include/linux/mm.h | 11 +++++++++
> include/linux/percpu.h | 4 +++-
> include/linux/pgalloc.h | 13 +++++++++++
> include/linux/vmalloc.h | 3 +++
> mm/Kconfig | 9 ++++++++
> mm/internal.h | 5 +++-
> mm/kmsan/hooks.c | 14 +++++------
> mm/percpu-internal.h | 14 +++++++++++
> mm/percpu-vm.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> mm/percpu.c | 58 +++++++++++++++++++++++++++++++++++++---------
> mm/sparse-vmemmap.c | 4 ++--
> mm/vmalloc.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------
> 25 files changed, 712 insertions(+), 144 deletions(-)
>
>
> Thanks,
> Yang
>
On 7/16/26 6:23 AM, Ryan Roberts wrote:
> On 15/07/2026 19:04, Yang Shi wrote:
>> Hi,
>>
>> This is v2 RFC. In v2 a lot problems found out by Sashiko were fixed and more
>> feature gaps were closed (please see the below changelog for the details).
>> Although there are still some open issues, for example, it just can support
>> 48 bits VA (for 4K and 64K) and 47 bits VA (for 16K), KPTI support has not
>> been solved yet, etc, but I think the delta should be big enough and worth
>> a new RFC to gather comments in order to make sure I'm on the right track.
>>
>> Some more benchmarks were done, for example, some latency related benchmarks
>> that I mentioned at LSFMM because I thought responsiveness should be improved
>> due to the removal of preempt_disable. Collected more PMU counters as well.
>> Please refer to the benchmark section for more details.
> Hi Yang,
>
> When we spoke about this at LSFMM, I said I would take the series for a spin on
> other hardware to see if the performance gains generalise. I did make a start on
> that but the kernel wouldn't even boot and I never found time to debug. Sorry
> that I didn't let you know sooner.
No worries. There are some known issues, for example, KPTI doesn't work
yet so I need to disable CONFIG_UNMAP_KERNEL_AT_EL0 in order to make it
boot on Altra/AltraMax or any others which need KPTI, and 16K/64K page
sizes didn't work in v1, but they are fixed in this spin.
> But I think that there are other issues with this concept that likely block it
> from being accepted upstream; see below...
>
>> Look forward to comments.
>>
>>
>> Changelog
>> v2: * Added 3-level and 2-level page table support.
>> * Tested with 16K and 64K page size. But we just support 48 bits VA (4K
>> and 64K) and 47 bits VA with 16K for now. Please refer to the below
>> "known issue" section for the detail reason.
>> * Added support for memory hotplug.
>> * Added support for KASAN (generic).
>> * Treated percpu and local percpu area address as vmalloc address.
>> * Fixed build failure for x86.
>> * Fixed build failure for !CONFIG_NUMA.
>> * Added KASAN support for local percpu area.
>> * Some other misc bug fixes found out by Sashiko.
>> * More code refactor and cleanup.
>> * Regorganized the patches.
>> * More benchmarks, refer to benchmark section for more details.
>> * Rebased to v7.2-rc1.
>>
>>
>> Introduction
>> ============
>> This patch series implemented the LSFMM 2026 proposal for optimizing
>> this_cpu_*() ops on ARM64. For the details of the proposal, Please refer to:
>> https://lore.kernel.org/linux-mm/CAHbLzkpcN-T8MH6=W3jCxcFj1gVZp8fRqe231yzZT-rV_E_org@mail.gmail.com/
>> I didn't repeat it in the cover letter because there is no change to the
>> proposal.
>>
>> The series is based on 7.1-rc1. It is basically minimum viable patches.
>> There are still a few hacks in this series and it may break something,
>> for example, KPTI, SMT machines which shared TLB, etc. But it shoule be
>> good enough for now to demonstrate the core idea. The main purpose of the
>> RFC is to gather feedback, figure out missing parts and risks, and make sure
>> we are on the right track, as well as hopefully it can help the discussion
>> for the upcoming LSFMM.
>>
>> I broke the patches down to arch-dependent and arch-independent parts so that
>> hopefully the interested persons can do experiments on other architectures,
>> for example, S390, easier.
>>
>> A new kernel config is introduced, HAVE_LOCAL_PER_CPU_MAP. The architectures
>> which can support this feature will select it. Allocating and freeing percpu
>> local mapping is protected by this config so that others won't pay the cost.
> I think this approach is a bit of a sledge hammer to crack a nut. If the problem
> is the cost this_cpu_* ops on arm64 due to the need to disable/enable
> preemption, I think we can solve that much more simply with [1], which
> introduces in-kernel restartable sequences. Using this, the operation no longer
> needs to disable preemption but can instead just detect when it is preempted and
> retry.
>
> [1]
> https://lore.kernel.org/all/20260223163843.GR1282955@noisy.programming.kicks-ass.net/
>
> We have observed a few performance regressions recently, for which the root
> cause is increased use of this_cpu_*. We have somebody at Arm about to start an
> investigation into whether in-kernel rseq can solve the problem.
It is good to know someone is exploring the alternative path. But I
doubt restart sequence can really boost the performance. It needs check
whether preemption (or cpu migration) happens in the middle of percpu
operations then manipulate the ip to restart the operation. It sounds
like it just moved the cost from one place to the other place and it
also seems hacky TBH. It may reduce the preempt_disable/preempt_enable
cost somehow for some less contended cases, but it may be worse for high
contended cases. The percpu page table can work well for both.
S390 implemented restart sequence. Heiko didn't share too much benchmark
data, just confirmed there is no regression and I had a lengthy
discussion with him
(https://lore.kernel.org/lkml/20260520092243.264847-1-hca@linux.ibm.com/).
Improving this_cpu performance is just one of the usecase of percpu page
table. The other potential usecase is kernel text replication I
mentioned at LSFMM. The idea is not new, someone else has explored it,
see
https://lore.kernel.org/linux-arm-kernel/ZMKNYEkM7YnrDtOt@shell.armlinux.org.uk/.
Percpu page table can make kernel support it more naturally.
>>
>> Known Issues
>> ============
>> 1. KPTI
>> -------
>> We need determine what CPU we are on, then switch to the right page table.
>> Currently arm64 kernel fetches tramp_pg_dir via swapper_pg_dir - fixed_offset,
>> and fetches swapper_pg_dir from ttbr1. But ttbr1 may not hold swapper_pg_dir
>> anymore except CPU #0. So we need to figure out the other way to handle it.
>> Switching to tramp_pg_dir should be easy, but the reverse seems harder because
>> tramp_pg_dir just maps the trampoline vectors.
>> Maybe we can do two steps switch. Switch to swapper_pg_dir at the first step,
>> then switch to per cpu page table (for entry) or tramp page table (for exit).
>> Nobody should call this_cpu_*() at either userspace -> kernel entry stage or
>> kernel -> userspace exit stage.
>>
>> 2. SW PAN
>> ---------
>> Has the similar issue as KPTI. It installs reserved_pg_dir to TTBR0 when running
>> in kernel space, but fetching reserved_pg_dir via swapper_pg_dir - fixed_offset.
>> Maybe we can save the physical address of swapper_pg_dir in a variable, then load
>> it from that variable instead of ttbr1.
>>
>> 3. Shared TLB machines
>> ----------------------
>> Some machines may share TLB between CPUs, for example, SMT machines may share
>> TLB between the two hardware threads in one core.
>> The per cpu page table just can't work with it. Maybe we need a new
>> cpufeature to indicate whether per cpu page table is allowed or not. Then
>> just enable it for not-shared-TLB machines.
> The architectural feature you're referring to here is FEAT_TTCNP (common not
> private). There are many CPUs out there that use this feature (not just those
> that support SMT).
>
> FEAT_TTCNP is mandatory from Armv8.2 and the presence of the feature only tells
> you whether you can legally set the CnP bit - it doesn't actually tell you if
> the HW does any sharing - there is no way to detect this. So making per-cpu
> pgtables mutually exclusive with CNP is a non-starter as all CPUs from v8.2
> onwards advertise CNP.
>
> Even ignoring the CNP problem, you'll end up installing the per-cpu TLB entries
> tagged with whatever user ASID happens to be installed at the time. Which means
> you could end up with lots of duplicated TLB entries all differing only by ASID.
> It's probably not the end of the world, but it seems... inefficient.
Really? I thought ASID will be ignored if nG == 0.
> The only way I can see to make this work fully is to rely on FEAT_ASID2, which
> allows specifying separate ASIDs for TTBR0 and TTBR1. Then each CPU can have
> it's own (permanently installed) ASID for local mappings in TTBR1. That would
> solve the CNP issue. FEAT_ASID2 is optional from v9.4 and mandatory from v9.5.
I agree it is safer to handle CNP problem by depending on FEAT_ASID2.
>> 4. Don't support all VA bits
>> ----------------------------
>> We just support 48 bits VA (4K and 64K) and 47 bits VA (16K) for now. For 4K
>> and 64K, supporting other VA bits is not hard, we just need to determine the
>> size for percpu and local percpu area.
>> But it is harder for supporting 48 bits VA + 16K page size. We just have two
>> top level kernel page table entries with this configuration, but we assume we
>> just need to sync up kernel page table at the top level for now. We need to
>> sync up kernel page table at the second level in order to support it. I'm not
>> sure whether it is worth it or not.
> If your proposal is to only support per-cpu pgtables for certain VA
> combinations, I don't think that will fly. We'd have too much of a test burden
> for these different configs.
OK, I can try to add support for other VA combinations. That should
require sync'ing up percpu page tables at both the top level and the
second level.
Thanks,
Yang
>> Benchmark
>> =========
>> The benchmarks are done on 160 core AmpereOne machine. The baseline is
>> v7.2-rc1 kernel.
>>
>> 1. Reduction of kernel text size
>> --------------------------------
>> The patchset can reduce at least 11 instructions for this_cpu_*() ops. Both
>> preempt_disable() and preempt_enable() need 4 instructions to manipulate
>> the preempt count, and preempt_enable() needs more instructions (compare +
>> READ + compare) to determine whether reschedule is needed or not.
>> Because this_cpu_*() ops are inlined and called in a lot of places so we
>> can save a lot of instructions.
>>
>> The size of kernel text is reduced by ~184KB with default Fedora kernel
>> config. This also helps reduce kernel icache miss rate and stalled frontend
>> cycles as kernel build benchmark result showed.
>>
>> 2. Kernel Build
>> ---------------
>> Run kernel build (make -j160) with the default Fedora kernel config in a
>> memcg.
>> 13% - 18% sys time improvment
>> 3% - 7% wall time improvement
>>
>> 5% fewer kernel icache miss, 5% fewer executed kernel instructions and
>> 15% fewer stalled frontend cycles for kernel.
>>
>> 3. stress-ng vm ops
>> -------------------
>> stress-ng --vm 160 --vm-bytes 128M --vm-ops 100000000
>> 8.5% improvement
>>
>> 4. stress-ng vm ops + fork
>> --------------------------
>> stress-ng --mmapfork 160 --mmapfork-bytes 128M --mmapfork-ops 500
>> 15% improvement
>>
>> 5. Specjbb
>> ----------
>> The specjbb test latency curves showed the patched kernel has consistently
>> lower p99 latency (the lower the better) than the baseline.
>>
>> 2.5% improvement on max-jOPS and 4% - 5% improvement on critical-jOPS.
>> The specjbb benchmark is quite sensitive to latency and responsiveness,
>> particularly critical-jOPS result. The patches are supposed to improve the
>> responsiveness due to the reduction of preempt-disabled critical sections.
>>
>> 6. MySQL
>> --------
>> 1% - 2% gains on read-only test, 2% - 4% gains on write-only test. Also see
>> 15% decrease on frontend cache stall.
> These gains all look great but I expect we will see similar gains using the rseq
> approach, which is simpler and can be applied universally.
>
> Thanks,
> Ryan
>
>> Regression test
>> ===============
>> 1. memcg creation
>> -----------------
>> Create 10K memcgs. Each memcg creation needs to allocate multiple percpu
>> variables, for example, percpu refcnt, rstat and objcg percpu refcnt.
>>
>> Consumed 2112K more virtual memory for percpu “local mapping” and a few
>> more mega bytes consumed by per cpu page tables.
>> No noticeable regression was found for elapsed time.
>>
>> 2. fork test
>> ------------
>> stress-ng --fork 160 --fork-ops 10000000
>> fork() needs to allocate multiple percpu variables, for example, rss
>> counters and mm_cid_cpu.
>>
>> Roughly 1% regression was found. However stress-ng fork test has quites
>> small address space, the real life workloads typically have much larger
>> address space and do more complicated works. The stress-ng mmapfork
>> benchmark saw 15% improvement.
>>
>>
>> The organization of patches
>> ===========================
>> The refactor and prepatory patches (patch 1 - patch 4)
>> Percpu page table support patches (patch 5 - patch 8)
>> Local percpu area support patches (patch 7 - patch 15)
>> Use local percpu area for this_cpu ops (patch 16)
>>
>>
>> Yang Shi (16):
>> drivers: arch_numa: move percpu set up code to arch
>> arm64: kconfig: make percpu related configs not depend on NUMA
>> mm: pgalloc: introduce {pud|pmd}_populate_sync()
>> vmalloc: pass in pgd pointer for vmap{__vunmap}_range_noflush()
>> arm64: mm: enable percpu kernel page table
>> arm64: mm: defined {pud|pmd}_populate_sync()
>> arm64: mm: sync percpu page table for memory hotplug/unplug
>> arm64: kasan: sync up kasan shadow area page table
>> arm64: mm: define percpu virtual space area
>> mm: percpu: prepare to use dedicated percpu area
>> arm64: mm: map local percpu first chunk
>> mm: percpu: set up first chunk and reserve chunk
>> arm64: mm: introduce __per_cpu_local_off
>> mm: percpu: allocate and free local percpu vm area
>> arm64: kconfig: select HAVE_LOCAL_PER_CPU_MAP
>> arm64: percpu: use local percpu for this_cpu_*() APIs
>>
>> arch/arm64/Kconfig | 12 +++++++---
>> arch/arm64/include/asm/mmu.h | 5 ++++
>> arch/arm64/include/asm/mmu_context.h | 9 +++++++-
>> arch/arm64/include/asm/percpu.h | 37 ++++++++++++++++++++++++++++-
>> arch/arm64/include/asm/pgalloc.h | 24 +++++++++++++++++++
>> arch/arm64/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++---
>> arch/arm64/kernel/setup.c | 3 +++
>> arch/arm64/kernel/smp.c | 44 +++++++++++++++++++++++++++++++++++
>> arch/arm64/mm/kasan_init.c | 47 +++++++++++++++++++++++--------------
>> arch/arm64/mm/mmu.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------
>> arch/arm64/mm/ptdump.c | 4 ++++
>> arch/riscv/kernel/smp.c | 51 ++++++++++++++++++++++++++++++++++++++++
>> drivers/base/arch_numa.c | 51 +---------------------------------------
>> include/linux/mm.h | 11 +++++++++
>> include/linux/percpu.h | 4 +++-
>> include/linux/pgalloc.h | 13 +++++++++++
>> include/linux/vmalloc.h | 3 +++
>> mm/Kconfig | 9 ++++++++
>> mm/internal.h | 5 +++-
>> mm/kmsan/hooks.c | 14 +++++------
>> mm/percpu-internal.h | 14 +++++++++++
>> mm/percpu-vm.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> mm/percpu.c | 58 +++++++++++++++++++++++++++++++++++++---------
>> mm/sparse-vmemmap.c | 4 ++--
>> mm/vmalloc.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------
>> 25 files changed, 712 insertions(+), 144 deletions(-)
>>
>>
>> Thanks,
>> Yang
>>
On Tue, Jul 21, 2026 at 04:20:50PM -0700, Yang Shi wrote: > On 7/16/26 6:23 AM, Ryan Roberts wrote: > > On 15/07/2026 19:04, Yang Shi wrote: > > But I think that there are other issues with this concept that likely block it > > from being accepted upstream; see below... > > We have observed a few performance regressions recently, for which the root > > cause is increased use of this_cpu_*. We have somebody at Arm about to start an > > investigation into whether in-kernel rseq can solve the problem. > > It is good to know someone is exploring the alternative path. But I doubt > restart sequence can really boost the performance. It needs check whether > preemption (or cpu migration) happens in the middle of percpu operations > then manipulate the ip to restart the operation. With the rseq approach, that check happens out-of-line in exception handling code, *only* when preemption occurs, so in the fast path there is *no code whatsoever* to check for preemption/migration, and no branches to cause a restart. Please note that with the approach I've prototyped, the exception handling code doesn't even adjust the PC; we just adjust a GPR containing the address. The forward progress guarantees (and behaviour under contention) should be the same as a single instruction. The fixup logic itself is trivial. Please see my message where I outlined that approach in detail: https://lore.kernel.org/linux-arm-kernel/al_DpFJFcmVhxpvW@J2N7QTR9R3/ I expect that should come with a reasonable benefit, but I don't have benchmark figures yet as I haven't finished converting the xchg and cmpxchg implementations. > It sounds like it just moved the cost from one place to the other > place and it also seems hacky TBH. The existing (high) cost is removed from the fast path. A new (low) cost is added to the rare path where we perform an exception return into the middle of a critical section. I don't think this is any more hacky than percpu page tables. It's far more contained to the relevant pieces of code (only the this_cpu_*() ops and exception entry/return). > It may reduce the preempt_disable/preempt_enable cost somehow for some > less contended cases, but it may be worse for high contended cases. > The percpu page table can work well for both. It entirely removes the preempt_disable/preempt_enable cost. I don't follow your concern with contention. Any contention on the memory location is going to be independent of preemption, and if a thread is preempted mid-operation, the impact is going to be the same with either the adress fixup or percpu page tables -- I don't think you avoid that. > S390 implemented restart sequence. Heiko didn't share too much benchmark > data, just confirmed there is no regression and I had a lengthy discussion > with him > (https://lore.kernel.org/lkml/20260520092243.264847-1-hca@linux.ibm.com/). As above, for s390 Heiko didn't implement a restart sequence; he implemented an address fixup sequence. There is no restart. The discussion you had there seemed to focus on the s390 implementation details (e.g. why he used an address fixup rather than a restart), and the benefits of removing the preempt_count manipulation (which all the approaches, including mine, do). Am I missing some key detail? > Improving this_cpu performance is just one of the usecase of percpu page > table. The other potential usecase is kernel text replication I mentioned at > LSFMM. The idea is not new, someone else has explored it, see https://lore.kernel.org/linux-arm-kernel/ZMKNYEkM7YnrDtOt@shell.armlinux.org.uk/. That was looked at in the past, but people weren't keen given the complexity it would lead to, which is why it never saw much traction. > Percpu page table can make kernel support it more naturally. Perhaps, but the only difference is that you've moving the support code into the core kernel mm code. I think the general concerns people have with percpu page tables still apply. [...] > > > Known Issues > > > ============ > > > 3. Shared TLB machines > > > ---------------------- > > > Some machines may share TLB between CPUs, for example, SMT machines may share > > > TLB between the two hardware threads in one core. > > > The per cpu page table just can't work with it. Maybe we need a new > > > cpufeature to indicate whether per cpu page table is allowed or not. Then > > > just enable it for not-shared-TLB machines. > > The architectural feature you're referring to here is FEAT_TTCNP (common not > > private). There are many CPUs out there that use this feature (not just those > > that support SMT). > > > > FEAT_TTCNP is mandatory from Armv8.2 and the presence of the feature only tells > > you whether you can legally set the CnP bit - it doesn't actually tell you if > > the HW does any sharing - there is no way to detect this. So making per-cpu > > pgtables mutually exclusive with CNP is a non-starter as all CPUs from v8.2 > > onwards advertise CNP. > > > > Even ignoring the CNP problem, you'll end up installing the per-cpu TLB entries > > tagged with whatever user ASID happens to be installed at the time. Which means > > you could end up with lots of duplicated TLB entries all differing only by ASID. > > It's probably not the end of the world, but it seems... inefficient. > > Really? I thought ASID will be ignored if nG == 0. The nG bit only applies to leaf entries (Page or Block descriptors). All intermediate entries (Table descriptors) are ASID tagged. > > The only way I can see to make this work fully is to rely on FEAT_ASID2, which > > allows specifying separate ASIDs for TTBR0 and TTBR1. Then each CPU can have > > it's own (permanently installed) ASID for local mappings in TTBR1. That would > > solve the CNP issue. FEAT_ASID2 is optional from v9.4 and mandatory from v9.5. > > I agree it is safer to handle CNP problem by depending on FEAT_ASID2. I think that depending on FEAT_ASID2 is a non-starter. There is no extant hardware with FEAT_ASID2, and I don't think we want to maintain distinct versions of hte percpu code accross architecture versions. Mark.
On Thu, Jul 16, 2026 at 02:23:33PM +0100, Ryan Roberts wrote:
> We have observed a few performance regressions recently, for which the
> root cause is increased use of this_cpu_*. We have somebody at Arm
> about to start an investigation into whether in-kernel rseq can solve
> the problem.
FWIW, I had a look into something rseq like (resetting the PC upon
preemption or any exception), and I ended up with a GPR fixup scheme.
That avoids the need to disable/enable preemption, and only requires a
read of current and a couple of stores, which *should* be cheap.
The approach is similar to Peter Zijlstra's in-kernel rseq scheme [1],
and Heiko Carsten's approach for s390 [2]:
[1] https://lore.kernel.org/lkml/20260223163843.GR1282955@noisy.programming.kicks-ass.net/
[2] https://lore.kernel.org/lkml/20260526055702.1429061-1-hca@linux.ibm.com/
A key difference is that I ignore the PC entirely, and *only* track whether
GPRs are in use by a critical section. The GPR fixup is idempotent, and
can safely be applied anywhere in the critical section. Ignoring the PC
means that (in theory at least) this should work with kprobes, etc.
I've rebased and cleaned that up a bit, and pushed a WIP version to my
arm64/percpu-fixup branch:
git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git
I'll aim to have a more complete/polished version out in the near
future.
The key idea is that every percpu op has a critical section during which
it maintains three distinct GPRs:
<pcp> : The original __percpu pointer.
<off> : The percpu offset.
<addr>: The final pointer (<pcp> + <off>).
Whenever the kernel performs an exception return back into a critical
section, it can fix up <off> and <addr>, as described below. The
critical section is roughly as follows:
/*
* Prologue.
*
* At the start, <pcp> is already set, but <off> and <addr> are
* uninitialised.
*
* After this STR, upon an exception return into this critical
* section, the kernel will:
* 1. Update <off> to the current CPU's offset.
* 2. Update <addr> to be <pcp> + <off>.
*/
mrs <tsk>, sp_el0
mov <tmp>, ENCODE_REGISTER_NUMBERS(<pcp>, <off>, <addr>)
strh <tmp>, [<tsk>, #TSK_TI_PCPU_GPRS]
/*
* Generate the address. Any preemption within this critical
* section will result in <off> and <addr> being updated to
* match the CPU that this is resumed upon.
*/
mrs <off>, TPIDR_ELx
add <addr>, <pcp>, <off>
/* The actual operation, e.g. read/write/cmpxchg */
DO_SOMETHING_WITH(<addr>)
/*
* Epilogue.
*
* After this STR, the kernel will no longer apply the fixups.
*/
strh wzr, [<tsk>, #TSK_TI_PCPU_GPRS]
To handle nesting, the exception code is updated to save/restore the
register numbers in pt_regs (and clearing the active value at entry) so
that this safely nests.
With that, a simple test case such as:
u64 outline_this_cpu_read_u64(u64 __percpu *p)
{
return this_cpu_read(*p);
}
... is reduced from:
<outline_this_cpu_read_u64>:
paciasp
stp x29, x30, [sp, #-32]!
mrs x1, sp_el0
mov x29, sp
ldr w2, [x1, #8]
add w2, w2, #0x1
str w2, [x1, #8]
mrs x2, tpidr_el1
ldr x0, [x0, x2]
ldr x2, [x1, #8]
sub x2, x2, #0x1
str w2, [x1, #8]
cbz x2, 1f
ldr x1, [x1, #8]
cbnz x1, 2f
1:
str x0, [sp, #24]
bl 0 <preempt_schedule_notrace>
ldr x0, [sp, #24]
2:
ldp x29, x30, [sp], #32
autiasp
ret
... down to:
<outline_this_cpu_read_u64>:
mrs x2, sp_el0 // Prologue
mov x4, #0xc80 // Prologue
strh w4, [x2, #20] // Prologue
mrs x4, tpidr_el1
add x3, x0, x4
ldr x1, [x3]
strh wzr, [x2, #20] // Epilogue
mov x0, x1
ret
... which is clearly much better.
As noted above, the branch is a WIP. There are a bunch of things to do
(in particular, factoring out the xchg and cmpxchg assembly), but I
don't currently see a major technical blocker for the fixup approach.
Mark.
© 2016 - 2026 Red Hat, Inc.