When rodata=full kernel linear mapping is mapped by PTE due to arm's
break-before-make rule.

This resulted in a couple of problems:
  - performance degradation
  - more TLB pressure
  - memory waste for kernel page table

There are some workarounds to mitigate the problems, for example, using
rodata=on, but this compromises the security measurement.

With FEAT_BBM level 2 support, splitting large block page table to
smaller ones doesn't need to make the page table entry invalid anymore.
This allows kernel split large block mapping on the fly.

Add kernel page table split support and use large block mapping by
default when FEAT_BBM level 2 is supported for rodata=full.  When
changing permissions for kernel linear mapping, the page table will be
split to PTE level.

The machine without FEAT_BBM level 2 will fallback to have kernel linear
mapping PTE-mapped when rodata=full.

With this we saw significant performance boost with some benchmarks with
keeping rodata=full security protection in the mean time.

The test was done on AmpereOne machine (192 cores, 1P) with 256GB memory and
4K page size + 48 bit VA.

Function test (4K/16K/64K page size)
  - Kernel boot.  Kernel needs change kernel linear mapping permission at
    boot stage, if the patch didn't work, kernel typically didn't boot.
  - Module stress from stress-ng. Kernel module load change permission for
    module sections.
  - A test kernel module which allocates 80% of total memory via vmalloc(),
    then change the vmalloc area permission to RO, then change it back
    before vfree(). Then launch a VM which consumes almost all physical
    memory.
  - VM with the patchset applied in guest kernel too.
  - Kernel build in VM with patched guest kernel.

Memory consumption
Before:
MemTotal:       258988984 kB
MemFree:        254821700 kB

After:
MemTotal:       259505132 kB
MemFree:        255410264 kB

Around 500MB more memory are free to use.  The larger the machine, the
more memory saved.

Performance benchmarking
* Memcached
We saw performance degradation when running Memcached benchmark with
rodata=full vs rodata=on.  Our profiling pointed to kernel TLB pressure.
With this patchset we saw ops/sec is increased by around 3.5%, P99
latency is reduced by around 9.6%.
The gain mainly came from reduced kernel TLB misses.  The kernel TLB
MPKI is reduced by 28.5%.

The benchmark data is now on par with rodata=on too.

* Disk encryption (dm-crypt) benchmark
Ran fio benchmark with the below command on a 128G ramdisk (ext4) with disk
encryption (by dm-crypt).
fio --directory=/data --random_generator=lfsr --norandommap --randrepeat 1 \
    --status-interval=999 --rw=write --bs=4k --loops=1 --ioengine=sync \
    --iodepth=1 --numjobs=1 --fsync_on_close=1 --group_reporting --thread \
    --name=iops-test-job --eta-newline=1 --size 100G

The IOPS is increased by 90% - 150% (the variance is high, but the worst
number of good case is around 90% more than the best number of bad case).
The bandwidth is increased and the avg clat is reduced proportionally.

* Sequential file read
Read 100G file sequentially on XFS (xfs_io read with page cache populated).
The bandwidth is increased by 150%.


Yang Shi (3):
      arm64: cpufeature: detect FEAT_BBM level 2
      arm64: mm: support large block mapping when rodata=full
      arm64: cpufeature: workaround AmpereOne FEAT_BBM level 2

 arch/arm64/include/asm/cpufeature.h |  24 ++++++++++++++++++
 arch/arm64/include/asm/pgtable.h    |   7 +++++-
 arch/arm64/kernel/cpufeature.c      |  11 ++++++++
 arch/arm64/mm/mmu.c                 |  31 +++++++++++++++++++++--
 arch/arm64/mm/pageattr.c            | 173 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 arch/arm64/tools/cpucaps            |   1 +
 6 files changed, 238 insertions(+), 9 deletions(-)

On Mon, Nov 18, 2024 at 10:16:07AM -0800, Yang Shi wrote:
> 
> When rodata=full kernel linear mapping is mapped by PTE due to arm's
> break-before-make rule.
> 
> This resulted in a couple of problems:
>   - performance degradation
>   - more TLB pressure
>   - memory waste for kernel page table
> 
> There are some workarounds to mitigate the problems, for example, using
> rodata=on, but this compromises the security measurement.
> 
> With FEAT_BBM level 2 support, splitting large block page table to
> smaller ones doesn't need to make the page table entry invalid anymore.
> This allows kernel split large block mapping on the fly.

I think you can still get TLB conflict aborts in this case, so this
doesn't work. Hopefully the architecture can strengthen this in the
future to give you what you need.

Will

On Tue, 10 Dec 2024 11:31:52 +0000
Will Deacon <will@kernel.org> wrote:

> On Mon, Nov 18, 2024 at 10:16:07AM -0800, Yang Shi wrote:
> > 
> > When rodata=full kernel linear mapping is mapped by PTE due to arm's
> > break-before-make rule.
> > 
> > This resulted in a couple of problems:
> >   - performance degradation
> >   - more TLB pressure
> >   - memory waste for kernel page table
> > 
> > There are some workarounds to mitigate the problems, for example, using
> > rodata=on, but this compromises the security measurement.
> > 
> > With FEAT_BBM level 2 support, splitting large block page table to
> > smaller ones doesn't need to make the page table entry invalid anymore.
> > This allows kernel split large block mapping on the fly.  
> 
> I think you can still get TLB conflict aborts in this case, so this
> doesn't work. Hopefully the architecture can strengthen this in the
> future to give you what you need.
> 
> Will
> 

Hi All,

Given we have two threads on this topic, replying here as well...

Huawei has implementations that support BBML2, and might report TLB conflict
abort after changing block size directly until an appropriate TLB invalidation
instruction completes and this Implementation Choice is architecturally compliant.

I'm not trying to restrict potential solutions, but just making the point that we
will be interested in solutions that handle the conflict abort.

Jonathan

On Tue, 10 Dec 2024, Will Deacon wrote:

> > With FEAT_BBM level 2 support, splitting large block page table to
> > smaller ones doesn't need to make the page table entry invalid anymore.
> > This allows kernel split large block mapping on the fly.
>
> I think you can still get TLB conflict aborts in this case, so this
> doesn't work. Hopefully the architecture can strengthen this in the

Which platforms get TLB conflicts? Ours does not. Is this an errata on
some platforms?

On 12/10/24 3:31 AM, Will Deacon wrote:
> On Mon, Nov 18, 2024 at 10:16:07AM -0800, Yang Shi wrote:
>> When rodata=full kernel linear mapping is mapped by PTE due to arm's
>> break-before-make rule.
>>
>> This resulted in a couple of problems:
>>    - performance degradation
>>    - more TLB pressure
>>    - memory waste for kernel page table
>>
>> There are some workarounds to mitigate the problems, for example, using
>> rodata=on, but this compromises the security measurement.
>>
>> With FEAT_BBM level 2 support, splitting large block page table to
>> smaller ones doesn't need to make the page table entry invalid anymore.
>> This allows kernel split large block mapping on the fly.
> I think you can still get TLB conflict aborts in this case, so this
> doesn't work. Hopefully the architecture can strengthen this in the
> future to give you what you need.

Hi Will,

Thanks for responding. This is a little bit surprising. I thought 
FEAT_BBM level 2 can handle the TLB conflict gracefully. At least its 
description made me assume so. And Catalin also mentioned FEAT_BBM level 
2 can be used to split vmemmap page table in HVO patch discussion 
(https://lore.kernel.org/all/Zo68DP6siXfb6ZBR@arm.com/).

It sounds a little bit contradicting if the TLB conflict still can 
happen with FEAT_BBM level 2. It makes the benefit of FEAT_BBM level 2 
much less than expected.

Is it out of question to handle the TLB conflict aborts? IIUC we should 
just need flush TLB then resume, and it doesn't require to hold any 
locks as well.

And I chatted with our architects, I was told the TLB conflict abort 
doesn't happen on AmpereOne. Maybe this is why I didn't see the problem 
when I tested the patches.

Thanks,
Yang


>
> Will

Hey,

On Tue, Dec 10, 2024 at 11:33:16AM -0800, Yang Shi wrote:
> On 12/10/24 3:31 AM, Will Deacon wrote:
> > On Mon, Nov 18, 2024 at 10:16:07AM -0800, Yang Shi wrote:
> > > When rodata=full kernel linear mapping is mapped by PTE due to arm's
> > > break-before-make rule.
> > > 
> > > This resulted in a couple of problems:
> > >    - performance degradation
> > >    - more TLB pressure
> > >    - memory waste for kernel page table
> > > 
> > > There are some workarounds to mitigate the problems, for example, using
> > > rodata=on, but this compromises the security measurement.
> > > 
> > > With FEAT_BBM level 2 support, splitting large block page table to
> > > smaller ones doesn't need to make the page table entry invalid anymore.
> > > This allows kernel split large block mapping on the fly.
> > I think you can still get TLB conflict aborts in this case, so this
> > doesn't work. Hopefully the architecture can strengthen this in the
> > future to give you what you need.
> 
> Thanks for responding. This is a little bit surprising. I thought FEAT_BBM
> level 2 can handle the TLB conflict gracefully. At least its description
> made me assume so. And Catalin also mentioned FEAT_BBM level 2 can be used
> to split vmemmap page table in HVO patch discussion
> (https://lore.kernel.org/all/Zo68DP6siXfb6ZBR@arm.com/).
> 
> It sounds a little bit contradicting if the TLB conflict still can happen
> with FEAT_BBM level 2. It makes the benefit of FEAT_BBM level 2 much less
> than expected.

You can read the Arm ARM just as badly as I can :)

 | I_HYQMB
 |
 | If any level is supported and the TLB entries are not invalidated after
 | the writes that modified the translation table entries are completed,
 | then a TLB conflict abort can be generated because in a TLB there might
 | be multiple translation table entries that all translate the same IA.

Note *any level*.

Furthermore:

 | R_FWRMB
 |
 | If all of the following apply, then a TLB conflict abort is reported
 | to EL2:
 | * Level 1 or level 2 is supported.
 | * Stage 2 translations are enabled in the current translation regime.
 | * A TLB conflict abort is generated due to changing the block size or
 |   Contiguous bit.

I think this series is trying to handle some of this:

https://lore.kernel.org/r/20241211154611.40395-1-miko.lenczewski@arm.com

> Is it out of question to handle the TLB conflict aborts? IIUC we should just
> need flush TLB then resume, and it doesn't require to hold any locks as
> well.

See my reply here:

https://lore.kernel.org/r/20241211210243.GA17155@willie-the-truck

> And I chatted with our architects, I was told the TLB conflict abort doesn't
> happen on AmpereOne. Maybe this is why I didn't see the problem when I
> tested the patches.

I'm actually open to having an MIDR-based lookup for this if its your own
micro-architecture.

Will

On 12/11/24 2:30 PM, Will Deacon wrote:
> Hey,
>
> On Tue, Dec 10, 2024 at 11:33:16AM -0800, Yang Shi wrote:
>> On 12/10/24 3:31 AM, Will Deacon wrote:
>>> On Mon, Nov 18, 2024 at 10:16:07AM -0800, Yang Shi wrote:
>>>> When rodata=full kernel linear mapping is mapped by PTE due to arm's
>>>> break-before-make rule.
>>>>
>>>> This resulted in a couple of problems:
>>>>     - performance degradation
>>>>     - more TLB pressure
>>>>     - memory waste for kernel page table
>>>>
>>>> There are some workarounds to mitigate the problems, for example, using
>>>> rodata=on, but this compromises the security measurement.
>>>>
>>>> With FEAT_BBM level 2 support, splitting large block page table to
>>>> smaller ones doesn't need to make the page table entry invalid anymore.
>>>> This allows kernel split large block mapping on the fly.
>>> I think you can still get TLB conflict aborts in this case, so this
>>> doesn't work. Hopefully the architecture can strengthen this in the
>>> future to give you what you need.
>> Thanks for responding. This is a little bit surprising. I thought FEAT_BBM
>> level 2 can handle the TLB conflict gracefully. At least its description
>> made me assume so. And Catalin also mentioned FEAT_BBM level 2 can be used
>> to split vmemmap page table in HVO patch discussion
>> (https://lore.kernel.org/all/Zo68DP6siXfb6ZBR@arm.com/).
>>
>> It sounds a little bit contradicting if the TLB conflict still can happen
>> with FEAT_BBM level 2. It makes the benefit of FEAT_BBM level 2 much less
>> than expected.
> You can read the Arm ARM just as badly as I can :)
>
>   | I_HYQMB
>   |
>   | If any level is supported and the TLB entries are not invalidated after
>   | the writes that modified the translation table entries are completed,
>   | then a TLB conflict abort can be generated because in a TLB there might
>   | be multiple translation table entries that all translate the same IA.
>
> Note *any level*.
>
> Furthermore:
>
>   | R_FWRMB
>   |
>   | If all of the following apply, then a TLB conflict abort is reported
>   | to EL2:
>   | * Level 1 or level 2 is supported.
>   | * Stage 2 translations are enabled in the current translation regime.
>   | * A TLB conflict abort is generated due to changing the block size or
>   |   Contiguous bit.

Thank you so much for pinpointing the document.

>
> I think this series is trying to handle some of this:
>
> https://lore.kernel.org/r/20241211154611.40395-1-miko.lenczewski@arm.com

Thanks for sharing the series. It is new. Yes, both are trying to add 
BBMlv2 support to optimize some usecases.

>
>> Is it out of question to handle the TLB conflict aborts? IIUC we should just
>> need flush TLB then resume, and it doesn't require to hold any locks as
>> well.
> See my reply here:
>
> https://lore.kernel.org/r/20241211210243.GA17155@willie-the-truck

Yeah, it is hard to guarantee recursive TLB conflict abort never happens.

>
>> And I chatted with our architects, I was told the TLB conflict abort doesn't
>> happen on AmpereOne. Maybe this is why I didn't see the problem when I
>> tested the patches.
> I'm actually open to having an MIDR-based lookup for this if its your own
> micro-architecture.

I think it actually makes our life easier. We can just enable BBMlv2 for 
the CPUs which can handle TLB conflict gracefully, so we don't worry 
about handling TLB conflict abort at all. I can implement this in v2.

> Will

Gently ping...


Any comments on this RFC? Look forward to discussing them.

Thanks,
Yang


On 11/18/24 10:16 AM, Yang Shi wrote:
> When rodata=full kernel linear mapping is mapped by PTE due to arm's
> break-before-make rule.
>
> This resulted in a couple of problems:
>    - performance degradation
>    - more TLB pressure
>    - memory waste for kernel page table
>
> There are some workarounds to mitigate the problems, for example, using
> rodata=on, but this compromises the security measurement.
>
> With FEAT_BBM level 2 support, splitting large block page table to
> smaller ones doesn't need to make the page table entry invalid anymore.
> This allows kernel split large block mapping on the fly.
>
> Add kernel page table split support and use large block mapping by
> default when FEAT_BBM level 2 is supported for rodata=full.  When
> changing permissions for kernel linear mapping, the page table will be
> split to PTE level.
>
> The machine without FEAT_BBM level 2 will fallback to have kernel linear
> mapping PTE-mapped when rodata=full.
>
> With this we saw significant performance boost with some benchmarks with
> keeping rodata=full security protection in the mean time.
>
> The test was done on AmpereOne machine (192 cores, 1P) with 256GB memory and
> 4K page size + 48 bit VA.
>
> Function test (4K/16K/64K page size)
>    - Kernel boot.  Kernel needs change kernel linear mapping permission at
>      boot stage, if the patch didn't work, kernel typically didn't boot.
>    - Module stress from stress-ng. Kernel module load change permission for
>      module sections.
>    - A test kernel module which allocates 80% of total memory via vmalloc(),
>      then change the vmalloc area permission to RO, then change it back
>      before vfree(). Then launch a VM which consumes almost all physical
>      memory.
>    - VM with the patchset applied in guest kernel too.
>    - Kernel build in VM with patched guest kernel.
>
> Memory consumption
> Before:
> MemTotal:       258988984 kB
> MemFree:        254821700 kB
>
> After:
> MemTotal:       259505132 kB
> MemFree:        255410264 kB
>
> Around 500MB more memory are free to use.  The larger the machine, the
> more memory saved.
>
> Performance benchmarking
> * Memcached
> We saw performance degradation when running Memcached benchmark with
> rodata=full vs rodata=on.  Our profiling pointed to kernel TLB pressure.
> With this patchset we saw ops/sec is increased by around 3.5%, P99
> latency is reduced by around 9.6%.
> The gain mainly came from reduced kernel TLB misses.  The kernel TLB
> MPKI is reduced by 28.5%.
>
> The benchmark data is now on par with rodata=on too.
>
> * Disk encryption (dm-crypt) benchmark
> Ran fio benchmark with the below command on a 128G ramdisk (ext4) with disk
> encryption (by dm-crypt).
> fio --directory=/data --random_generator=lfsr --norandommap --randrepeat 1 \
>      --status-interval=999 --rw=write --bs=4k --loops=1 --ioengine=sync \
>      --iodepth=1 --numjobs=1 --fsync_on_close=1 --group_reporting --thread \
>      --name=iops-test-job --eta-newline=1 --size 100G
>
> The IOPS is increased by 90% - 150% (the variance is high, but the worst
> number of good case is around 90% more than the best number of bad case).
> The bandwidth is increased and the avg clat is reduced proportionally.
>
> * Sequential file read
> Read 100G file sequentially on XFS (xfs_io read with page cache populated).
> The bandwidth is increased by 150%.
>
>
> Yang Shi (3):
>        arm64: cpufeature: detect FEAT_BBM level 2
>        arm64: mm: support large block mapping when rodata=full
>        arm64: cpufeature: workaround AmpereOne FEAT_BBM level 2
>
>   arch/arm64/include/asm/cpufeature.h |  24 ++++++++++++++++++
>   arch/arm64/include/asm/pgtable.h    |   7 +++++-
>   arch/arm64/kernel/cpufeature.c      |  11 ++++++++
>   arch/arm64/mm/mmu.c                 |  31 +++++++++++++++++++++--
>   arch/arm64/mm/pageattr.c            | 173 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
>   arch/arm64/tools/cpucaps            |   1 +
>   6 files changed, 238 insertions(+), 9 deletions(-)
>
>