0) Background:
   We found that AmpereOne benefits from aggressive prefetches when
   using 4K page size.

1) This patch:
    1.1) adds new WORKAROUND_AMPERE_AC03_PREFETCH capability.
    1.2) uses MIDR_AMPERE1 to filter the processor.
    1.3) uses alternative_if to alternative the code
         for AmpereOne.
    1.4) adds software prefetches for the specific loop.
    	 Also add a macro add_prefetch.

2) Test result:
    In hugetlb or tmpfs, We can get big seqential read performance improvement
    up to 1.3x ~ 1.4x.


Huang Shijie (4):
  extable: add __sort_main_extable
  arm64: alternative: handle the kernel exception table
  arm64: copy_template.S: add loop_for_copy_128_bytes macro
  arm64: add software prefetches for AmpereOne

 arch/arm64/Kconfig.platforms    |  7 +++
 arch/arm64/kernel/alternative.c | 18 +++++++
 arch/arm64/kernel/cpu_errata.c  |  9 ++++
 arch/arm64/lib/copy_template.S  | 87 +++++++++++++++++++++++----------
 arch/arm64/tools/cpucaps        |  1 +
 include/linux/extable.h         |  2 +
 kernel/extable.c                |  8 ++-
 7 files changed, 105 insertions(+), 27 deletions(-)

-- 
2.40.1

On Wed, Nov 22, 2023 at 10:29 AM Huang Shijie
<shijie@os.amperecomputing.com> wrote:

> 0) Background:
>    We found that AmpereOne benefits from aggressive prefetches when
>    using 4K page size.
>
> 1) This patch:
>     1.1) adds new WORKAROUND_AMPERE_AC03_PREFETCH capability.
>     1.2) uses MIDR_AMPERE1 to filter the processor.
>     1.3) uses alternative_if to alternative the code
>          for AmpereOne.
>     1.4) adds software prefetches for the specific loop.
>          Also add a macro add_prefetch.
>
> 2) Test result:
>     In hugetlb or tmpfs, We can get big seqential read performance improvement
>     up to 1.3x ~ 1.4x.

In June 2022 Fujitsu tried to add a similar feature for A64FX, here is
the essence
of my feedback from back then, it applies here as well:
https://lore.kernel.org/linux-arm-kernel/CACRpkdbPLFOoPdX4L6ABV8GKpC8cQGP3s2aN2AvRHEK49U9VMg@mail.gmail.com/#t

TL;DR: this is a hack, if you want to accelerate the memory hierarchy,
then work with the MM developers to figure out how to do that in a
structured and scientific way that will work with any prefetching hardware
on any CPU.

Yours,
Linus Walleij

On Wed, Nov 22, 2023 at 05:28:51PM +0800, Huang Shijie wrote:
> 0) Background:
>    We found that AmpereOne benefits from aggressive prefetches when
>    using 4K page size.

We tend to shy away from micro-architecture specific optimisations in
the arm64 kernel as they're pretty unmaintainable, hard to test properly,
generally lead to bloat and add additional obstacles to updating our
library routines.

Admittedly, we have something for Thunder-X1 in copy_page() (disguised
as ARM64_HAS_NO_HW_PREFETCH) but, frankly, that machine needed all the
help it could get and given where it is today I suspect we could drop
that code without any material consequences.

So I'd really prefer not to merge this; modern CPUs should do better at
copying data. It's copy_to_user(), not rocket science.

Will

On Wed, Nov 22, 2023 at 09:48:57AM +0000, Will Deacon wrote:
> On Wed, Nov 22, 2023 at 05:28:51PM +0800, Huang Shijie wrote:
> > 0) Background:
> >    We found that AmpereOne benefits from aggressive prefetches when
> >    using 4K page size.
> 
> We tend to shy away from micro-architecture specific optimisations in
> the arm64 kernel as they're pretty unmaintainable, hard to test properly,
> generally lead to bloat and add additional obstacles to updating our
> library routines.
> 
> Admittedly, we have something for Thunder-X1 in copy_page() (disguised
> as ARM64_HAS_NO_HW_PREFETCH) but, frankly, that machine needed all the
> help it could get and given where it is today I suspect we could drop
> that code without any material consequences.
> 
> So I'd really prefer not to merge this; modern CPUs should do better at
> copying data. It's copy_to_user(), not rocket science.

I agree, and I'd also like to drop ARM64_HAS_NO_HW_PREFETCH.

Mark.

On Wed, 22 Nov 2023 11:40:09 +0000,
Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Wed, Nov 22, 2023 at 09:48:57AM +0000, Will Deacon wrote:
> > On Wed, Nov 22, 2023 at 05:28:51PM +0800, Huang Shijie wrote:
> > > 0) Background:
> > >    We found that AmpereOne benefits from aggressive prefetches when
> > >    using 4K page size.
> > 
> > We tend to shy away from micro-architecture specific optimisations in
> > the arm64 kernel as they're pretty unmaintainable, hard to test properly,
> > generally lead to bloat and add additional obstacles to updating our
> > library routines.
> > 
> > Admittedly, we have something for Thunder-X1 in copy_page() (disguised
> > as ARM64_HAS_NO_HW_PREFETCH) but, frankly, that machine needed all the
> > help it could get and given where it is today I suspect we could drop
> > that code without any material consequences.
> > 
> > So I'd really prefer not to merge this; modern CPUs should do better at
> > copying data. It's copy_to_user(), not rocket science.
> 
> I agree, and I'd also like to drop ARM64_HAS_NO_HW_PREFETCH.

+1. Also, as the (most probably) sole user of this remarkable
implementation, I hacked -rc2 to drop ARM64_HAS_NO_HW_PREFETCH. The
result is that a kernel compilation job regressed by 0.4%, something
that I consider being pure noise.

If nobody beats me to it, I'll send the patch.

	M.

-- 
Without deviation from the norm, progress is not possible.