So this is the result of me doing some profiling on my 128-core Altra
box.  I've sent out versions of this before, but they've all been fairly
ugly partial series.

This is the full cleaned-up series with patches split up to be logical,
and with fixes from some of the commentary from previous patches.

The first four patches are for the 'runtime constant' code, where I did
the initial implementation on x86-64 just because I was more comfy with
that, and the arm64 version of it came once I had the x86-64 side
working.

The horror that is __d_lookup_rcu() shows up a lot more on my Altra box
because of the relatively pitiful caches, but it's something that I've
wanted on x86-64 before.  The arm64 numbers just made me bite the
bullet on the whole runtime constant thing.

The last three patches are purely arm64-specific, and just fix up some
nasty code generation in the user access functions.  I just noticed that
I will need to implement 'user_access_save()' for KCSAN now that I do
the unsafe user access functions. 

Anyway, that 'user_access_save/restore()' issue only shows up with
KCSAN.  And it would be a no-op thanks to arm64 doing SMAP the right way
(pet peeve: arm64 did what I told the x86 designers to do originally,
but they claimed was too hard, so we ended up with that CLAC/STAC
instead)... 

Sadly that "no-op for KCSAN" would is except for the horrid
CONFIG_ARM64_SW_TTBR0_PAN case, which is why I'm not touching it.  I'm
hoping some hapless^Whelpful arm64 person is willing to tackle this (or
maybe make KCSAN and ARM64_SW_TTBR0_PAN incompatible in the Kconfig). 

Note: the final access_ok() change in 7/7 is a API relaxation and
cleanup, and as such much more worrisome than the other patches.  It's
_simpler_ than the other patches, but the others aren't intended to
really change behavior.  That one does. 

Linus Torvalds (7):
  vfs: dcache: move hashlen_hash() from callers into d_hash()
  add default dummy 'runtime constant' infrastructure
  x86: add 'runtime constant' support
  arm64: add 'runtime constant' support
  arm64: start using 'asm goto' for get_user() when available
  arm64: start using 'asm goto' for put_user() when available
  arm64: access_ok() optimization

 arch/arm64/include/asm/runtime-const.h |  75 ++++++++++
 arch/arm64/include/asm/uaccess.h       | 191 +++++++++++++++++--------
 arch/arm64/kernel/mte.c                |  12 +-
 arch/arm64/kernel/vmlinux.lds.S        |   3 +
 arch/x86/include/asm/runtime-const.h   |  61 ++++++++
 arch/x86/kernel/vmlinux.lds.S          |   3 +
 fs/dcache.c                            |  17 ++-
 include/asm-generic/Kbuild             |   1 +
 include/asm-generic/runtime-const.h    |  15 ++
 include/asm-generic/vmlinux.lds.h      |   8 ++
 10 files changed, 319 insertions(+), 67 deletions(-)
 create mode 100644 arch/arm64/include/asm/runtime-const.h
 create mode 100644 arch/x86/include/asm/runtime-const.h
 create mode 100644 include/asm-generic/runtime-const.h

-- 
2.45.1.209.gc6f12300df

On Mon, Jun 10, 2024 at 01:48:14PM -0700, Linus Torvalds wrote:
> The last three patches are purely arm64-specific, and just fix up some
> nasty code generation in the user access functions.  I just noticed that
> I will need to implement 'user_access_save()' for KCSAN now that I do
> the unsafe user access functions. 
> 
> Anyway, that 'user_access_save/restore()' issue only shows up with
> KCSAN.  And it would be a no-op thanks to arm64 doing SMAP the right way
> (pet peeve: arm64 did what I told the x86 designers to do originally,
> but they claimed was too hard, so we ended up with that CLAC/STAC
> instead)... 
> 
> Sadly that "no-op for KCSAN" would is except for the horrid
> CONFIG_ARM64_SW_TTBR0_PAN case, which is why I'm not touching it.  I'm
> hoping some hapless^Whelpful arm64 person is willing to tackle this (or
> maybe make KCSAN and ARM64_SW_TTBR0_PAN incompatible in the Kconfig). 

Given how badly things go when we get this wrong (e.g. TLB corruption), I'd
like to say "just mark it incompatible", applying to all instrumentation, not
just KCSAN.

Any new microarchitecture since ~2014 has real HW PAN, and IIUC it's really
only Cortex-{A53,A57,A72} that don't have it, so I think it'd be fair to say
don't use sanitizers with SW PAN on those CPUs.

Otherwise, I came up with the below (untested). It's a bit horrid because we
could have instrumentation in the middle of __uaccess_ttbr0_{enable,disable}(),
and so we aways need the ISB just in case, and TBH I'm not sure that we use
user_access_{save,restore}() in all the places we should.

Mark.

diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index 30e2f8fa87a4..83301400ec4c 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -92,6 +92,38 @@ static inline void __uaccess_ttbr0_enable(void)
        local_irq_restore(flags);
 }
 
+static inline unsigned long user_access_ttbr0_save(void)
+{
+       if (!system_uses_ttbr0_pan())
+               return 0;
+
+       /*
+        * If TTBR1 has an ASID other than the reserved ASID, then we have an
+        * active user TTBR0 or are part-way through enabling/disabling TTBR0
+        * access.
+        */
+       if (read_sysreg(ttbr1_el1) & TTBR_ASID_MASK) {
+               __uaccess_ttbr0_disable();
+               return 1;
+       }
+
+       /*
+        * Instrumentation could fire during __uaccess_ttbr0_disable() between
+        * the final write to TTBR1 and before the trailing ISB. We need an ISB
+        * to ensure that we don't continue to use the old ASID.
+        */
+       isb();
+       return 0;
+}
+
+static inline void user_access_ttbr0_restore(unsigned long enabled)
+{
+       if (!system_uses_ttbr0_pan() || !enabled)
+               return;
+
+       __uaccess_ttbr0_enable();
+}
+
 static inline bool uaccess_ttbr0_disable(void)
 {
        if (!system_uses_ttbr0_pan())
@@ -117,8 +149,20 @@ static inline bool uaccess_ttbr0_enable(void)
 {
        return false;
 }
+
+static inline unsigned long user_access_ttbr0_save(void)
+{
+       return 0;
+}
+
+static inline void user_access_ttbr0_restore(unsigned long)
+{
+}
 #endif
 
+#define user_access_save       user_access_ttbr0_save
+#define user_access_restore    user_access_ttbr0_restore
+
 static inline void __uaccess_disable_hw_pan(void)
 {
        asm(ALTERNATIVE("nop", SET_PSTATE_PAN(0), ARM64_HAS_PAN,

On Wed, 12 Jun 2024 at 11:41, Mark Rutland <mark.rutland@arm.com> wrote:
>
> Given how badly things go when we get this wrong (e.g. TLB corruption), I'd
> like to say "just mark it incompatible", applying to all instrumentation, not
> just KCSAN.

Ack. I'll start out with just KCSAN (since that's the actual technical
issue right now). But since the SW PAN support is hopefully not
something that we should worry about going forward, I wouldn't mind it
being de-emphasized.

It's not like PAN is something that should necessarily be everywhere.
The real advantage of SMAP on x86 (and then PAN on arm) is that it
catches wild kernel pointers. As long as the HW support is common
enough, people will find bugs on those platforms.

So I think the advantage of SW PAN was "it will find issues early
before HW PAN is widely available". It might be time to lay SW PAN
entirely to rest now.

I'll send out a new version of the arm64 patches with the KCSAN build
failure fixed (with the simple no-op save/restore functions by making
KCSAN and SW PAN mutually incompatible), and with the virtual address
fix you pointed out in the other email.

              Linus

On Wed, 12 Jun 2024 at 13:02, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I'll send out a new version of the arm64 patches with the KCSAN build
> failure fixed (with the simple no-op save/restore functions by making
> KCSAN and SW PAN mutually incompatible), and with the virtual address
> fix you pointed out in the other email.

Actually, unless somebody really wants to see the whole series again,
here's just the diff between the end result of the series.

The actual changes are done in the relevant commits (ie the "asm goto
for get_user()" one for the KCSAN issue, and the "arm64 runtime
constant" commit for the I$ fixup).

Holler if you want to see the full series again.

It might be worth noting that I initially made the arm64 KCSAN support
have a "depend on !ARM64_SW_TTBR0_PAN" condition, but decided that
it's actually better to do it the other way around, and make
ARM64_SW_TTBR0_PAN depend on KCSAN not being enabled.

That way we're basically more eagerly disabling the thing that should
go away, and we're also having the KCSAN code be checked for
allmodconfig builds.

But hey, it's a judgement call.

               Linus