[PATCH v3 00/21] arm64: Move kernel mode FPSIMD buffer to the stack

Ard Biesheuvel posted 21 patches 4 months ago
There is a newer version of this series
arch/arm/include/asm/simd.h                  |   7 +
arch/arm64/crypto/aes-ce-ccm-glue.c          | 116 +++++------
arch/arm64/crypto/aes-ce-glue.c              |  87 ++++----
arch/arm64/crypto/aes-glue.c                 | 139 ++++++-------
arch/arm64/crypto/aes-neonbs-glue.c          | 150 +++++++-------
arch/arm64/crypto/ghash-ce-glue.c            |  27 ++-
arch/arm64/crypto/nhpoly1305-neon-glue.c     |   5 +-
arch/arm64/crypto/polyval-ce-glue.c          |  12 +-
arch/arm64/crypto/sha3-ce-glue.c             |  10 +-
arch/arm64/crypto/sm3-ce-glue.c              |  15 +-
arch/arm64/crypto/sm3-neon-glue.c            |  16 +-
arch/arm64/crypto/sm4-ce-ccm-glue.c          |  49 ++---
arch/arm64/crypto/sm4-ce-cipher-glue.c       |  10 +-
arch/arm64/crypto/sm4-ce-gcm-glue.c          |  62 ++----
arch/arm64/crypto/sm4-ce-glue.c              | 214 +++++++++-----------
arch/arm64/crypto/sm4-neon-glue.c            |  25 +--
arch/arm64/include/asm/fpu.h                 |  16 +-
arch/arm64/include/asm/neon.h                |   4 +-
arch/arm64/include/asm/processor.h           |   2 +-
arch/arm64/include/asm/simd.h                |  10 +
arch/arm64/include/asm/xor.h                 |  22 +-
arch/arm64/kernel/fpsimd.c                   |  44 ++--
crypto/aegis128-neon.c                       |  33 ++-
drivers/net/ethernet/mellanox/mlx5/core/wc.c |  19 +-
lib/crc/arm/crc-t10dif.h                     |  16 +-
lib/crc/arm/crc32.h                          |  11 +-
lib/crc/arm64/crc-t10dif.h                   |  16 +-
lib/crc/arm64/crc32.h                        |  16 +-
lib/crypto/arm/chacha.h                      |   6 +-
lib/crypto/arm/poly1305.h                    |   6 +-
lib/crypto/arm/sha1.h                        |  13 +-
lib/crypto/arm/sha256.h                      |  12 +-
lib/crypto/arm/sha512.h                      |   6 +-
lib/crypto/arm64/chacha.h                    |  11 +-
lib/crypto/arm64/poly1305.h                  |   6 +-
lib/crypto/arm64/sha1.h                      |   7 +-
lib/crypto/arm64/sha256.h                    |  19 +-
lib/crypto/arm64/sha512.h                    |   8 +-
lib/raid6/neon.c                             |  17 +-
lib/raid6/recov_neon.c                       |  15 +-
40 files changed, 587 insertions(+), 692 deletions(-)
[PATCH v3 00/21] arm64: Move kernel mode FPSIMD buffer to the stack
Posted by Ard Biesheuvel 4 months ago
From: Ard Biesheuvel <ardb@kernel.org>

Move the buffer for preserving/restoring the kernel mode FPSIMD state on a
context switch out of struct thread_struct, and onto the stack, so that
the memory cost is not imposed needlessly on all tasks in the system.

Changes since v2:
- Fix generic kernel mode FPU api instead of removing it.

- Rebase onto v6.18-rc0 and fix the fallout

- Prefer WARN() over BUG() in kernel_neon_begin/end

- Avoid unnecessary cmpxchg() calls

- When invoked in softirq context, use the caller provided buffer rather
  than the one stored in the task struct - this permits callers from
  task context (including users of the generic kernel mode FPU api) to
  pass NULL as the buffer when running with preemption disabled.

- Add acks from Kees and Eric; Mark's was dropped along with the patch
  in question.

- Fix new occurrence of kernel_neon_begin/end in Mellanox driver.

Changes since v1:
- Add a patch reverting the arm64 support for the generic
  kernel_fpu_begin()/end() API, which is problematic on arm64.

- Introduce a new 'ksimd' scoped guard that encapsulates the calls the
  kernel_neon_begin() and kernel_neon_end() at a higher level of
  abstraction. This makes it straight-forward to plumb in the stack
  buffer without complicating the callers.

- Move all kernel mode NEON users on arm64 (and some on ARM) over to the
  new API.

- Add Mark's ack to patches #6 - #8

Cc: Marc Zyngier <maz@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Mark Brown <broonie@kernel.org>
Cc: Eric Biggers <ebiggers@kernel.org>

Ard Biesheuvel (21):
  crypto/arm64: aes-ce-ccm - Avoid pointless yield of the NEON unit
  crypto/arm64: sm4-ce-ccm - Avoid pointless yield of the NEON unit
  crypto/arm64: sm4-ce-gcm - Avoid pointless yield of the NEON unit
  arm64/simd: Add scoped guard API for kernel mode SIMD
  ARM/simd: Add scoped guard API for kernel mode SIMD
  crypto: aegis128-neon - Move to more abstract 'ksimd' guard API
  raid6: Move to more abstract 'ksimd' guard API
  lib/crc: Switch ARM and arm64 to 'ksimd' scoped guard API
  lib/crypto: Switch ARM and arm64 to 'ksimd' scoped guard API
  crypto/arm64: aes-ccm - Switch to 'ksimd' scoped guard API
  crypto/arm64: aes-blk - Switch to 'ksimd' scoped guard API
  crypto/arm64: aes-gcm - Switch to 'ksimd' scoped guard API
  crypto/arm64: nhpoly1305 - Switch to 'ksimd' scoped guard API
  crypto/arm64: polyval - Switch to 'ksimd' scoped guard API
  crypto/arm64: sha3 - Switch to 'ksimd' scoped guard API
  crypto/arm64: sm3 - Switch to 'ksimd' scoped guard API
  crypto/arm64: sm4 - Switch to 'ksimd' scoped guard API
  arm64/xorblocks:  Switch to 'ksimd' scoped guard API
  net/mlx5: Switch to more abstract scoped ksimd guard API on arm64
  arm64/fpu: Enforce task-context only for generic kernel mode FPU
  arm64/fpsimd: Allocate kernel mode FP/SIMD buffers on the stack

 arch/arm/include/asm/simd.h                  |   7 +
 arch/arm64/crypto/aes-ce-ccm-glue.c          | 116 +++++------
 arch/arm64/crypto/aes-ce-glue.c              |  87 ++++----
 arch/arm64/crypto/aes-glue.c                 | 139 ++++++-------
 arch/arm64/crypto/aes-neonbs-glue.c          | 150 +++++++-------
 arch/arm64/crypto/ghash-ce-glue.c            |  27 ++-
 arch/arm64/crypto/nhpoly1305-neon-glue.c     |   5 +-
 arch/arm64/crypto/polyval-ce-glue.c          |  12 +-
 arch/arm64/crypto/sha3-ce-glue.c             |  10 +-
 arch/arm64/crypto/sm3-ce-glue.c              |  15 +-
 arch/arm64/crypto/sm3-neon-glue.c            |  16 +-
 arch/arm64/crypto/sm4-ce-ccm-glue.c          |  49 ++---
 arch/arm64/crypto/sm4-ce-cipher-glue.c       |  10 +-
 arch/arm64/crypto/sm4-ce-gcm-glue.c          |  62 ++----
 arch/arm64/crypto/sm4-ce-glue.c              | 214 +++++++++-----------
 arch/arm64/crypto/sm4-neon-glue.c            |  25 +--
 arch/arm64/include/asm/fpu.h                 |  16 +-
 arch/arm64/include/asm/neon.h                |   4 +-
 arch/arm64/include/asm/processor.h           |   2 +-
 arch/arm64/include/asm/simd.h                |  10 +
 arch/arm64/include/asm/xor.h                 |  22 +-
 arch/arm64/kernel/fpsimd.c                   |  44 ++--
 crypto/aegis128-neon.c                       |  33 ++-
 drivers/net/ethernet/mellanox/mlx5/core/wc.c |  19 +-
 lib/crc/arm/crc-t10dif.h                     |  16 +-
 lib/crc/arm/crc32.h                          |  11 +-
 lib/crc/arm64/crc-t10dif.h                   |  16 +-
 lib/crc/arm64/crc32.h                        |  16 +-
 lib/crypto/arm/chacha.h                      |   6 +-
 lib/crypto/arm/poly1305.h                    |   6 +-
 lib/crypto/arm/sha1.h                        |  13 +-
 lib/crypto/arm/sha256.h                      |  12 +-
 lib/crypto/arm/sha512.h                      |   6 +-
 lib/crypto/arm64/chacha.h                    |  11 +-
 lib/crypto/arm64/poly1305.h                  |   6 +-
 lib/crypto/arm64/sha1.h                      |   7 +-
 lib/crypto/arm64/sha256.h                    |  19 +-
 lib/crypto/arm64/sha512.h                    |   8 +-
 lib/raid6/neon.c                             |  17 +-
 lib/raid6/recov_neon.c                       |  15 +-
 40 files changed, 587 insertions(+), 692 deletions(-)


base-commit: 0d97f2067c166eb495771fede9f7b73999c67f66
-- 
2.51.0.710.ga91ca5db03-goog
Re: [PATCH v3 00/21] arm64: Move kernel mode FPSIMD buffer to the stack
Posted by Eric Biggers 3 months, 3 weeks ago
On Wed, Oct 08, 2025 at 05:45:34PM +0200, Ard Biesheuvel wrote:
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> Move the buffer for preserving/restoring the kernel mode FPSIMD state on a
> context switch out of struct thread_struct, and onto the stack, so that
> the memory cost is not imposed needlessly on all tasks in the system.

This patchset needs the following fixup to build:

diff --git a/lib/crc/arm/crc-t10dif.h b/lib/crc/arm/crc-t10dif.h
index 7f02a767f675e..aaeeab0defb5d 100644
--- a/lib/crc/arm/crc-t10dif.h
+++ b/lib/crc/arm/crc-t10dif.h
@@ -17,11 +17,11 @@ asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u8 *buf, size_t len,
 				  u8 out[16]);
 
 static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length)
 {
 	if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE) {
-		if (static_branch_likely(&have_pmull) && likely(may_use_simd()) {
+		if (static_branch_likely(&have_pmull) && likely(may_use_simd())) {
 			scoped_ksimd()
 				return crc_t10dif_pmull64(crc, data, length);
 		} else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE &&
 			   static_branch_likely(&have_neon) &&
 			   likely(may_use_simd())) {
diff --git a/lib/crypto/arm/sha512.h b/lib/crypto/arm/sha512.h
index 8d3af965d2b34..ed9bd81d6d78d 100644
--- a/lib/crypto/arm/sha512.h
+++ b/lib/crypto/arm/sha512.h
@@ -16,10 +16,11 @@ asmlinkage void sha512_block_data_order_neon(struct sha512_block_state *state,
 
 static void sha512_blocks(struct sha512_block_state *state,
 			  const u8 *data, size_t nblocks)
 {
 	if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&
+	    static_branch_likely(&have_neon) && likely(may_use_simd())) {
 		scoped_ksimd()
 			sha512_block_data_order_neon(state, data, nblocks);
 	} else {
 		sha512_block_data_order(state, data, nblocks);
 	}
Re: [PATCH v3 00/21] arm64: Move kernel mode FPSIMD buffer to the stack
Posted by Ard Biesheuvel 3 months, 2 weeks ago
On Fri, 17 Oct 2025 at 19:08, Eric Biggers <ebiggers@kernel.org> wrote:
>
> On Wed, Oct 08, 2025 at 05:45:34PM +0200, Ard Biesheuvel wrote:
> > From: Ard Biesheuvel <ardb@kernel.org>
> >
> > Move the buffer for preserving/restoring the kernel mode FPSIMD state on a
> > context switch out of struct thread_struct, and onto the stack, so that
> > the memory cost is not imposed needlessly on all tasks in the system.
>
> This patchset needs the following fixup to build:
>

Indeed - thanks for the head's up, although the robots already
informed me as well.