From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC3AD2EA493 for ; Wed, 1 Oct 2025 21:03:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352635; cv=none; b=XslYs0Lhrfl4GxvPTlEkRFG1H+uI0+uTkQasp+eE9+BOxafQfhGDoiVPDjLlwhEFoNlbzdgxKhJZATr6nIAGNh8nznejMw761foAaFtif4uYo4mTWNXen/wi40S9BTIBmb7iu2yCaDgZuTpd7xU5kUmcdAWwQ+8T1Gvt5hbpLAw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352635; c=relaxed/simple; bh=TaDdrCr4EA15LrvASYKCxfIrPDw/VW9LUU5yJ++wxko=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lA3E/77EUWjgfm0z/y2iUbJxTRbWhGsf94q1NPUAsAJk1Y3UfrIwwTptSb1eyeAyBirToj5yh+vNwcHfsl1W7qK1T1WwxREJoyYozKfmuqBMTxCVxXhy2weDEFi5zBDbx9bMQuemloKE/OQjkNNZYfVjswBo6MBR/EnZF5SBWo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vVKp2pHl; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vVKp2pHl" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e3a049abaso1239065e9.0 for ; Wed, 01 Oct 2025 14:03:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352632; x=1759957432; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Yv4MPsa7zu8trmjDFhojytb8BQb2fz3TAffp8KKlObQ=; b=vVKp2pHl6ta5RDTqtN3Yq40h2I9t40cDOI/2pYOHsE3XfnCgd0wQaPfdfydNcZ2JiR K1IzJrk0U9AK5mB2eVDM7HyKprvDgKb17NrRYdyzriQdcaV6lFIJIvj7Hh32bfaOHwwe hvQVByjUMjfXI0GNN2qVU3wvN+cPtlBnrrmqdBafyPnVmZ9LAIbMfX4NNdevwaP5ZcmY MIQ87D2jrzBs8jFi2c6tmVvl9c+UOVKYGhimdD8n4l+bkIK0e55WscIeRZd7gc6xgxOz wf7EwDqWMSukn5mPbpoIv5TujVm53HL1Gj/24J2afeP1QHXxedO0WVj8hw/q79XlS9mN orlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352632; x=1759957432; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Yv4MPsa7zu8trmjDFhojytb8BQb2fz3TAffp8KKlObQ=; b=UnMBBonebfcDkS9cguH0ZbQpW1kHJZs3tsNxWILc73ST6A2rJdiCnAs7W6cFhrMQrp wrY4sYyzkjh0VZmkIWarh4idsn6xOOvUpNcWCNLW5wBG3tt2TZhG2UDdNw0u16iIbizq mwBV5AA6GPaOo6j0nHlw6iO+/fwBXkvMxcQ6+CQRlkNbBMbzQcF4nEN+IPeyMVAYNSf7 IPgCamtX2FUyY+sFan1qxQADzxxOIJhIB9e+51hR3yv1LC517IAJ+ReNjQozKoz/C9RP HxCNcnGxE6uLhEWjwh/VViA3diY/Ewi8AYd7FMC/4xVkOACNlgc/yOgjzIZ2QQH7g68B 4gMg== X-Forwarded-Encrypted: i=1; AJvYcCWYqWspUsWmQUUHP+NgtUhgWfh72VGVuDDQmi2ht6acDcMrWTllVU7u7ze9aayK6k/E2FX5XV+uGbFAj+E=@vger.kernel.org X-Gm-Message-State: AOJu0Yxdhbwykj0y7AC1SNrIY4utMk6pYNUzFCX0iaFSrK1KZDSkVnPt NFfXO7FzUUbVh4+z8E6GqEWv7IkH1W54bfeud5aawoKfYaAfTJh//1f7GIDlTVpxv5ACDqrnSA= = X-Google-Smtp-Source: AGHT+IEwfMr+9GH23CoDtyxHMgiZV15eJsCtIOXalrQTUCN9ExcDshQ1x2ZZkxN+VVOM8u52+gt+ZxW9 X-Received: from wmbh5.prod.google.com ([2002:a05:600c:a105:b0:46e:1ae9:749a]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600d:18:b0:46e:45d3:82fd with SMTP id 5b1f17b1804b1-46e61285dbamr33313365e9.31.1759352632472; Wed, 01 Oct 2025 14:03:52 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:03 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5902; i=ardb@kernel.org; h=from:subject; bh=Gm8Jj53sW6sJUiVR2FBV0YZV5f74EFJ+TCyGKzfcUfk=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutPMZpVErpLgli+4+e5Y8tWK+SIAO00XutE9MOx2av X89zbrQUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYy5yMjw3MuvfD1cQwyggdO PNXa/nrzxaLfjyNWNjy+GbZ3lp7X878M/920D9z+/e98p4s9c8G78K/ijhvMvDoV117q+ZdjEVD nyQ0A X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-23-ardb+git@google.com> Subject: [PATCH v2 01/20] arm64: Revert support for generic kernel mode FPU From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers , stable@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel On arm64, generic kernel mode FPU support, as used by the AMD GPU driver, involves dropping the -mgeneral-regs-only compiler flag, as that flag makes the use of double and float C types impossible. However, dropping that flag allows the compiler to use FPU and SIMD registers in other ways too, and for this reason, arm64 only permits doing so in strictly controlled contexts, i.e., isolated compilation units that get called from inside a kernel_neon_begin() and kernel_neon_end() pair. The users of the generic kernel mode FPU API lack such strict checks, and this may result in userland FP/SIMD state to get corrupted, given that touching FP/SIMD registers outside of a kernel_neon_begin/end pair does not fault, but silently operates on the userland state without preserving it. So disable this feature for the time being. This reverts commits 71883ae35278 arm64: implement ARCH_HAS_KERNEL_FPU_SUPPORT 7177089525d9 arm64: crypto: use CC_FLAGS_FPU for NEON CFLAGS 4be073931cd8 lib/raid6: use CC_FLAGS_FPU for NEON CFLAGS Cc: # v6.12+ Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 1 - arch/arm64/Makefile | 9 +----- arch/arm64/include/asm/fpu.h | 15 --------- arch/arm64/lib/Makefile | 6 ++-- lib/raid6/Makefile | 33 ++++++++++++++------ 5 files changed, 28 insertions(+), 36 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b81ab5fbde57..abf70929f675 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -32,7 +32,6 @@ config ARM64 select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_KCOV - select ARCH_HAS_KERNEL_FPU_SUPPORT if KERNEL_MODE_NEON select ARCH_HAS_KEEPINITRD select ARCH_HAS_MEMBARRIER_SYNC_CORE select ARCH_HAS_MEM_ENCRYPT diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 73a10f65ce8b..82209cc52a5a 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -33,14 +33,7 @@ ifeq ($(CONFIG_BROKEN_GAS_INST),y) $(warning Detected assembler with broken .inst; disassembly will be unreli= able) endif =20 -# The GCC option -ffreestanding is required in order to compile code conta= ining -# ARM/NEON intrinsics in a non C99-compliant environment (such as the kern= el) -CC_FLAGS_FPU :=3D -ffreestanding -# Enable -CC_FLAGS_FPU +=3D -isystem $(shell $(CC) -print-file-name=3Dinclude) -CC_FLAGS_NO_FPU :=3D -mgeneral-regs-only - -KBUILD_CFLAGS +=3D $(CC_FLAGS_NO_FPU) \ +KBUILD_CFLAGS +=3D -mgeneral-regs-only \ $(compat_vdso) $(cc_has_k_constraint) KBUILD_CFLAGS +=3D $(call cc-disable-warning, psabi) KBUILD_AFLAGS +=3D $(compat_vdso) diff --git a/arch/arm64/include/asm/fpu.h b/arch/arm64/include/asm/fpu.h deleted file mode 100644 index 2ae50bdce59b..000000000000 --- a/arch/arm64/include/asm/fpu.h +++ /dev/null @@ -1,15 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* - * Copyright (C) 2023 SiFive - */ - -#ifndef __ASM_FPU_H -#define __ASM_FPU_H - -#include - -#define kernel_fpu_available() cpu_has_neon() -#define kernel_fpu_begin() kernel_neon_begin() -#define kernel_fpu_end() kernel_neon_end() - -#endif /* ! __ASM_FPU_H */ diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 633e5223d944..291b616ab511 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -7,8 +7,10 @@ lib-y :=3D clear_user.o delay.o copy_from_user.o \ =20 ifeq ($(CONFIG_KERNEL_MODE_NEON), y) obj-$(CONFIG_XOR_BLOCKS) +=3D xor-neon.o -CFLAGS_xor-neon.o +=3D $(CC_FLAGS_FPU) -CFLAGS_REMOVE_xor-neon.o +=3D $(CC_FLAGS_NO_FPU) +CFLAGS_REMOVE_xor-neon.o +=3D -mgeneral-regs-only +CFLAGS_xor-neon.o +=3D -ffreestanding +# Enable +CFLAGS_xor-neon.o +=3D -isystem $(shell $(CC) -print-file-name=3Dinclude) endif =20 lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) +=3D uaccess_flushcache.o diff --git a/lib/raid6/Makefile b/lib/raid6/Makefile index 5be0a4e60ab1..903e287c50c8 100644 --- a/lib/raid6/Makefile +++ b/lib/raid6/Makefile @@ -34,6 +34,25 @@ CFLAGS_REMOVE_vpermxor8.o +=3D -msoft-float endif endif =20 +# The GCC option -ffreestanding is required in order to compile code conta= ining +# ARM/NEON intrinsics in a non C99-compliant environment (such as the kern= el) +ifeq ($(CONFIG_KERNEL_MODE_NEON),y) +NEON_FLAGS :=3D -ffreestanding +# Enable +NEON_FLAGS +=3D -isystem $(shell $(CC) -print-file-name=3Dinclude) +ifeq ($(ARCH),arm) +NEON_FLAGS +=3D -march=3Darmv7-a -mfloat-abi=3Dsoftfp -mfpu=3Dneon +endif +CFLAGS_recov_neon_inner.o +=3D $(NEON_FLAGS) +ifeq ($(ARCH),arm64) +CFLAGS_REMOVE_recov_neon_inner.o +=3D -mgeneral-regs-only +CFLAGS_REMOVE_neon1.o +=3D -mgeneral-regs-only +CFLAGS_REMOVE_neon2.o +=3D -mgeneral-regs-only +CFLAGS_REMOVE_neon4.o +=3D -mgeneral-regs-only +CFLAGS_REMOVE_neon8.o +=3D -mgeneral-regs-only +endif +endif + quiet_cmd_unroll =3D UNROLL $@ cmd_unroll =3D $(AWK) -v N=3D$* -f $(src)/unroll.awk < $< > $@ =20 @@ -57,16 +76,10 @@ targets +=3D vpermxor1.c vpermxor2.c vpermxor4.c vpermx= or8.c $(obj)/vpermxor%.c: $(src)/vpermxor.uc $(src)/unroll.awk FORCE $(call if_changed,unroll) =20 -CFLAGS_neon1.o +=3D $(CC_FLAGS_FPU) -CFLAGS_neon2.o +=3D $(CC_FLAGS_FPU) -CFLAGS_neon4.o +=3D $(CC_FLAGS_FPU) -CFLAGS_neon8.o +=3D $(CC_FLAGS_FPU) -CFLAGS_recov_neon_inner.o +=3D $(CC_FLAGS_FPU) -CFLAGS_REMOVE_neon1.o +=3D $(CC_FLAGS_NO_FPU) -CFLAGS_REMOVE_neon2.o +=3D $(CC_FLAGS_NO_FPU) -CFLAGS_REMOVE_neon4.o +=3D $(CC_FLAGS_NO_FPU) -CFLAGS_REMOVE_neon8.o +=3D $(CC_FLAGS_NO_FPU) -CFLAGS_REMOVE_recov_neon_inner.o +=3D $(CC_FLAGS_NO_FPU) +CFLAGS_neon1.o +=3D $(NEON_FLAGS) +CFLAGS_neon2.o +=3D $(NEON_FLAGS) +CFLAGS_neon4.o +=3D $(NEON_FLAGS) +CFLAGS_neon8.o +=3D $(NEON_FLAGS) targets +=3D neon1.c neon2.c neon4.c neon8.c $(obj)/neon%.c: $(src)/neon.uc $(src)/unroll.awk FORCE $(call if_changed,unroll) --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17ABE30EF80 for ; Wed, 1 Oct 2025 21:03:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352636; cv=none; b=QRY0AqGU4oLVgCtJwMRme1z2uM1m3+GrIF8gb5itY3/+GeDw4mrlhdbBKIxa+imijDglf0lni3YgCxQ0y8WVSDQ16ymYh+m/UKzcAIsUeym61XIsNc4IxcdxFnpCDf+1bfix70/pLXJ/8XySL+wl3nWe6AN7V03MSZpZ4cm2lHw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352636; c=relaxed/simple; bh=CcSQX2utR81UxJpArQsfhwlVVaWfbrFwN0ei3ceAlxE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XoP4KwJ8k38KCCFkTU4awW7kriNZXelf528hpZ8x+RDe+1ZJ/Ufs0mdBwklpdKMmLW/MW5/XbM6uQtXKO5//g7FjjHzPHiMvh2CV9DAegM1F8PWZFId/izZFLv1cxjOXX1zyVnUzGrCLyfEhqRYzBtyX9atxnbmNUr+9I//j1Ao= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CuEOYH/p; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CuEOYH/p" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-afcb72a8816so31997166b.0 for ; Wed, 01 Oct 2025 14:03:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352633; x=1759957433; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UYBWVcMrL+XqSLPgxbyO6R+Pmodg0fLAvWpUbIMCypQ=; b=CuEOYH/p6U+1yrP2pJY7YE0I00HIXk9lxXF0ss5ginJ4/1VvcPk21DRUkQ40FQsVMf BW8cDvnkbvUioGf0pXs7NUPLIjqlUQsiprgXNOt5BclDCl7eGLoAm11CemuqRrlbQv/P OQDIbjdhe+pFk39addXymvJ+OQLZelObTzmXc1g2kf4aAD+8wsUKEzog+c11X8kS8GpP nif25nfnP3OixGI24GsvEYoxMFGvs8rCNQ+02cjzGUFygBmrYhK6HV+smPtOg+1thShf rMlE8weLt8Pemb+TTXh6pRdREOX8dOL4yz8r6gYeswfqEj25JNwaDnQar9n2Scq3SQJm pNZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352633; x=1759957433; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UYBWVcMrL+XqSLPgxbyO6R+Pmodg0fLAvWpUbIMCypQ=; b=otNW8w2uagUqxAqidiLjtQof77LkWCCDSfFkPuIrwJvnfoBfS1yG+TmuhvYYMASeVS 9h/a9hGKMyarIFbNHW5bLKu7wZVdAAdZBVVst5FyflT6eLYVGFgME8+5W+NwxlIXwexj elWI2xa0zTQWguplNp48Eeq/taFyPKcL9qXTofxyrEH6y+/xAtZg8O4UjyaIZL8Wedu1 sjIviTIR1pIk76XnNZXMi1B80nCO4tohvFZnAYXVPoL9U098CcBlLXSDdTW6tAKH1yRE WlLvpdNx1XYEsxGfMVtckyrPPW/t6QDIxdXnlmQtxTvDhc7Bjl1xchsmMc9MsxDmMQzX CcPQ== X-Forwarded-Encrypted: i=1; AJvYcCUJe/kM7vRzhPXzidG0/nmZtJ2RhTypwW8FbZk5cfJQF7yqAM63xW0BRfAwFI32YPp9ZUGJtHcY1+QlNNY=@vger.kernel.org X-Gm-Message-State: AOJu0Yz0RkEd37LANcKTlVbzrRJ5ocxP8234OpE/S94AMySzJy27AMkB 8slhnnDqLDzZs/lfJZeirQKVZuuKOjRIQyZh22YyO6qXQ35QKPcpcrHukSCk4692TP/riVg0nw= = X-Google-Smtp-Source: AGHT+IHMlqwMCs9Npi3jOeY+LiGowC4953WrKZ9yJK8uSZJrjaUNV6KEIrxdClpYRueyTEYTFhCkPyP5 X-Received: from ejbwq1.prod.google.com ([2002:a17:907:641:b0:b3f:5a00:1605]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:3f8a:b0:b04:3dc7:7d80 with SMTP id a640c23a62f3a-b46e585ad3bmr632727366b.18.1759352633431; Wed, 01 Oct 2025 14:03:53 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:04 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1430; i=ardb@kernel.org; h=from:subject; bh=zW/Ll9u+2fAT/Va1/PfNMYyKFzy2uIhE96GfLDmS5ew=; b=kA0DAAoWMG4JVi59LVwByyZiAGjdltPIlHC+BqQZgjT+Q/Oqs9tM4IdV3oG7ERBq2+2gSCD9a 4h1BAAWCgAdFiEEEJv97rnLkRp9Q5odMG4JVi59LVwFAmjdltMACgkQMG4JVi59LVyPtwD/fjev K07ta1hTMaGo/CMNU/2cOZDAmzhu/3WAashGtlsA/joiQcKT/3M+Pd4404SNOcFD27Upbf7uPzp TSe/KAyMB X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-24-ardb+git@google.com> Subject: [PATCH v2 02/20] arm64/simd: Add scoped guard API for kernel mode SIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Encapsulate kernel_neon_begin() and kernel_neon_end() using a 'ksimd' cleanup guard. This hides the prototype of those functions, allowing them to be changed for arm64 but not ARM, without breaking code that is shared between those architectures (RAID6, AEGIS-128) It probably makes sense to expose this API more widely across architectures, as it affords more flexibility to the arch code to plumb it in, while imposing more rigid rules regarding the start/end bookends appearing in matched pairs. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/simd.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h index 8e86c9e70e48..d9f83c478736 100644 --- a/arch/arm64/include/asm/simd.h +++ b/arch/arm64/include/asm/simd.h @@ -6,12 +6,15 @@ #ifndef __ASM_SIMD_H #define __ASM_SIMD_H =20 +#include #include #include #include #include #include =20 +#include + #ifdef CONFIG_KERNEL_MODE_NEON =20 /* @@ -40,4 +43,8 @@ static __must_check inline bool may_use_simd(void) { =20 #endif /* ! CONFIG_KERNEL_MODE_NEON */ =20 +DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) + +#define scoped_ksimd() scoped_guard(ksimd) + #endif --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D76F30F92D for ; Wed, 1 Oct 2025 21:03:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352637; cv=none; b=fftaKxU0C9O2HFGMVCM2uMDpek7+9aNrlWYOIEilseZWBkdKUhvjVilI4f4MCDI//kjSzby6+HVX0TazfpTQarUhxAHVUKbnj6kCZKToKgQMvT8UHRwHYZ+BtqHarDnxEfvsm+dnTWtrXZXBqp7zZy6cR9IT/YxqiH5CzBh3zco= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352637; c=relaxed/simple; bh=GXeASNEqmczwPiXD+TO0rYwiXLFqmUMdsycyCVMnj/0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZGIQvqqUbuoAXh6JCeyjSnJ8kPAqQBIafkNvET3F1ifPxbp9HzTwyp0/t39uSPDNXOXsMYvFTKlEZ8bQWm+F/vr43NB7SoE+9aa/3NMRg6F1g7YTELlUBfucIsGpVLrziaEo27ouV3QlTFVvpFnJl2XBcRfovS0tijpgbxOgm/g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2PjoT70L; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2PjoT70L" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e32eb4798so1081015e9.2 for ; Wed, 01 Oct 2025 14:03:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352635; x=1759957435; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1FhWp47bs3reBDPv5Eo8/YKbhWPmlo11aWHsEN7F2Pk=; b=2PjoT70LGH0JnD3OlTa6+NIZvy1+Z5je/qye5U2evkAqywldrBCg2kP1HyQ+gP7bhQ 93o3ZzHlcItKJkflVq3yOabrJAzJe92bQfZVdQyMQo6VDnbDduNFjdBT1riXObdNR69J 0hmvLxvhchOGvbrgy6NjxtzzMpKWw6GqRd/seIsV8gZZNfmJgzMaJ6uvqLVd6PrkxGaD Ea/0RWO7Poh5/hoPZJD/ve5Ugo8qNcWYyHwAKqLbxvYXSNfdXHo+uCUGaU8naoRxyBnw suFQI5qmmvIZRtpykZictYcBDFfKDt3EWjMo/qeInmXbk8bjOMdAsy6VvYYxE2jtS1Xz fxiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352635; x=1759957435; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1FhWp47bs3reBDPv5Eo8/YKbhWPmlo11aWHsEN7F2Pk=; b=PxhvmBKeIcfma6BmAwqxOJGVUDe8vs0EKalgX9UQuS60smt+++lqya+6AQjh6atpaR VkhIRusR3RGw6AqYLvfy6cJFo99F6j/wcETIfxA0uAlE5K3a/qKLPOMjUrItyNpXnL8Q wBy8mB7b0qXSCw+7AAC1Tyn+cHIoNu749ReOIkK5q+28yspLhGHMHH6P6gvpBbQKHoDU gn1ozbqAiv+zmg8YrxKTTXpbZaCzf5fR9fWOPMMpJpOb0v6DQuao2YeIKyqN+Ww2piQz ukiuolFelzFLN+DyXFcDECDLl8JSFINn8jgtexgeqeNjbZ+GCEFG9OskLFSTRGg/k3hi Nqqw== X-Forwarded-Encrypted: i=1; AJvYcCWEJHy/iUWql+rbBGibFBQrrBSXvKK+4fhoVn6UF2d02MqRv5jhA4lZNbcSR7KRkHv5CYvpGdU+8JiMY8A=@vger.kernel.org X-Gm-Message-State: AOJu0Yz4AQ+/uLD3/WcCl7cEJ8qAv6ffH19TW6TwZ6FEh8zfO6n778tB l4bLX2/917AcXlaD++mueDd/sgzuTc0PYTCgSINHCJ4KZrpuNA/odv/JrS52COie2tMJXxCUZw= = X-Google-Smtp-Source: AGHT+IFI3zKtjw3viIN28FavI/x8N6Vmi9ld+eUE+6cV0FsK7qbWev2ZSe6+tyHLFwzSdNJeSUW5izog X-Received: from wmpb25.prod.google.com ([2002:a05:600c:4a99:b0:46e:2f1b:4ceb]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:198b:b0:45f:29ed:2cff with SMTP id 5b1f17b1804b1-46e613dd3femr38614835e9.35.1759352634802; Wed, 01 Oct 2025 14:03:54 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:05 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=982; i=ardb@kernel.org; h=from:subject; bh=o8h5bq3OJmsWnm3lH4s7jaZZXCXhEjYAKa43aK/6n5k=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutOs8NmwfJZcsl/9bfHnC3QmpN39NV4vzcFjwbPfeO t3Zis5KHaUsDGJcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiMzIZGX7UiG1UelXjuaJP p+dWpvAZjY/Rm7RvvhI8tKggf5JM6BmG/1Wqi6y/rcu5rndMTeJ40MRrHqvufL95KeRz6py6z7d PufICAA== X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-25-ardb+git@google.com> Subject: [PATCH v2 03/20] ARM/simd: Add scoped guard API for kernel mode SIMD From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Implement the ksimd scoped guard API so that it can be used by code that supports both ARM and arm64. Signed-off-by: Ard Biesheuvel --- arch/arm/include/asm/simd.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm/include/asm/simd.h b/arch/arm/include/asm/simd.h index be08a8da046f..8549fa8b7253 100644 --- a/arch/arm/include/asm/simd.h +++ b/arch/arm/include/asm/simd.h @@ -2,14 +2,21 @@ #ifndef _ASM_SIMD_H #define _ASM_SIMD_H =20 +#include #include #include #include =20 +#include + static __must_check inline bool may_use_simd(void) { return IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && !in_hardirq() && !irqs_disabled(); } =20 +DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) + +#define scoped_ksimd() scoped_guard(ksimd) + #endif /* _ASM_SIMD_H */ --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C72A530FC06 for ; Wed, 1 Oct 2025 21:03:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352640; cv=none; b=RUTbrrV3LhDZ7oh/YvTtqx4DZPHhq6okA5KI7fjz1NSfzMyUdHzri9QqIYtVszLhqugfzU5IUGKn1ZjyjDhgBSDIR8PzaP59llaN0lPjbkILBgX17/RYGnkAd1xVe/C5Oa1iNb1LGv8heck0dCfiZBx5wGBE+XxyYElchIYwWr8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352640; c=relaxed/simple; bh=y61+7VCzNMGSMbWtXfByDo69lK0ON+BCKh3sIOEFzkE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HGt4baePm9N/ZAlUnlXtz8Ol7uoGlelS0H7yID3QyYBaxbe/SOecxSXSP88oQGg84xj9Zk909XFC/XQm0Yameo6/qJhWI7Mv3be9lIAPv6lOc6RcEz86RNWMMoEUEH6me93+5Ql7guhEWt0PxvXdWwvz4vC06i9bceGU2QfZHn4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=joWHD8d2; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="joWHD8d2" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-634c48a0ce7so303866a12.1 for ; Wed, 01 Oct 2025 14:03:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352636; x=1759957436; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=acDSbPr0ckzlJYMJhQI+twxWM3EGvwpk6VEzCJHCwrY=; b=joWHD8d2x1bi8PXS+1tN+EACbQ+ehuWEkUIm2UjhZdyvaSfWJyM9aHbWWZLpPmzKOI crWV5ykV6rIDb+TuxPIySwlUyO9ERZwuhuFhisrd3Y0rj87eW+LMs4roJks0qBTFWAnr 2nzrdjh8Oyg4eRHBPRTLmjiPfd99jO0B+GpBhQ7TcLNOivVxwnOYEihjteRYNDWtlwKY RuGb5GNjVNf8qECH6+QLpXzSNTrrr971qAmq/seTTOiY/Z+r76enGEr5xayrFKk+zkzj ys2BmS7b47dCx9bKIT9WMOPTlLa7KE0AB41BZM9Aum3sSBtUokAD4kAMNXzaeGhewLT4 rB1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352636; x=1759957436; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=acDSbPr0ckzlJYMJhQI+twxWM3EGvwpk6VEzCJHCwrY=; b=ikavJnVlmBIrtzmcKbau3mhup2pDFL1sWDCZw0ilvLNKptx3mcxLfx7cBDIzavgVtT ft8ppr1lWoogqhm50c7nK72830/ARpwY9Z7/xC05HghSo5IOjnwI3Ma8p7OzQS925LIz 0pdZF6mMaaVpknrzCGujHO2zlLDXzZdYy3mAo8SnVsrcpdkIwD7Q+KzhGTUlsDHQVBUZ lVUrLI000jwEHB7dGVo6woou5GuMPR01tXJa9boCd6Het5xSCHBUMiwkZBhkEOyn5TWV qQ1/47tZbq6AfUHAjgRkwo8yiti/RjQftKSUva987LmB3diOY6LXEj8YswTWmziAaHC+ Y9GA== X-Forwarded-Encrypted: i=1; AJvYcCXvA3T8mNUQFS6IJCROgG1+d4bTn2VjZR+oOyYHobvtE0WhVr8QxF0YcNFv3055h1U9AS84S4vVYlbYb+o=@vger.kernel.org X-Gm-Message-State: AOJu0YzXsEZvs6O/cvq87AZPqrAqiXbS7xXAddLc34ZVPnMEJlULCphK X+g96HTQz6ZaB6tM9N0inkG6C1REBA/VuL358KwCh7VgPKRhg9lVHn+N1h4CNA06Eg50B5RtYw= = X-Google-Smtp-Source: AGHT+IEnx5RXeiL9kmHFg/RRSgpRtVYIAVUq2zDXnriSjY4GRqOqm0pZKleynTYr8BnmTOEPCHcJAcWB X-Received: from edwn19.prod.google.com ([2002:a05:6402:4d3:b0:636:7d5e:f1f0]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:42c2:b0:634:b4cb:c892 with SMTP id 4fb4d7f45d1cf-63678c998cdmr5701808a12.32.1759352635997; Wed, 01 Oct 2025 14:03:55 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:06 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2459; i=ardb@kernel.org; h=from:subject; bh=cjwJWt4i7jKdiITTdNRuozrMElZ3Ow1rsFcjCp41v6I=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutDtRV62tdzTOWvv8H6N85apP8zU+pHE0XX+8dkfc+ XknPnrM6ShlYRDjYpAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQATMdZk+F87y/Ne5BfOHc8/ 3mZY2B+3Pdrzc9G9588Cp8dLvpx1WsObkWHnjJRdwZa9pbmFmr8/W8ztFJOJlbz35bApq+W96/U ajMwA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-26-ardb+git@google.com> Subject: [PATCH v2 04/20] crypto: aegis128-neon - Move to more abstract 'ksimd' guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Move away from calling kernel_neon_begin() and kernel_neon_end() directly, and instead, use the newly introduced scoped_ksimd() API. This permits arm64 to modify the kernel mode NEON API without affecting code that is shared between ARM and arm64. Signed-off-by: Ard Biesheuvel --- crypto/aegis128-neon.c | 33 +++++++------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/crypto/aegis128-neon.c b/crypto/aegis128-neon.c index 9ee50549e823..b41807e63bd3 100644 --- a/crypto/aegis128-neon.c +++ b/crypto/aegis128-neon.c @@ -4,7 +4,7 @@ */ =20 #include -#include +#include =20 #include "aegis.h" #include "aegis-neon.h" @@ -24,32 +24,28 @@ void crypto_aegis128_init_simd(struct aegis_state *stat= e, const union aegis_block *key, const u8 *iv) { - kernel_neon_begin(); - crypto_aegis128_init_neon(state, key, iv); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_init_neon(state, key, iv); } =20 void crypto_aegis128_update_simd(struct aegis_state *state, const void *ms= g) { - kernel_neon_begin(); - crypto_aegis128_update_neon(state, msg); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_update_neon(state, msg); } =20 void crypto_aegis128_encrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size) { - kernel_neon_begin(); - crypto_aegis128_encrypt_chunk_neon(state, dst, src, size); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_encrypt_chunk_neon(state, dst, src, size); } =20 void crypto_aegis128_decrypt_chunk_simd(struct aegis_state *state, u8 *dst, const u8 *src, unsigned int size) { - kernel_neon_begin(); - crypto_aegis128_decrypt_chunk_neon(state, dst, src, size); - kernel_neon_end(); + scoped_ksimd() + crypto_aegis128_decrypt_chunk_neon(state, dst, src, size); } =20 int crypto_aegis128_final_simd(struct aegis_state *state, @@ -58,12 +54,7 @@ int crypto_aegis128_final_simd(struct aegis_state *state, unsigned int cryptlen, unsigned int authsize) { - int ret; - - kernel_neon_begin(); - ret =3D crypto_aegis128_final_neon(state, tag_xor, assoclen, cryptlen, - authsize); - kernel_neon_end(); - - return ret; + scoped_ksimd() + return crypto_aegis128_final_neon(state, tag_xor, assoclen, + cryptlen, authsize); } --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10FCE30FF2E for ; Wed, 1 Oct 2025 21:03:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352640; cv=none; b=FGTvQ19TK3+pX+sw0gW/xi+6d8kfg5QvZtenQXlr0QnS5F2v6+OY8oU7djOL81BUs0lRSZ+icMRzrE/v2Z+vB264O/ROQS196QT0n8jBbRWg1zWKo0ugZF84pFxo0KmlH4Vk44cIrOCCUs6n03+wTDu7J4QgqOOIN7eVj4FykqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352640; c=relaxed/simple; bh=nadp2g4HyADqc9HwrCLpsPTlFfrdWtCsrrf5SqzqoPY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ivZOqwOb4Sk4NhoW37lUO0wipGpjWFucTSMJRm5trDVsKyYhYtMQgk+EnWo44DZmnstLhLzqjPtMlT/DPIjlxyIooENfQ7AtCYmMqsB0CTsstGfiW3FmJdJWcZe88pbHFwqk6iuCuXLjTxUrZTs7cr1W3podG+HJho8EbS7/QdM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jVV2aSE7; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jVV2aSE7" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b3cbee9769fso45929166b.3 for ; Wed, 01 Oct 2025 14:03:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352637; x=1759957437; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nLN5jr5EZSwszf+F9TmirI7MwTEPAK5WzBEEPFGFb+c=; b=jVV2aSE7+z4M1GA169jDvqZznvrLuakEEvE8JVEaB6RCO3PdIH7227PyVGfWkLduwi mplg5J3t9n/s2OWHnCHjARQPfkM3iZEcjbOUWIcOWGgRhjiaKUxHf0MtUuf0I0L0WcxZ 2l5Ybx1H14BNAR70nfGjxDnhUuI6hKtN/ppOpyfZz8/f/Qitp2VVIg8mgWYapEplfCsQ JdP0jaK3q+tbxS3yeqbgowHqqtbDWeqDj/NEp7uGQ+Jr7N+lnMYvtWCBqJbjw1jC7NGd 4BMXjaHRnNVN/0y0D+lIDtPhPl1jm4KIhz/nCBEB/CtFgXl1T90k0TIKEs9PhxdtBQHN 4ABg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352637; x=1759957437; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nLN5jr5EZSwszf+F9TmirI7MwTEPAK5WzBEEPFGFb+c=; b=gL0Ssx5+iXop4vxMX69wmOXfR+ZM1t43jSjKGNxBooX8umayC0wBVzEFxbtmcJc4Sz Bb7uiwVFdwA0IXsvEwzBIaIy8ttw+nnY8J8TuOW8/mQYuPBAz2UgkYFOgUhMy1qnvF3S G914f+WnrgrvHf5HHZfrW2dc/lV/QLvUzz+g2A+8rmKeLM+kED9Kej9yQ4tHG0LI9+nS Nn3aRHPquyh2IMvC/IShdoEHzVAtQ47e5wuaUtZF2t55moRG4lhbeAL377rtG/1gZ6mW TgRSsgenWXi4XtprPXpQ36OVY/6bFMXa+oj8Lu2AAh2yv50hvoPZBQBEwTH+sanoEBRC oPVA== X-Forwarded-Encrypted: i=1; AJvYcCVsma4e4A2ojYfCUbClnj/kw/8wJ/aosh1wsCgsD4lyI53Qj4CTMmLDv6S6+oTkKa2VJoO4YrcAfq664+E=@vger.kernel.org X-Gm-Message-State: AOJu0YxupZvYt6wKvdA+jSlNAXeKVWM9tR+tEIpKKgnKxbIB5AQHpyuQ 5yeUXtSBxDAL3dPNMh8DPPd8L+/n9347b1AlCISV80wg0uFSsXjs4Fl3sOoEg+19vO1EVwStmw= = X-Google-Smtp-Source: AGHT+IE/5dzsYm+uKsCA7BqxixvVFjLG0cblUGfuQzl+SHoar/yQvT4DNAxneFAQwNCEwxJSwxiGrNJ9 X-Received: from ejbgx6.prod.google.com ([2002:a17:906:f1c6:b0:b3d:b7b8:e443]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:724a:b0:b41:abc9:613c with SMTP id a640c23a62f3a-b46e97621d3mr576098566b.51.1759352637265; Wed, 01 Oct 2025 14:03:57 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:07 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3184; i=ardb@kernel.org; h=from:subject; bh=gbq7vX1V656Kd5zOXbjYzxNSx5HVr+W5CHBw6KSs3yM=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutAd/TfvXOrnNLdzMmuP5wDbmqvKCn8Y7b23smLTMO eYwY+nPjlIWBjEuBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRK48Y/ldUdS5R2rJRRfXS RDOpUsODP9J6kjct2vbf13Ji73OtackM/0vYOs/H38s/k8jeMbdyy6Hvl1ee9mG9fkrdSVhmk13 dZVYA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-27-ardb+git@google.com> Subject: [PATCH v2 05/20] raid6: Move to more abstract 'ksimd' guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Move away from calling kernel_neon_begin() and kernel_neon_end() directly, and instead, use the newly introduced scoped_ksimd() API. This permits arm64 to modify the kernel mode NEON API without affecting code that is shared between ARM and arm64. Signed-off-by: Ard Biesheuvel --- lib/raid6/neon.c | 17 +++++++---------- lib/raid6/recov_neon.c | 15 ++++++--------- 2 files changed, 13 insertions(+), 19 deletions(-) diff --git a/lib/raid6/neon.c b/lib/raid6/neon.c index 0a2e76035ea9..6d9474ce6da9 100644 --- a/lib/raid6/neon.c +++ b/lib/raid6/neon.c @@ -8,10 +8,9 @@ #include =20 #ifdef __KERNEL__ -#include +#include #else -#define kernel_neon_begin() -#define kernel_neon_end() +#define scoped_ksimd() #define cpu_has_neon() (1) #endif =20 @@ -32,10 +31,9 @@ { \ void raid6_neon ## _n ## _gen_syndrome_real(int, \ unsigned long, void**); \ - kernel_neon_begin(); \ - raid6_neon ## _n ## _gen_syndrome_real(disks, \ + scoped_ksimd() \ + raid6_neon ## _n ## _gen_syndrome_real(disks, \ (unsigned long)bytes, ptrs); \ - kernel_neon_end(); \ } \ static void raid6_neon ## _n ## _xor_syndrome(int disks, \ int start, int stop, \ @@ -43,10 +41,9 @@ { \ void raid6_neon ## _n ## _xor_syndrome_real(int, \ int, int, unsigned long, void**); \ - kernel_neon_begin(); \ - raid6_neon ## _n ## _xor_syndrome_real(disks, \ - start, stop, (unsigned long)bytes, ptrs); \ - kernel_neon_end(); \ + scoped_ksimd() \ + raid6_neon ## _n ## _xor_syndrome_real(disks, \ + start, stop, (unsigned long)bytes, ptrs);\ } \ struct raid6_calls const raid6_neonx ## _n =3D { \ raid6_neon ## _n ## _gen_syndrome, \ diff --git a/lib/raid6/recov_neon.c b/lib/raid6/recov_neon.c index 70e1404c1512..9d99aeabd31a 100644 --- a/lib/raid6/recov_neon.c +++ b/lib/raid6/recov_neon.c @@ -7,11 +7,10 @@ #include =20 #ifdef __KERNEL__ -#include +#include #include "neon.h" #else -#define kernel_neon_begin() -#define kernel_neon_end() +#define scoped_ksimd() #define cpu_has_neon() (1) #endif =20 @@ -55,9 +54,8 @@ static void raid6_2data_recov_neon(int disks, size_t byte= s, int faila, qmul =3D raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila] ^ raid6_gfexp[failb]]]; =20 - kernel_neon_begin(); - __raid6_2data_recov_neon(bytes, p, q, dp, dq, pbmul, qmul); - kernel_neon_end(); + scoped_ksimd() + __raid6_2data_recov_neon(bytes, p, q, dp, dq, pbmul, qmul); } =20 static void raid6_datap_recov_neon(int disks, size_t bytes, int faila, @@ -86,9 +84,8 @@ static void raid6_datap_recov_neon(int disks, size_t byte= s, int faila, /* Now, pick the proper data tables */ qmul =3D raid6_vgfmul[raid6_gfinv[raid6_gfexp[faila]]]; =20 - kernel_neon_begin(); - __raid6_datap_recov_neon(bytes, p, q, dq, qmul); - kernel_neon_end(); + scoped_ksimd() + __raid6_datap_recov_neon(bytes, p, q, dq, qmul); } =20 const struct raid6_recov_calls raid6_recov_neon =3D { --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A0653101AE for ; Wed, 1 Oct 2025 21:03:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352641; cv=none; b=kbPk6Fq9Km/Mb5i5cdYPb9lcoookJzl5/frlMBqYIUfXSZp9CnH1RC8ZHOchCkbunU0kuwi6csH6hV30zKESM2kcC58V+L3FjvpC7ouDGwjMlQJGXp/EUKd6bEsMZN2Csi8uvmlBfz1MM57y7rre9DKNEtL+UPp+rGlc4gpLMM8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352641; c=relaxed/simple; bh=G71rjTtZIaGIXR3UGbv3E5LniLfYD2qy9Arh2X+LvNw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eW2cgjVXsm1jIBaYIDSRZH7JvHJZ2SYS32b/4TA/TujCFOj6X27o2HZBPVWsSEOV4QDTYZ7IEZ5S6sbf2h1wi96yS1VzQozj8EL8One8/H8h19X8CGmX2ctgHCEy6/MvoUdyTD4/pcsTF91EIT9+yjZFN3Pzz7ybdpreHHj2dQI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=c1+HtXub; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="c1+HtXub" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3ece0fd841cso78701f8f.0 for ; Wed, 01 Oct 2025 14:03:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352638; x=1759957438; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4j20MEC8Ax3r4R1mIrlpRXC+UmhNM+andOU7drpFlJg=; b=c1+HtXub1tsW6Ro/L61pROfXURB4H1hVBy/UvsvtwubqYow8d4Jywu4xt6S29GXX/C 52xPUQ1o7dxEfXlf8FU8XGM6UFfTI2e52i90Ur6MAuHGq/Q317Z9ewr212k+1bkX8Ljk GdWjmj1rsJuUGEgcZmJmk1Fkd4dvpri6tZY5xXQYcPMzGAT37il9C+4RWej+QSe5WV5X Bi644UlA4m8QQtP4/1icEOc2VCvSH9QoIL0jHUZn0O3eL7oIryyguz/thA9d7L2MmwDZ 0TBSqoJa7PqNuOZhrB/maL7kQUmpmZgNXxWqwnO3HXbHiFYV8DtgumHMF++1c4wMhheD Kxdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352638; x=1759957438; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4j20MEC8Ax3r4R1mIrlpRXC+UmhNM+andOU7drpFlJg=; b=XO5eHMkPgGVNDxGz3gKvlkMsVQD87VmI91Q4QknmYQnna3ooBqrfrKJO7LwRml9w5E 5aY4X8NYvmdVpluKQaWKP73Vbf1pV4P0sxiZ6QjHKFJTEcMphzJV3peEnzDtk9cC2FIa b2HTck83WQcNwmHUJ1MYoZRU9ionGTrZzROqIgETF+oELszhTlRXE8OtEXeXAj40Hko/ l5Ys8Ka+m055UFIUSlbrAneK0SZzK9iNOv4YjsyDzC97JXRRAPexuSTxYclcT1QqtfK6 YnoM4+gi+INf/OOt7XGIUnCBqldW3Gwk3iRQvmiSq2G1slmxNmYZKmzM3/9RK6G/fWbY zE7Q== X-Forwarded-Encrypted: i=1; AJvYcCWuGd+NHq6FyUS/Hb6pwMHcYuxNKID2b2qy/XjAwSwz2drtM3h2xxCQJrhsDoDBNAhgNT6dG6xVDiIGQRI=@vger.kernel.org X-Gm-Message-State: AOJu0YzZRG7i4b3ABaX5GT88r85pFD0O871eWm5f/Vr5JwEnwFp0u9OZ Tkp33tBD9wu5XQlyRFcS9RhDdbGG6483niC7u699UISCpO2ULBsvqiigKHAHivLNSZS0MgF7gA= = X-Google-Smtp-Source: AGHT+IEuG1RbtJ6FiwIucAXGOGuLO6+jN3odMsNV/hZ3o58B9bsNzZE7dpr8nZZ7AG2wrt76ICGCYLz6 X-Received: from wmph10.prod.google.com ([2002:a05:600c:498a:b0:46e:3d73:fc5f]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:220b:b0:402:71e6:5e00 with SMTP id ffacd0b85a97d-425577ed455mr3244884f8f.7.1759352638294; Wed, 01 Oct 2025 14:03:58 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:08 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=923; i=ardb@kernel.org; h=from:subject; bh=cfu0++gxesQ+j0mFCsE0nnO6ije/EROXIJJwVNT18uY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutMcBulsb9Sf4Rn696XB38WaDfWe15F4pq8cpdulUF insmBLQUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACYiF8zwT73e6+Ks9K/+Zebd R+6sq/m6USFAcL2p+MU135V7PqktCmRk+LXMZ/6d0FIN7xxPpoX/jt5W3mGrLFV3wuGL08ZXstw XeQE= X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-28-ardb+git@google.com> Subject: [PATCH v2 06/20] crypto/arm64: aes-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it explicitly in order to prevent scheduling latency spikes. Reviewed-by: Mark Brown Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce= -ccm-glue.c index 2d791d51891b..2eb4e76cabc3 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -114,11 +114,8 @@ static u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[= ], u32 abytes, in +=3D adv; abytes -=3D adv; =20 - if (unlikely(rem)) { - kernel_neon_end(); - kernel_neon_begin(); + if (unlikely(rem)) macp =3D 0; - } } else { u32 l =3D min(AES_BLOCK_SIZE - macp, abytes); =20 --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10ED7310620 for ; Wed, 1 Oct 2025 21:04:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352642; cv=none; b=AY4+lFiPUKr+Ruu1k1hio0BBKHBsg4nzg1323M2ZiCFHrZgV2AicRp8Ax1iJJNXqjz30qA7EQiVo94yFQECAActAfh1Bd+YAEoQsE+U7Ea+jvc7pSP0Lcs7ZK7myspcL3gE5OaVzRv5C28PtNZINR+NI/38/pZylcNadAD8mlo0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352642; c=relaxed/simple; bh=JgomGjz8a4MC2QvV0Ypiia/XgnueJCeB/ViBoqFjfsg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sRQwqFveNfASmDKi0RRfJYecxBP0OPKmGtuXlJSW2Z3UFGbc0dopI0vYrt1ONu7VL60jvm8nbrVqEGFAdZoTDPrx6TD0PS9IxbQ5Vp1v/led4Wfvu3el7yKsA77X4aa2dyqqbtTeu8GpvFLWhewKtgM576RUIF5zn42LKqIwRhs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=h8VIrNJ/; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="h8VIrNJ/" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e4fa584e7so1075565e9.0 for ; Wed, 01 Oct 2025 14:04:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352639; x=1759957439; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=T9XPXiALrIFv+t8yA8FOCyMwqSyC0AWzcr+6NOSN1Gk=; b=h8VIrNJ/VmrYYsz/GtcOwpi0Og8JENmIMPHy4XbGbP86tSCo7qB8KkI4Fy8gK3ugaW VUogwLnpXyQOWJnXnYYGqKdHwONZMyR7ZLDkEbXuafu2efcEcfGjsvvmJa/vBx96DyYO Ab22Dy0ITwtUeDBd1DFcvVOSpxnyNY1mL2OuoNVkCCYWl7aIbU/VYrGMi7l6lq6viwMV VaonQoAENqWA7/dJvKS3Lq3jZ02xVoJjZ1DGNcOKGZ9JFomWsuWCtD5ZYM7MV9YN2ZPp el8zId77cvrTOVKeb0pQ5a/0LUK6QFGeBslOl2OPoitw5vMGpzRDg02oOltrvlZ1xYpe mUPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352639; x=1759957439; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T9XPXiALrIFv+t8yA8FOCyMwqSyC0AWzcr+6NOSN1Gk=; b=e4MhjWLR/CnsMcCkMO3o4Wt61A8mfBHlesuTDU2iDHWKDCzJDvjtmM4yYAq9U6eX8g gEVTY4BRZvBG/2QVmn50HWI9aZaYOFksiYRHCAhzS6+3V1zd7HVz7SmHo4IikbnwM9Js yXlTIDiMfVEyXCSatp2S8CncUIrPMkBoNzGrkVVNRsbocVlCZTD9gpWsHCGZEDSbV4uJ uIDZx4UL1lAKhYtBqLYjIpHutYmz1q/TwoJ0kZnuLLq16zxI/HG3iTYK5aNdFJPCSWSL d2jJjLKs5CzeqVN9AyNpXBPMlmQZ7oFQ+ng5DORi0tStRem6co5MrDKI/e+0AK6CUYt3 471w== X-Forwarded-Encrypted: i=1; AJvYcCXyItP+BCCcFk4+8BpelLorGXwHcOxSibSTzmJoheYi0o1/aENpfKH6/GQtofS3laemVgMdVDWXAjLVTVI=@vger.kernel.org X-Gm-Message-State: AOJu0YyiBMZAilikqwX6H/hUvJ0IxmCxaKQoVFYaHkH2qKs8AKSgfqfF aBXt8xV1iVKI2dGGK50doZBkKAi9pRP/zr3gvjh8nVDd9J1uqtqOt37SPBW9EoyElZyiFc3hEw= = X-Google-Smtp-Source: AGHT+IHiZKFz0FRSgxw7esbxI8PZwcXr45UkMRXq0q8oyZAHd/2cmjG7u/BqIWintM4Gmbm+ulYQB/Ip X-Received: from wmcu7.prod.google.com ([2002:a7b:c047:0:b0:46e:3190:9ce]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8b85:b0:46e:41b0:f0cb with SMTP id 5b1f17b1804b1-46e612cb765mr41091005e9.25.1759352639460; Wed, 01 Oct 2025 14:03:59 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:09 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1444; i=ardb@kernel.org; h=from:subject; bh=zYFiNr/eldWQX1gt5iI0fKjxqDInIe25UpxzRqLUttk=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutBe85rrrjE9313UZy3CfWrLq8sGllfGsK2ovnFXc6 m3AqyzXUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZS0MTI0Hh27bH1f5+ka1/h 3LiIW9F7bdJDrp8Tcy4K3tTZdLVb6g3DP43ZXL/X7Ix7b1m3Nfm/Y/5h3sZDi0KXCneJHVugbyP CygsA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-29-ardb+git@google.com> Subject: [PATCH v2 07/20] crypto/arm64: sm4-ce-ccm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). Reviewed-by: Mark Brown Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sm4-ce-ccm-glue.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce= -ccm-glue.c index e9cc1c1364ec..f9771ab2a05f 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -179,11 +179,7 @@ static int ccm_crypt(struct aead_request *req, struct = skcipher_walk *walk, walk->src.virt.addr, walk->iv, walk->nbytes - tail, mac); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); } =20 if (walk->nbytes) { @@ -193,15 +189,13 @@ static int ccm_crypt(struct aead_request *req, struct= skcipher_walk *walk, =20 sm4_ce_ccm_final(rkey_enc, ctr0, mac); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, 0); } else { sm4_ce_ccm_final(rkey_enc, ctr0, mac); - - kernel_neon_end(); } =20 + kernel_neon_end(); + return err; } =20 --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A5AF310654 for ; Wed, 1 Oct 2025 21:04:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352643; cv=none; b=LcXwcWvSBblO5X+KKuyF4o9pzGJ9VIGxkxF/bCuvb44D9a7bL6GHBoEl7VxHK3awRC3cfMce7JO3YCiNAQSdVgiyB/k8xcPzGeDfqbXTj/Tn/pH/PIVnHwwxJvPhlGfGT2S1Cc/pw8fM2kFmCEhpMquReJgU3rIHjE8Yhffqw1k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352643; c=relaxed/simple; bh=+MReetO5lOEq+ENKq1vyXwob8BhS+VD/ZAqXqiNcwSU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=iVb8hZCKgWYJgLPI+O95vvJwVbPqAGNXrmq1YDCf34TXJoQNmRRedW7CKcYyycWYNFGlLUf8qiwWcQt2zp43QaQ+iMvKzcEkf+UAv5xTouCc823H+lWInMCj5h8r0bEhBzXgdZlcBYEAGccs2TA139qEzmwQKxIwVrgz7kQ07fk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fm5dCh83; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fm5dCh83" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-4255d025f41so138573f8f.1 for ; Wed, 01 Oct 2025 14:04:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352640; x=1759957440; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rKVgCyEYUlEYJnl7IzVVeeoQQVhKM/SWcfJ5A7yRcl4=; b=fm5dCh83lLBqxcTwL72MOpunDhd7CLCknZBefrVd0m5Lc0Xhj/0cWjeaV1LEnWqNh1 wjlfObIHEPqW038ECtx3S3egOhQ/PsqI8Y+y/ybcQGCG/SZdM4Np7QuoTIcf0fcMWLKz C2QnzmF/oQ60hRjMbhqH4TOSlpIZmS8H5QCfGO7yoRqN34arxl9XTmysIcDSf5vR0jzX N2kAfwsj47aGSM0UrRRagYoONm2aZtPRi2Yw0Tvu1q56IiqQcAYwXQzIoaMoC69dgns1 yOlKBltbXYSCZCKiQW/EyIJAeYYdGCktVNwX+R19dhh75uCPQzWIXt3TfyaoHzCjCsGa ibrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352640; x=1759957440; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rKVgCyEYUlEYJnl7IzVVeeoQQVhKM/SWcfJ5A7yRcl4=; b=Z2aDU7VSwg04iTH5/eeNDCdgOHhE52rTDVbfMGOV5KwS92idO9Fez+EkgS5ONc/E4+ QRj4p76CmErvd2jRv2fIzmn0oCqlLilM7R4vjekuMZHL6ooLeEoUuEmFJWfAY2/ecRuV Ojjl/djUgi4gTOVPz+lxNWTlfI2adfLt5baQHnjvQVxLW1o3kP/DXDe70QeQ4CF0LRDh m2RUuceAw4RiX53JL9HXzaYufHCyDRP4oOKqTgZKgYRshWaSXA7sWh/yF5ga4As6to+4 JMNiGr7/3eha3p7UF5teVcLu/nJ7DNIY+rZ4+PLDWwB7SUFGOxRYxvLcT75xN9K7G5pC qt0A== X-Forwarded-Encrypted: i=1; AJvYcCX5jBlrOPDV2dTxm/5En4XX9FVaYhm3CcHA82FFT6G1yHV3IEb/bwYBiEd7g7H8A4F328efWXVMpK3vY7U=@vger.kernel.org X-Gm-Message-State: AOJu0YzHb7InFGk84jC5M7mydOvavBM6EmbDiMpu0sLSnFLvJ4Uv2V8t EPC6at+hLRmCtY7KsXvePKBnFWfkrhkj4rak19hzn4Jk3fu5ad2bZTJAuBQ7eh/oprCXlruO6g= = X-Google-Smtp-Source: AGHT+IHDmr/Nj33e2mS518oZ37yPaOtdq4b9XOTrdSeSEajRVgzVPmNn2xLCvIBcNV1g6wzgv51cRxYJ X-Received: from wrge9.prod.google.com ([2002:a05:6000:1789:b0:421:8d6a:e0b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:18a6:b0:3eb:a237:a051 with SMTP id ffacd0b85a97d-42557820770mr3555146f8f.58.1759352640509; Wed, 01 Oct 2025 14:04:00 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:10 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1542; i=ardb@kernel.org; h=from:subject; bh=ibLsffhkx3eQuDIH6wng+X3cde8PFK2qiv2gBQvEgB0=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutDcHLTbs9tkZJfys9AfjVivVRQv3qXhZt9g/3VRX8 NRJsOZvRykLgxgXg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZhImxDDf4esiwetRG6+KPnm dV3iVm7kOcUz9zdPjTqh5XVbccJP178M/0xnpNX4dnd6mPue5Dt550ecgevSD2vKLggFBe54s/6 hGDsA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-30-ardb+git@google.com> Subject: [PATCH v2 08/20] crypto/arm64: sm4-ce-gcm - Avoid pointless yield of the NEON unit From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Kernel mode NEON sections are now preemptible on arm64, and so there is no need to yield it when calling APIs that may sleep. Also, move the calls to kernel_neon_end() to the same scope as kernel_neon_begin(). This is needed for a subsequent change where a stack buffer is allocated transparently and passed to kernel_neon_begin(). Reviewed-by: Mark Brown Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sm4-ce-gcm-glue.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce= -gcm-glue.c index c2ea3d5f690b..170cd0151385 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -165,26 +165,22 @@ static int gcm_crypt(struct aead_request *req, struct= skcipher_walk *walk, ctx->ghash_table, (const u8 *)&lengths); =20 - kernel_neon_end(); - - return skcipher_walk_done(walk, 0); + err =3D skcipher_walk_done(walk, 0); + goto out; } =20 sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, walk->nbytes - tail, ghash, ctx->ghash_table, NULL); =20 - kernel_neon_end(); - err =3D skcipher_walk_done(walk, tail); - - kernel_neon_begin(); } =20 sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, NULL, NULL, iv, walk->nbytes, ghash, ctx->ghash_table, (const u8 *)&lengths); =20 +out: kernel_neon_end(); =20 return err; --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 37D443112BC for ; Wed, 1 Oct 2025 21:04:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352645; cv=none; b=kxbhFRL5ZE2brVevN94WNZm6mC/geNkEaaBEh/ZssIwnCF8LmpFSsrkfYtjxwat1yuPy1RiE8tB9vs2fwleN1hsXJ1Fa9VpyM/dOFMBa28l13Gae3kgtZGNHYkEVjhMC4mu7jJAV1IjLXwzwTL3yjwAMr/AeAko5ERDn1Mf2qTQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352645; c=relaxed/simple; bh=DRYUDExVWqLdJx9CQihH/WVn5tjIj8wKdtaKatPJHTg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mqPOSZGPxvo8thyorxBjoRNCigFPEPlh7SjWAZyH399TziyhDq93MoKI1Rrt6cfJRfNOtTdpssyyY9CtGdXCofyvc45hfp68XoLAnB4jDcqfhTy08KrnWC+6PFzzgs61Fq09ms0vHqipt1d/oBZW1QJlMSBoFX1e6QRosgQFN/U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=oxrhPPOm; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="oxrhPPOm" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3f6b44ab789so86937f8f.3 for ; Wed, 01 Oct 2025 14:04:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352641; x=1759957441; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vE8eXwwbGKy5p5WbHU/i1dMObS51JLHu730fwFC+Odw=; b=oxrhPPOmXUBYITnhFzHIpIkkoeKNcX23Rmm02DLrZknKOLSK2ftkw4aeZhxBYYxy23 vWpX081AFeTDbokDVj/qfqtNfEnn/Wv2fdo+a1SHwLPK0yH5gXP6KYxRXgT1yKWNGw3Z wIUIE8rLdXtZg/omEhGoRJa+TMtYeWzgNJbFSIArN3BGjYkZMdYH4nJBPMMEJfWgVvlL nF5DHWb3fATaz83kIf5sPFc/oHY86MgNusqGAkAsFjpMGrKeqIjAQxiDzZ/RBpj08n4D z3Z9NdmIH+lin1O+CQ+CHWnvu1QqwIJMGRtN5HkF/NAF/Q0ylXijI59T28V9z9+KIFj3 17qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352641; x=1759957441; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vE8eXwwbGKy5p5WbHU/i1dMObS51JLHu730fwFC+Odw=; b=eEdZMSuCiyo5GeRL2wAdya7Eueh9HyOamVGgJcpLcOmSsLZq8kUfHCJQmCiDSfqRUL WinDoFPlbrcLwcL/T/8arbAiVFDaCUxG9qYx6NrjKhQoAoxln+OPZ5WnpOvoAN1lQ7br 0TiR9uYwJ22ZT0Y86iJ0X4Qc2FPk7f5DEIzeD9zFv7fIxjrN/vm3MY11LLiDfK6JVgUq +DM8nFATGsv8cmtO9sA739DiU89Z29kLrh1pf+SB3fP6TKshQdAa3kTUhT4vAGdIGxRT KOY1QYG2/JnPF/pM2CHWb8/lLebfsd13RNrkPoiAu8Z8/Qd6Din2F43KFERvY7OEEWa7 20eA== X-Forwarded-Encrypted: i=1; AJvYcCUvMFgCI36YNrPiykOXzl8wep8FZ8nxzBo0iAp1KNA+zZ/4QHDkjAomRMmc4GQi9TsFh9kj1O7BLyNJ3DE=@vger.kernel.org X-Gm-Message-State: AOJu0YwDcpfcaE46Hp5JH6Vu6BrscUV6/lTGL4W3Nh8BGV6ImDQbxU/i 9n+Fvl/tS7cWGZzagkh5+uoKrKAKlIg/uOs3xhn6/aPoE8Fy7alXCXc66dYEeAzkxlw3Oq1myg= = X-Google-Smtp-Source: AGHT+IH7ZtZzU237M2m1tNwTXmJBPEEEg2ifHIsvYzIIZQnXlN2lH+rAVb8JyGkAh+fVohPUM/1C32AM X-Received: from wmlv6.prod.google.com ([2002:a05:600c:2146:b0:46e:19f9:cfe9]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8414:b0:46e:206a:78cc with SMTP id 5b1f17b1804b1-46e612c9042mr40692505e9.28.1759352641678; Wed, 01 Oct 2025 14:04:01 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:11 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=5792; i=ardb@kernel.org; h=from:subject; bh=u6utO88LPPilhPgrmOnUNkYpCEnFY27X7HNtZdaOtj0=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutA9cXKLzKkytjZfvzuMw27rm9IWJmorzmss7FggGd Tm/kFjfUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaiYsLwv25T0T8WyXkR2rOy xBgyucWuPZu+c35P51XpqCxjrQq1xQz/3Zkj9jtHHaqsdJ52+exfZmlp3ycpykUPWr0XWZw6cje DHwA= X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-31-ardb+git@google.com> Subject: [PATCH v2 09/20] lib/crc: Switch ARM and arm64 to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Before modifying the prototypes of kernel_neon_begin() and kernel_neon_end() to accommodate kernel mode FP/SIMD state buffers allocated on the stack, move arm64 to the new 'ksimd' scoped guard API, which encapsulates the calls to those functions. For symmetry, do the same for 32-bit ARM too. Signed-off-by: Ard Biesheuvel --- lib/crc/arm/crc-t10dif.h | 16 +++++----------- lib/crc/arm/crc32.h | 11 ++++------- lib/crc/arm64/crc-t10dif.h | 16 +++++----------- lib/crc/arm64/crc32.h | 16 ++++++---------- 4 files changed, 20 insertions(+), 39 deletions(-) diff --git a/lib/crc/arm/crc-t10dif.h b/lib/crc/arm/crc-t10dif.h index 2edf7e9681d0..133a773b8248 100644 --- a/lib/crc/arm/crc-t10dif.h +++ b/lib/crc/arm/crc-t10dif.h @@ -7,7 +7,6 @@ =20 #include =20 -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); @@ -22,21 +21,16 @@ asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u= 8 *buf, size_t len, static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { - if (static_branch_likely(&have_pmull)) { - if (crypto_simd_usable()) { - kernel_neon_begin(); - crc =3D crc_t10dif_pmull64(crc, data, length); - kernel_neon_end(); - return crc; - } + if (static_branch_likely(&have_pmull) && crypto_simd_usable()) { + scoped_ksimd() + return crc_t10dif_pmull64(crc, data, length); } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_neon) && crypto_simd_usable()) { u8 buf[16] __aligned(16); =20 - kernel_neon_begin(); - crc_t10dif_pmull8(crc, data, length, buf); - kernel_neon_end(); + scoped_ksimd() + crc_t10dif_pmull8(crc, data, length, buf); =20 return crc_t10dif_generic(0, buf, sizeof(buf)); } diff --git a/lib/crc/arm/crc32.h b/lib/crc/arm/crc32.h index 018007e162a2..32ad299319cd 100644 --- a/lib/crc/arm/crc32.h +++ b/lib/crc/arm/crc32.h @@ -10,7 +10,6 @@ #include =20 #include -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32); @@ -44,9 +43,8 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, siz= e_t len) len -=3D n; } n =3D round_down(len, 16); - kernel_neon_begin(); - crc =3D crc32_pmull_le(p, n, crc); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_pmull_le(p, n, crc); p +=3D n; len -=3D n; } @@ -73,9 +71,8 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, size_= t len) len -=3D n; } n =3D round_down(len, 16); - kernel_neon_begin(); - crc =3D crc32c_pmull_le(p, n, crc); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32c_pmull_le(p, n, crc); p +=3D n; len -=3D n; } diff --git a/lib/crc/arm64/crc-t10dif.h b/lib/crc/arm64/crc-t10dif.h index c4521a7f1ee9..dcbee08801d6 100644 --- a/lib/crc/arm64/crc-t10dif.h +++ b/lib/crc/arm64/crc-t10dif.h @@ -9,7 +9,6 @@ =20 #include =20 -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_asimd); @@ -24,21 +23,16 @@ asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const= u8 *buf, size_t len); static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length) { if (length >=3D CRC_T10DIF_PMULL_CHUNK_SIZE) { - if (static_branch_likely(&have_pmull)) { - if (crypto_simd_usable()) { - kernel_neon_begin(); - crc =3D crc_t10dif_pmull_p64(crc, data, length); - kernel_neon_end(); - return crc; - } + if (static_branch_likely(&have_pmull) && crypto_simd_usable()) { + scoped_ksimd() + return crc_t10dif_pmull_p64(crc, data, length); } else if (length > CRC_T10DIF_PMULL_CHUNK_SIZE && static_branch_likely(&have_asimd) && crypto_simd_usable()) { u8 buf[16]; =20 - kernel_neon_begin(); - crc_t10dif_pmull_p8(crc, data, length, buf); - kernel_neon_end(); + scoped_ksimd() + crc_t10dif_pmull_p8(crc, data, length, buf); =20 return crc_t10dif_generic(0, buf, sizeof(buf)); } diff --git a/lib/crc/arm64/crc32.h b/lib/crc/arm64/crc32.h index 6e5dec45f05d..2b5cbb686a13 100644 --- a/lib/crc/arm64/crc32.h +++ b/lib/crc/arm64/crc32.h @@ -2,7 +2,6 @@ =20 #include #include -#include #include =20 #include @@ -24,9 +23,8 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, siz= e_t len) return crc32_le_base(crc, p, len); =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && crypto_simd_usab= le()) { - kernel_neon_begin(); - crc =3D crc32_le_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_le_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; @@ -44,9 +42,8 @@ static inline u32 crc32c_arch(u32 crc, const u8 *p, size_= t len) return crc32c_base(crc, p, len); =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && crypto_simd_usab= le()) { - kernel_neon_begin(); - crc =3D crc32c_le_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32c_le_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; @@ -64,9 +61,8 @@ static inline u32 crc32_be_arch(u32 crc, const u8 *p, siz= e_t len) return crc32_be_base(crc, p, len); =20 if (len >=3D min_len && cpu_have_named_feature(PMULL) && crypto_simd_usab= le()) { - kernel_neon_begin(); - crc =3D crc32_be_arm64_4way(crc, p, len); - kernel_neon_end(); + scoped_ksimd() + crc =3D crc32_be_arm64_4way(crc, p, len); =20 p +=3D round_down(len, 64); len %=3D 64; --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40382311953 for ; Wed, 1 Oct 2025 21:04:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352647; cv=none; b=pxyKqKeWMYBO3vkr+akH28WxrvXigaa7e8Lgu9RHJL3JJZrAj69ZWM9p/L1CGTspkoBF/itD5O05OFNjjhI8IqEIlB5nY7h7RnmhZ0JkM0uiPqjUunZY6nIVhkZX7uo0BjsP9yN4IMt3BTtl2g5M3JUvdhOgwuFRhoqSdXXQCso= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352647; c=relaxed/simple; bh=zG+BMEqFMjLimSd+uHsTmtpB2rdFbSN9A/I9ca/nWJc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NPiemp49UCLNVC8H2gg+qzBYtfrRmBxMQMgTi6zGuPFbZ0ZLZ0VAsd39q3i8gVRG4UyP+M+wkz2A4kAx0ep74hCeCF8BjhaxnyKQ9TNSeBUbqOuK1GPBBnTAkwMN92g8hI018AvzDeDcl+FHcCIgQFIKD3W4ZjhMtVvk2vXnKXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=o8IV+b2/; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="o8IV+b2/" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e3e177893so1535965e9.2 for ; Wed, 01 Oct 2025 14:04:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352642; x=1759957442; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wjS59fjLSBp23Zb/zP6WgADegb2YBV3/Wv9KxGpAaxY=; b=o8IV+b2/1q6jm/GUPgeWTEe9y0iy1F+1PX3QUYe0GKXACEcW44HFdL9tySc7wgH4hc LpT69GZCpXaUFfNwXgDVglEkXeFe2mJHPN/TwE4Y2VTFGCbZIKnOt7e32l/PPJUJAFy/ /rl82LVCKOZI4A++vApXFh1mmwqziu66kIjbQqmomI//tLJBaTYSs/2sH7opr8guFf8v YnD+BXVVy/4B5l46gq7wDYp9G00mkkxlnER5N+ldHQsI9PPD9wViFcB1aB84ttFS30BA ObF+w5pCP5uCdFrsfczejRY9RFah/smeQo8bPOiJogLLSv7tPWAECRVJdvGn151dEKc1 iafg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352642; x=1759957442; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wjS59fjLSBp23Zb/zP6WgADegb2YBV3/Wv9KxGpAaxY=; b=jxVg8EHaEgnCG89WZjgWGNCDuNZw9DVFDW4mN91CrEQ5VbwU8HgpMlB168cAfdBirL 5cRyrLsGCh1jhnFhDe9QluC2QCc/q9oiXFb0P+QWngI6LBOQQDu3ngH93ER/2dEW/fem ZH/6c4zkeIIIRtfsmFloBjL2e9Jp+Zwby8AaUTgit/vaCd2PocTshYb2lD3EYeU9LsAp 7S1Kw8BUEFq/WDFbpRzF90B2c+cejO6fpjMJ/BwdZJCyCDXoN/toUrozQ7iJvJUvXtBz NcExDMJuc1zGffJCMzczICruiFvHHLJ2GiNx8GEm3H3qkPKpcW1nyLEZuz5hi3fobfoh 01og== X-Forwarded-Encrypted: i=1; AJvYcCU+K0tra5PFa/+SQiaQkYkYIXmWeNTHVe6m2CnaGTgqLIDgjz/i8RAma4QoCMQ3b8zrYYOnSbfz1EVg8pw=@vger.kernel.org X-Gm-Message-State: AOJu0Yz61R/QAzcbabLt6wwzhRxjo9sPD5GLfoBJFJ7SQ6Ew1UiAfrki KOBzTyfOg0Ubm+n8DRJjTCL2G/322OcJ8vR2wBKChovXM+p6roHk3gb5nhaOgkZTCDmdvj3ajA= = X-Google-Smtp-Source: AGHT+IEuIn0Xw8cSb0i16YD4euJ4+Kdw5OD8+fY/WVV0z4DuW0G513nRxE00Rg9xEjZ/WTTSw7yU1AAt X-Received: from wmcu7.prod.google.com ([2002:a7b:c047:0:b0:46e:3190:9ce]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:820c:b0:46e:4882:94c7 with SMTP id 5b1f17b1804b1-46e612cb269mr38395125e9.28.1759352642677; Wed, 01 Oct 2025 14:04:02 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:12 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=9679; i=ardb@kernel.org; h=from:subject; bh=2x6PFDcDyExpe+QvLhnrBBMaZb8fzyr9//78pWmLSeM=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutC8aWtMOahe/uP15eVdzpfyKr+IGu7exxaxkcM5rX +RYwv2qo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEwkcRUjQwdTaMu7taGPZ98u lwx8s7Px0My84NWPth/49DM9af5WG6CKr+tNrpyZcUZph+bui8JeH/80b/1yz3l5gtNDxsddDT8 yOAA= X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-32-ardb+git@google.com> Subject: [PATCH v2 10/20] lib/crypto: Switch ARM and arm64 to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Before modifying the prototypes of kernel_neon_begin() and kernel_neon_end() to accommodate kernel mode FP/SIMD state buffers allocated on the stack, move arm64 to the new 'ksimd' scoped guard API, which encapsulates the calls to those functions. For symmetry, do the same for 32-bit ARM too. Signed-off-by: Ard Biesheuvel --- lib/crypto/arm/chacha-glue.c | 6 ++---- lib/crypto/arm/poly1305-glue.c | 6 ++---- lib/crypto/arm/sha1.h | 13 ++++++------- lib/crypto/arm/sha256.h | 14 +++++++------- lib/crypto/arm/sha512.h | 6 +++--- lib/crypto/arm64/chacha-neon-glue.c | 11 ++++------- lib/crypto/arm64/poly1305-glue.c | 6 ++---- lib/crypto/arm64/sha1.h | 7 +++---- lib/crypto/arm64/sha256.h | 15 +++++++-------- lib/crypto/arm64/sha512.h | 8 ++++---- 10 files changed, 40 insertions(+), 52 deletions(-) diff --git a/lib/crypto/arm/chacha-glue.c b/lib/crypto/arm/chacha-glue.c index 88ec96415283..9c2e8d5edf20 100644 --- a/lib/crypto/arm/chacha-glue.c +++ b/lib/crypto/arm/chacha-glue.c @@ -14,7 +14,6 @@ =20 #include #include -#include #include =20 asmlinkage void chacha_block_xor_neon(const struct chacha_state *state, @@ -90,9 +89,8 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *ds= t, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, bytes, SZ_4K); =20 - kernel_neon_begin(); - chacha_doneon(state, dst, src, todo, nrounds); - kernel_neon_end(); + scoped_ksimd() + chacha_doneon(state, dst, src, todo, nrounds); =20 bytes -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm/poly1305-glue.c b/lib/crypto/arm/poly1305-glue.c index 2d86c78af883..3e4624477e9f 100644 --- a/lib/crypto/arm/poly1305-glue.c +++ b/lib/crypto/arm/poly1305-glue.c @@ -6,7 +6,6 @@ */ =20 #include -#include #include #include #include @@ -39,9 +38,8 @@ void poly1305_blocks_arch(struct poly1305_block_state *st= ate, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, len, SZ_4K); =20 - kernel_neon_begin(); - poly1305_blocks_neon(state, src, todo, padbit); - kernel_neon_end(); + scoped_ksimd() + poly1305_blocks_neon(state, src, todo, padbit); =20 len -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm/sha1.h b/lib/crypto/arm/sha1.h index fa1e92419000..a4296ffefd05 100644 --- a/lib/crypto/arm/sha1.h +++ b/lib/crypto/arm/sha1.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); @@ -22,12 +21,12 @@ static void sha1_blocks(struct sha1_block_state *state, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && likely(may_use_simd())) { - kernel_neon_begin(); - if (static_branch_likely(&have_ce)) - sha1_ce_transform(state, data, nblocks); - else - sha1_transform_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() { + if (static_branch_likely(&have_ce)) + sha1_ce_transform(state, data, nblocks); + else + sha1_transform_neon(state, data, nblocks); + } } else { sha1_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm/sha256.h b/lib/crypto/arm/sha256.h index da75cbdc51d4..df861cc5b9ff 100644 --- a/lib/crypto/arm/sha256.h +++ b/lib/crypto/arm/sha256.h @@ -4,7 +4,7 @@ * * Copyright 2025 Google LLC */ -#include +#include #include =20 asmlinkage void sha256_block_data_order(struct sha256_block_state *state, @@ -22,12 +22,12 @@ static void sha256_blocks(struct sha256_block_state *st= ate, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && crypto_simd_usable()) { - kernel_neon_begin(); - if (static_branch_likely(&have_ce)) - sha256_ce_transform(state, data, nblocks); - else - sha256_block_data_order_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() { + if (static_branch_likely(&have_ce)) + sha256_ce_transform(state, data, nblocks); + else + sha256_block_data_order_neon(state, data, nblocks); + } } else { sha256_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm/sha512.h b/lib/crypto/arm/sha512.h index f147b6490d6c..35b80e7e7db7 100644 --- a/lib/crypto/arm/sha512.h +++ b/lib/crypto/arm/sha512.h @@ -6,6 +6,7 @@ */ =20 #include +#include #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); @@ -20,9 +21,8 @@ static void sha512_blocks(struct sha512_block_state *stat= e, { if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && static_branch_likely(&have_neon) && likely(crypto_simd_usable())) { - kernel_neon_begin(); - sha512_block_data_order_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + sha512_block_data_order_neon(state, data, nblocks); } else { sha512_block_data_order(state, data, nblocks); } diff --git a/lib/crypto/arm64/chacha-neon-glue.c b/lib/crypto/arm64/chacha-= neon-glue.c index d0188f974ca5..a3d109f0ce1e 100644 --- a/lib/crypto/arm64/chacha-neon-glue.c +++ b/lib/crypto/arm64/chacha-neon-glue.c @@ -25,7 +25,6 @@ #include =20 #include -#include #include =20 asmlinkage void chacha_block_xor_neon(const struct chacha_state *state, @@ -67,9 +66,8 @@ void hchacha_block_arch(const struct chacha_state *state, if (!static_branch_likely(&have_neon) || !crypto_simd_usable()) { hchacha_block_generic(state, out, nrounds); } else { - kernel_neon_begin(); - hchacha_block_neon(state, out, nrounds); - kernel_neon_end(); + scoped_ksimd() + hchacha_block_neon(state, out, nrounds); } } EXPORT_SYMBOL(hchacha_block_arch); @@ -84,9 +82,8 @@ void chacha_crypt_arch(struct chacha_state *state, u8 *ds= t, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, bytes, SZ_4K); =20 - kernel_neon_begin(); - chacha_doneon(state, dst, src, todo, nrounds); - kernel_neon_end(); + scoped_ksimd() + chacha_doneon(state, dst, src, todo, nrounds); =20 bytes -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm64/poly1305-glue.c b/lib/crypto/arm64/poly1305-g= lue.c index 31aea21ce42f..c83ce7d835d9 100644 --- a/lib/crypto/arm64/poly1305-glue.c +++ b/lib/crypto/arm64/poly1305-glue.c @@ -6,7 +6,6 @@ */ =20 #include -#include #include #include #include @@ -38,9 +37,8 @@ void poly1305_blocks_arch(struct poly1305_block_state *st= ate, const u8 *src, do { unsigned int todo =3D min_t(unsigned int, len, SZ_4K); =20 - kernel_neon_begin(); - poly1305_blocks_neon(state, src, todo, padbit); - kernel_neon_end(); + scoped_ksimd() + poly1305_blocks_neon(state, src, todo, padbit); =20 len -=3D todo; src +=3D todo; diff --git a/lib/crypto/arm64/sha1.h b/lib/crypto/arm64/sha1.h index f822563538cc..3d0da0045fed 100644 --- a/lib/crypto/arm64/sha1.h +++ b/lib/crypto/arm64/sha1.h @@ -4,7 +4,6 @@ * * Copyright 2025 Google LLC */ -#include #include #include =20 @@ -20,9 +19,9 @@ static void sha1_blocks(struct sha1_block_state *state, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha1_ce_transform(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha1_ce_transform(state, data, nblocks); + data +=3D (nblocks - rem) * SHA1_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); diff --git a/lib/crypto/arm64/sha256.h b/lib/crypto/arm64/sha256.h index a211966c124a..0a9f9d70bb43 100644 --- a/lib/crypto/arm64/sha256.h +++ b/lib/crypto/arm64/sha256.h @@ -4,7 +4,7 @@ * * Copyright 2025 Google LLC */ -#include +#include #include #include =20 @@ -27,17 +27,16 @@ static void sha256_blocks(struct sha256_block_state *st= ate, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha256_ce_transform(state, - data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha256_ce_transform(state, data, + nblocks); + data +=3D (nblocks - rem) * SHA256_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); } else { - kernel_neon_begin(); - sha256_block_neon(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + sha256_block_neon(state, data, nblocks); } } else { sha256_block_data_order(state, data, nblocks); diff --git a/lib/crypto/arm64/sha512.h b/lib/crypto/arm64/sha512.h index 6abb40b467f2..1b6c3974d553 100644 --- a/lib/crypto/arm64/sha512.h +++ b/lib/crypto/arm64/sha512.h @@ -5,7 +5,7 @@ * Copyright 2025 Google LLC */ =20 -#include +#include #include #include =20 @@ -25,9 +25,9 @@ static void sha512_blocks(struct sha512_block_state *stat= e, do { size_t rem; =20 - kernel_neon_begin(); - rem =3D __sha512_ce_transform(state, data, nblocks); - kernel_neon_end(); + scoped_ksimd() + rem =3D __sha512_ce_transform(state, data, nblocks); + data +=3D (nblocks - rem) * SHA512_BLOCK_SIZE; nblocks =3D rem; } while (nblocks); --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77929311C1D for ; Wed, 1 Oct 2025 21:04:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352649; cv=none; b=jrLWJdDYDqQkT15flzLxxHqj+8gA4acCM+jQ5k4vozRVYGQ2UiSctsPcb4W+eJuD6tntD5TF4kdLnWCFukgLwMnarf1AfKT0+I5FDqiApG8VEOEPV4ZjBgMGfjQYMkxuR2MeILz8pZPruicSzIYH042ejPZO8Ljcy2X+2Lekraw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352649; c=relaxed/simple; bh=dDKP7cf/IwxD7BPlX7fXIXQ/LuMlM/dY4qugXMO1+Ks=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YzM+7+6qwhsszyI2rAcfETMU12SfTqPFo4REvA5kINB3ogS3MeeVGTuO4c1hRqD6uBgBv32UjzgzjeqXbpDbNIg3Swwlzm1azqLrhafvxNsyQQ9akuHc/g4DcK+QMzwSNEGO/EDRh1BnnzJs/au0dF5NDeSUoMQ1iSR9g8h5JEo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bpLStXfw; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bpLStXfw" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-40cfb98eddbso623394f8f.0 for ; Wed, 01 Oct 2025 14:04:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352644; x=1759957444; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qcvrL2whjEnKHntdPNXOlg2YOLj5Bio52ZSIjpgqpWY=; b=bpLStXfwP631JVblTXBDL3zxuO7JksO9pzhHiigI1ZXoxTprmub6ZgxdWEikCfYIXN RSctJcNcL1v9BLAolzr8ztMh3zzCau0BOLUIMXeqDLlDe3Q7+Iy4MsacRyGPat7KoJUo Iys78XhKxD1ej8qn7AFcvM/+nc1w37aiGK4GyuFN6E+Khl3vw+yKa6F/U0Mz+3gC1zV/ hI94lYewB1Mmb53BPonHO5WNfxSVnJux6/t3H3RMjouUDiXvVq+sfuchdeWUr90EhSzS aFJ2UDMZJ4bWw8lU9qgXJFT9d/EPNJTCDSUgRPqvbJTBeMoJ6AH26PQOdbC+0S5LI7BL BhNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352644; x=1759957444; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qcvrL2whjEnKHntdPNXOlg2YOLj5Bio52ZSIjpgqpWY=; b=lmy+6JLc7fX9rEu7I6lL0tmMN88r+2WPBiUb5ABd8cg7PdnzikWIvtxPuWiy249OCr H6sxWjRmm/V9YFNecndzeymUONr2dqOXjAXPoidtWLLAw7DWo+XUVBarhKflgvRXLphe y0YvK9xicCTy2rixumRWzJZ3lywh+2qH1tYXkq7WCUjkgfKVJ+1A/UFt4FJRN2G+HOUr 9iMCkvnq4myL3UJNPygN9eT/vG4mv8CE4XKhGHStak1MVaJ/HZwHAyQFVd4XKkqSPPUA K88dTGaH3U4jEuPT8UFG0cPOWJwOIaozYwsApxqWp1QCuaFuIFMMjuOMM0tGlO33brh7 Bx7A== X-Forwarded-Encrypted: i=1; AJvYcCU4ZAR4dsfgJ/BYdiFU42TYx7tOZr1PKcaWhcF+l0YN5qMEXMUvAsRzVZm1KkGkKoDl2YZxqknJ3F2ML00=@vger.kernel.org X-Gm-Message-State: AOJu0YzrdSOtSat6o1xaq13d1P6YI+kAUXSa1VUO/o1DA6dj/6KIC14m Y/Rd2KINTp3hXddtnUJXStqyYZoSPQomtxKVThilgR78ZG2HPXsCm1QrTCsEc4XFaO07sTQ6Dw= = X-Google-Smtp-Source: AGHT+IFgnronrhyrUGQbEd1tw99/RmLAuopJxtgZuzG70+rn7tFXZp9Z4jQhbFBg2A3csmpX/luoY+KJ X-Received: from wmep20.prod.google.com ([2002:a05:600c:4314:b0:46e:3212:7c8f]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:584a:0:b0:425:58d0:483a with SMTP id ffacd0b85a97d-4255d294cb7mr680156f8f.3.1759352643936; Wed, 01 Oct 2025 14:04:03 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:13 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4507; i=ardb@kernel.org; h=from:subject; bh=ZZLbTVVt22jlbrdSe9T5dbgBqhdIsvid01EOnLK51YY=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutB+MjZ87zJ8nF8qUiPf+vhsS3eN260KLvOGy28p7j a8kHH/eUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbCVMfwP1b11I7kc0k/2j7z 3BdgviGzfdeviMjbLzzm+r6qz/+9zJ+RYb7Fzy6L4LN3/7zofmkyK15hq9g6/nWafRsuVhbyX/9 7kQEA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-33-ardb+git@google.com> Subject: [PATCH v2 11/20] crypto/arm64: aes-ccm - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 135 ++++++++++---------- 1 file changed, 66 insertions(+), 69 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce= -ccm-glue.c index 2eb4e76cabc3..c4fd648471f1 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -8,7 +8,6 @@ * Author: Ard Biesheuvel */ =20 -#include #include #include #include @@ -16,6 +15,8 @@ #include #include =20 +#include + #include "aes-ce-setkey.h" =20 MODULE_IMPORT_NS("CRYPTO_INTERNAL"); @@ -184,40 +185,38 @@ static int ccm_encrypt(struct aead_request *req) if (unlikely(err)) return err; =20 - kernel_neon_begin(); - - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - do { - u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; - const u8 *src =3D walk.src.virt.addr; - u8 *dst =3D walk.dst.virt.addr; - u8 buf[AES_BLOCK_SIZE]; - u8 *final_iv =3D NULL; - - if (walk.nbytes =3D=3D walk.total) { - tail =3D 0; - final_iv =3D orig_iv; - } - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], - src, walk.nbytes); - - ce_aes_ccm_encrypt(dst, src, walk.nbytes - tail, - ctx->key_enc, num_rounds(ctx), - mac, walk.iv, final_iv); - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, dst, walk.nbytes); - - if (walk.nbytes) { - err =3D skcipher_walk_done(&walk, tail); - } - } while (walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { + u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; + const u8 *src =3D walk.src.virt.addr; + u8 *dst =3D walk.dst.virt.addr; + u8 buf[AES_BLOCK_SIZE]; + u8 *final_iv =3D NULL; + + if (walk.nbytes =3D=3D walk.total) { + tail =3D 0; + final_iv =3D orig_iv; + } + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], + src, walk.nbytes); + + ce_aes_ccm_encrypt(dst, src, walk.nbytes - tail, + ctx->key_enc, num_rounds(ctx), + mac, walk.iv, final_iv); + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, dst, walk.nbytes); + + if (walk.nbytes) { + err =3D skcipher_walk_done(&walk, tail); + } + } while (walk.nbytes); + } =20 if (unlikely(err)) return err; @@ -251,40 +250,38 @@ static int ccm_decrypt(struct aead_request *req) if (unlikely(err)) return err; =20 - kernel_neon_begin(); - - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - do { - u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; - const u8 *src =3D walk.src.virt.addr; - u8 *dst =3D walk.dst.virt.addr; - u8 buf[AES_BLOCK_SIZE]; - u8 *final_iv =3D NULL; - - if (walk.nbytes =3D=3D walk.total) { - tail =3D 0; - final_iv =3D orig_iv; - } - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], - src, walk.nbytes); - - ce_aes_ccm_decrypt(dst, src, walk.nbytes - tail, - ctx->key_enc, num_rounds(ctx), - mac, walk.iv, final_iv); - - if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) - memcpy(walk.dst.virt.addr, dst, walk.nbytes); - - if (walk.nbytes) { - err =3D skcipher_walk_done(&walk, tail); - } - } while (walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); + + do { + u32 tail =3D walk.nbytes % AES_BLOCK_SIZE; + const u8 *src =3D walk.src.virt.addr; + u8 *dst =3D walk.dst.virt.addr; + u8 buf[AES_BLOCK_SIZE]; + u8 *final_iv =3D NULL; + + if (walk.nbytes =3D=3D walk.total) { + tail =3D 0; + final_iv =3D orig_iv; + } + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(&buf[sizeof(buf) - walk.nbytes], + src, walk.nbytes); + + ce_aes_ccm_decrypt(dst, src, walk.nbytes - tail, + ctx->key_enc, num_rounds(ctx), + mac, walk.iv, final_iv); + + if (unlikely(walk.nbytes < AES_BLOCK_SIZE)) + memcpy(walk.dst.virt.addr, dst, walk.nbytes); + + if (walk.nbytes) { + err =3D skcipher_walk_done(&walk, tail); + } + } while (walk.nbytes); + } =20 if (unlikely(err)) return err; --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CB26311C2C for ; Wed, 1 Oct 2025 21:04:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352650; cv=none; b=hR0slCpuLubvpH0RzmRPlkP/D6PA5zR1Vzabloam/ANH8yLi0GVlEBNCbZeTCInfXMk6XZaa06x5DLUoIfDkTcXTj/vLAQWQ1HeRCpkdFPdom0fnTJh16PVZSGSQe3LjdtU2zj5A4vzzApvCuE6n0JOqHlyHn+htALmj1xc3y34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352650; c=relaxed/simple; bh=FsmKCKW0+3JBks3b7w6jj2RyXifJfqC/SFYqHtUwJ78=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g51wJJYdVEOHnBfrfwWflvYug9CWWn08NL501Ocye1wrMIBfGR2NgKQwJctycyCs2ozH8IYrB5oaJZ3z4wPNU0V1q9lxEgHGbxIcIFGBUk1pWG5Mq0MB/8vlF68HdFE1t22NtXoBWggATIsOnzZTd4oAfQEu8c9rAgsahgV4+xE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OIyTiwxX; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OIyTiwxX" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b3d525fb67eso31702466b.0 for ; Wed, 01 Oct 2025 14:04:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352645; x=1759957445; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2tGgNTZuDE42TuUV9n3Q45jrq8v49nbNgvB3NUvZDpE=; b=OIyTiwxXJqYc3bc9z1xzDbAx7FM9nAkzRPHyDtJIRdGq6ztkqjlpNChexJ8dz97+EZ xI6n+oe6pP2IjyjxwMAXY77W/LFD9T1rm5WosyZU8zKnkCfLAkE8YTMvUJxm07U6pVl+ Elkv4P3Ho8v4reQSSvv6b/sKTUOt+nqnYnualvVbdB/JIiKEFrbh6ff33OjmgtRQqvay ndyT0L4MAnrS8cvXIatnzQYAhtRiRsgul2ZVaIngLnZTr0fNCQwASQ5pjvaHwiTKWjd+ qKO05GNsTKkvpyIyhtkpNCYSCo/g8vMxjlNntRHhHXh93ewQs0pnso2BBpaVxaWfC7iy +pMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352645; x=1759957445; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2tGgNTZuDE42TuUV9n3Q45jrq8v49nbNgvB3NUvZDpE=; b=pHXMx+5zhvG89Zn5uxN6euC48wK5aSb0fhf8K8LOBP1Y50Pa4T4Y+V76TeEunowoTS 9hGrPlWwSFND2MbNJan2DwqAhMfmA3uKxQQ/JCe8uWlBPwh/gyt/QlnjqLl5AQ0OoYlr IKIV4nF/SjWEu4l33b8FxZ+iBYEUTbcf/+nOSZCgKFe3UVApxwjzZDOgOCuVhghk/6UU YjZj00r9N7vQQXKfrkqqX6lj++8igLmTFOSVdkIW9y5otcUbfd5WgYNUuZiZHjqSAxzE qUsVD/ma5UJuOnoqvPrX7Eh+oByD4/jbiGI+FVL5+rORE05gj4hMW41jxi+vwnesf5MO 7mnA== X-Forwarded-Encrypted: i=1; AJvYcCUB/wHUEdKRq7HxrGS9MJa+vaXZdu23o/BEwLcYr6s6Slv0/5U2nSltcysTyGbWkF4Kzn1QexPUUsl4s0U=@vger.kernel.org X-Gm-Message-State: AOJu0YyK3RXJK4PwWRjaKnindq4EK6KfMg0DhvAqZuE0FjKk3FFiVl0e dtNT+F7yqWuRVyQEMid4KoMdgr+1zn6D7J5HF1tQZsgSQ9rNAtiBjXxwwypSEOBdXMmgsGD3pQ= = X-Google-Smtp-Source: AGHT+IGRMPSEJRGhX6bzJNMLIpuTgPU8l+6UbKzu27Pb1Dg+NmBxLA1o3IMc5SIrwprjppFeYEZeOCGP X-Received: from edvd18.prod.google.com ([2002:aa7:ce12:0:b0:633:c29c:e02b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:1c91:b0:b45:b1f2:fab4 with SMTP id a640c23a62f3a-b46e4d7bd97mr600170166b.12.1759352644910; Wed, 01 Oct 2025 14:04:04 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:14 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=20375; i=ardb@kernel.org; h=from:subject; bh=SDCOXpC6hIQ3gVfCMmGTmKQTQWVfl68LHJ4l6pbLEnM=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutL8d7XOnC9eGvfaNMHSbsXBGcCnntPo0b63wLwlpc 3qCixs6SlkYxLgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwETC1jAyTPwwP7vSzti9yfX9 Wf2bOtVNSdKrvtx8fV3Y+Jee5Ona+4wMx/oL2dSL8jZpOfsv57HlZ/N50jJtZ/XSnvK5sTf8bc1 5AA== X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-34-ardb+git@google.com> Subject: [PATCH v2 12/20] crypto/arm64: aes-blk - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-glue.c | 87 ++++++------ arch/arm64/crypto/aes-glue.c | 139 ++++++++---------- arch/arm64/crypto/aes-neonbs-glue.c | 150 ++++++++++---------- 3 files changed, 181 insertions(+), 195 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glu= e.c index 00b8749013c5..a4dad370991d 100644 --- a/arch/arm64/crypto/aes-ce-glue.c +++ b/arch/arm64/crypto/aes-ce-glue.c @@ -52,9 +52,8 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); } =20 static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const = src[]) @@ -66,9 +65,8 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); } =20 int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, @@ -94,47 +92,48 @@ int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const = u8 *in_key, for (i =3D 0; i < kwords; i++) ctx->key_enc[i] =3D get_unaligned_le32(in_key + i * sizeof(u32)); =20 - kernel_neon_begin(); - for (i =3D 0; i < sizeof(rcon); i++) { - u32 *rki =3D ctx->key_enc + (i * kwords); - u32 *rko =3D rki + kwords; - - rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ rcon[i] ^ rki[0]; - rko[1] =3D rko[0] ^ rki[1]; - rko[2] =3D rko[1] ^ rki[2]; - rko[3] =3D rko[2] ^ rki[3]; - - if (key_len =3D=3D AES_KEYSIZE_192) { - if (i >=3D 7) - break; - rko[4] =3D rko[3] ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - } else if (key_len =3D=3D AES_KEYSIZE_256) { - if (i >=3D 6) - break; - rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - rko[6] =3D rko[5] ^ rki[6]; - rko[7] =3D rko[6] ^ rki[7]; + scoped_ksimd() { + for (i =3D 0; i < sizeof(rcon); i++) { + u32 *rki =3D ctx->key_enc + (i * kwords); + u32 *rko =3D rki + kwords; + + rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ + rcon[i] ^ rki[0]; + rko[1] =3D rko[0] ^ rki[1]; + rko[2] =3D rko[1] ^ rki[2]; + rko[3] =3D rko[2] ^ rki[3]; + + if (key_len =3D=3D AES_KEYSIZE_192) { + if (i >=3D 7) + break; + rko[4] =3D rko[3] ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + } else if (key_len =3D=3D AES_KEYSIZE_256) { + if (i >=3D 6) + break; + rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + rko[6] =3D rko[5] ^ rki[6]; + rko[7] =3D rko[6] ^ rki[7]; + } } - } =20 - /* - * Generate the decryption keys for the Equivalent Inverse Cipher. - * This involves reversing the order of the round keys, and applying - * the Inverse Mix Columns transformation on all but the first and - * the last one. - */ - key_enc =3D (struct aes_block *)ctx->key_enc; - key_dec =3D (struct aes_block *)ctx->key_dec; - j =3D num_rounds(ctx); - - key_dec[0] =3D key_enc[j]; - for (i =3D 1, j--; j > 0; i++, j--) - __aes_ce_invert(key_dec + i, key_enc + j); - key_dec[i] =3D key_enc[0]; + /* + * Generate the decryption keys for the Equivalent Inverse + * Cipher. This involves reversing the order of the round + * keys, and applying the Inverse Mix Columns transformation on + * all but the first and the last one. + */ + key_enc =3D (struct aes_block *)ctx->key_enc; + key_dec =3D (struct aes_block *)ctx->key_dec; + j =3D num_rounds(ctx); + + key_dec[0] =3D key_enc[j]; + for (i =3D 1, j--; j > 0; i++, j--) + __aes_ce_invert(key_dec + i, key_enc + j); + key_dec[i] =3D key_enc[0]; + } =20 - kernel_neon_end(); return 0; } EXPORT_SYMBOL(ce_aes_expandkey); diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 81560f722b9d..4b76d8dfa2c5 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -5,8 +5,6 @@ * Copyright (C) 2013 - 2017 Linaro Ltd */ =20 -#include -#include #include #include #include @@ -20,6 +18,9 @@ #include #include =20 +#include +#include + #include "aes-ce-setkey.h" =20 #ifdef USE_V8_CRYPTO_EXTENSIONS @@ -187,10 +188,9 @@ static int __maybe_unused ecb_encrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -207,10 +207,9 @@ static int __maybe_unused ecb_decrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -225,10 +224,9 @@ static int cbc_encrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_enc, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_enc, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -254,10 +252,9 @@ static int cbc_decrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_dec, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_dec, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -323,10 +320,9 @@ static int cts_cbc_encrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -380,10 +376,9 @@ static int cts_cbc_decrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -416,11 +411,11 @@ static int __maybe_unused essiv_cbc_encrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_enc, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_encrypt_walk(req, &walk); @@ -438,11 +433,11 @@ static int __maybe_unused essiv_cbc_decrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_decrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_dec, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_decrypt_walk(req, &walk); @@ -478,10 +473,9 @@ static int __maybe_unused xctr_encrypt(struct skcipher= _request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv, byte_ctr); - kernel_neon_end(); + scoped_ksimd() + aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv, byte_ctr); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -523,10 +517,9 @@ static int __maybe_unused ctr_encrypt(struct skcipher_= request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -579,11 +572,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -601,11 +593,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (err) return err; =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -651,11 +642,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -674,11 +664,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) return err; =20 =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -827,10 +816,9 @@ static int cmac_setkey(struct crypto_shash *tfm, const= u8 *in_key, return err; =20 /* encrypt the zero vector */ - kernel_neon_begin(); - aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, ctx->key.key_enc, - rounds, 1); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, + ctx->key.key_enc, rounds, 1); =20 cmac_gf128_mul_by_x(consts, consts); cmac_gf128_mul_by_x(consts + 1, consts); @@ -856,10 +844,10 @@ static int xcbc_setkey(struct crypto_shash *tfm, cons= t u8 *in_key, if (err) return err; =20 - kernel_neon_begin(); - aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); - aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); - kernel_neon_end(); + scoped_ksimd() { + aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); + aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); + } =20 return cbcmac_setkey(tfm, key, sizeof(key)); } @@ -879,10 +867,9 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, = u8 const in[], int blocks, int rem; =20 do { - kernel_neon_begin(); - rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, - dg, enc_before, !enc_before); - kernel_neon_end(); + scoped_ksimd() + rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, + dg, enc_before, !enc_before); in +=3D (blocks - rem) * AES_BLOCK_SIZE; blocks =3D rem; } while (blocks); diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-ne= onbs-glue.c index c4a623e86593..d496effb0a5b 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -85,9 +85,8 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, cons= t u8 *in_key, =20 ctx->rounds =3D 6 + key_len / 4; =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); =20 return 0; } @@ -110,10 +109,9 @@ static int __ecb_crypt(struct skcipher_request *req, blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, - ctx->rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, + ctx->rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -146,9 +144,8 @@ static int aesbs_cbc_ctr_setkey(struct crypto_skcipher = *tfm, const u8 *in_key, =20 memcpy(ctx->enc, rk.key_enc, sizeof(ctx->enc)); =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); memzero_explicit(&rk, sizeof(rk)); =20 return 0; @@ -167,11 +164,11 @@ static int cbc_encrypt(struct skcipher_request *req) unsigned int blocks =3D walk.nbytes / AES_BLOCK_SIZE; =20 /* fall back to the non-bitsliced NEON implementation */ - kernel_neon_begin(); - neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->enc, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + neon_aes_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->enc, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -193,11 +190,10 @@ static int cbc_decrypt(struct skcipher_request *req) blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -220,30 +216,32 @@ static int ctr_encrypt(struct skcipher_request *req) const u8 *src =3D walk.src.virt.addr; u8 *dst =3D walk.dst.virt.addr; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - aesbs_ctr_encrypt(dst, src, ctx->key.rk, ctx->key.rounds, - blocks, walk.iv); - dst +=3D blocks * AES_BLOCK_SIZE; - src +=3D blocks * AES_BLOCK_SIZE; - } - if (nbytes && walk.nbytes =3D=3D walk.total) { - u8 buf[AES_BLOCK_SIZE]; - u8 *d =3D dst; - - if (unlikely(nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(buf + sizeof(buf) - nbytes, - src, nbytes); - - neon_aes_ctr_encrypt(dst, src, ctx->enc, ctx->key.rounds, - nbytes, walk.iv); + scoped_ksimd() { + if (blocks >=3D 8) { + aesbs_ctr_encrypt(dst, src, ctx->key.rk, + ctx->key.rounds, blocks, + walk.iv); + dst +=3D blocks * AES_BLOCK_SIZE; + src +=3D blocks * AES_BLOCK_SIZE; + } + if (nbytes && walk.nbytes =3D=3D walk.total) { + u8 buf[AES_BLOCK_SIZE]; + u8 *d =3D dst; + + if (unlikely(nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(buf + sizeof(buf) - + nbytes, src, nbytes); + + neon_aes_ctr_encrypt(dst, src, ctx->enc, + ctx->key.rounds, nbytes, + walk.iv); =20 - if (unlikely(nbytes < AES_BLOCK_SIZE)) - memcpy(d, dst, nbytes); + if (unlikely(nbytes < AES_BLOCK_SIZE)) + memcpy(d, dst, nbytes); =20 - nbytes =3D 0; + nbytes =3D 0; + } } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } return err; @@ -320,33 +318,33 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - if (first =3D=3D 1) - neon_aes_ecb_encrypt(walk.iv, walk.iv, - ctx->twkey, - ctx->key.rounds, 1); - first =3D 2; - - fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - - out +=3D blocks * AES_BLOCK_SIZE; - in +=3D blocks * AES_BLOCK_SIZE; - nbytes -=3D blocks * AES_BLOCK_SIZE; + scoped_ksimd() { + if (blocks >=3D 8) { + if (first =3D=3D 1) + neon_aes_ecb_encrypt(walk.iv, walk.iv, + ctx->twkey, + ctx->key.rounds, 1); + first =3D 2; + + fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); + + out +=3D blocks * AES_BLOCK_SIZE; + in +=3D blocks * AES_BLOCK_SIZE; + nbytes -=3D blocks * AES_BLOCK_SIZE; + } + if (walk.nbytes =3D=3D walk.total && nbytes > 0) { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + nbytes =3D first =3D 0; + } } - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - nbytes =3D first =3D 0; - } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -369,14 +367,16 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + } =20 return skcipher_walk_done(&walk, 0); } --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A66A311C2A for ; Wed, 1 Oct 2025 21:04:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352649; cv=none; b=dkwOtm9KojAErKNTYChvyOGGcKSmtAvW2t7WmSoukp7LFIqAFTEqmTVrTx38aIttQbyREg0eftTfzQAZCaJzcsjt1Xtj8acD3olTmh/Dd+VYdUaK83+J8OKz3o32fSxIYHZClgTMmXr46VKHEquFZt4iXO1iJJVhldQ+fr78Gws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352649; c=relaxed/simple; bh=/ZA4JA9wV0R82IktuAfOLSwLAhmDtcCLaF/KC91msqY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QVLg3vqtqrIdY0Rg+VXx7Fboalm01s5Ag4ghh9cRRWzvZxEFrFaPPJObAOLSRY1Hc2GwW9df0gpPl8/VKLXrFanzok91HlhnJu494kVeGZ21JilB/tuUZOSc9XAnOgh0/Cf89qZi4ktJSSeUt3VNUx9tjF56xDKhqLrM1wPOiDY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=B1wQqj5/; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="B1wQqj5/" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3f6b44ab789so86958f8f.3 for ; Wed, 01 Oct 2025 14:04:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352646; x=1759957446; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CeDKPAQMPbh0EqMW8jQOA9NzBOuPEFyE2AHCW54BErk=; b=B1wQqj5/UTdR+oD0PLp67YlrZ5w8PQQuxF1S4WzGFQxzQ+WEmJlteFaJFOV/Dyy+k6 fkOCNcFEKxhf3nH1Im7WdWgtj7pFtrtaRGxAQSdgV+g8Sj26V1eyUDRpRzoIqt3HmbkB IaWpK+TxNTDCaSSv6Lxr6w5Tk320dcxHCX7SGmBc96gYljKg5admD/LUvOMG98qF3efW JK4qIMWP8Rb251oFKR5fHSewbsKn0KCpOSEaOt8IUdfbIFibkigbKC0HhpkdGJMHxxeT FVm1Jc/OO5vdfbsaSuEb8GqZcahuKC8LxqKN2i5VEfoYbq5JWPVXIa8xxHborHw/9nzw Ny1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352646; x=1759957446; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CeDKPAQMPbh0EqMW8jQOA9NzBOuPEFyE2AHCW54BErk=; b=QtlOxsT0LsIK0SYiAOepjXDllDuHlx5cPwnzwsIxPQAJMmEo2Lkkd/QIIrHHfVePrB pohamMwAFJWl2t5bIb5glqzQVWHK33OU0MsyPg36vBXaS7IzgEafgyTX78jgHTatFAzT L/YpwrkdW74ggRzEaxd+AjcxxRCM0kSQSjfadLLhbAcSz3ol4JYrY5PngZdBUOHO36mf UWlc0/y6Z1aC11/ur5K5oszfUSd/v6te4DtBnSj6C1wZqImgIcx5p4KsGqx/C63zeb/K MlFK4okRwsArvZ4y3mHGO83vjMSUaxWLllBDa59582ywLlDx0tHFagkAqxP+W9v2TFhX Fr/w== X-Forwarded-Encrypted: i=1; AJvYcCW+K62Qp2jtdWa6v3dX8Jv2FuE1vnwNdp+Zr0YEzTpIOoG3uLihCcSJcIPGlI298L7IlRYDIKEblCajF44=@vger.kernel.org X-Gm-Message-State: AOJu0YxN66AvVOW34JGw0H3jMCa8K0L+tghYpG6FgUEa6B2grFOloz0S 8Gud8NFneBeofTxzvY5YQIDxh2SYMn4I7l61S4WuOMDIv0KaA1ZLjDu719jaIeI/WdIgoVEJHw= = X-Google-Smtp-Source: AGHT+IFzputzbAebEFyEfc/jsFLhZ5Ij21AZlBRWh2WXnyRCr1Ir7HRHLQr/BaUei+1rcPYe79TOPRsG X-Received: from wmbh26.prod.google.com ([2002:a05:600c:a11a:b0:45d:cfa4:ce0d]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2408:b0:3f0:2ab8:710f with SMTP id ffacd0b85a97d-425577ed732mr2808941f8f.8.1759352646016; Wed, 01 Oct 2025 14:04:06 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:15 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2238; i=ardb@kernel.org; h=from:subject; bh=6vnaDpO87kSFyERwmfNjHVSsNUwe+WjtwCLCIOuhD2k=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudMabfzoiq30TZq93WPL622O99VO1Yg0XZ+6/KH3hp c5lmzjbjlIWBjEuBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjARP1+G/wWmxxe9Ef5yUDnC 4UG55DG/L0npufYTWHPy3JZ7+Hvvf8TwT62otuLghCfsBRG+pqa91dvetu9/w7vik47Kx3/8xvz pjAA= X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-35-ardb+git@google.com> Subject: [PATCH v2 13/20] crypto/arm64: aes-gcm - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/ghash-ce-glue.c | 27 ++++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce= -glue.c index 4995b6e22335..7951557a285a 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -5,7 +5,6 @@ * Copyright (C) 2014 - 2018 Linaro Ltd. */ =20 -#include #include #include #include @@ -22,6 +21,8 @@ #include #include =20 +#include + MODULE_DESCRIPTION("GHASH and AES-GCM using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -74,9 +75,8 @@ void ghash_do_simd_update(int blocks, u64 dg[], const cha= r *src, u64 const h[][2], const char *head)) { - kernel_neon_begin(); - simd_update(blocks, dg, src, key->h, head); - kernel_neon_end(); + scoped_ksimd() + simd_update(blocks, dg, src, key->h, head); } =20 /* avoid hogging the CPU for too long */ @@ -329,11 +329,10 @@ static int gcm_encrypt(struct aead_request *req, char= *iv, int assoclen) tag =3D NULL; } =20 - kernel_neon_begin(); - pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, nrounds, - tag); - kernel_neon_end(); + scoped_ksimd() + pmull_gcm_encrypt(nbytes, dst, src, ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, nrounds, + tag); =20 if (unlikely(!nbytes)) break; @@ -399,11 +398,11 @@ static int gcm_decrypt(struct aead_request *req, char= *iv, int assoclen) tag =3D NULL; } =20 - kernel_neon_begin(); - ret =3D pmull_gcm_decrypt(nbytes, dst, src, ctx->ghash_key.h, - dg, iv, ctx->aes_key.key_enc, - nrounds, tag, otag, authsize); - kernel_neon_end(); + scoped_ksimd() + ret =3D pmull_gcm_decrypt(nbytes, dst, src, + ctx->ghash_key.h, + dg, iv, ctx->aes_key.key_enc, + nrounds, tag, otag, authsize); =20 if (unlikely(!nbytes)) break; --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07B04311C3F for ; Wed, 1 Oct 2025 21:04:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352650; cv=none; b=k3iT1aTUYiNyck1Mdd7Ri+rHvTERxESzXR22KNmdsunML5KZBZFHJWJP5r9bw25u/nlIgTNETC+RLueWx/3b3U35YpdExnmC0kP+ystY+W87uU4fOvl7NUvrh4WorEiNDIKUUX0QrXULiopO+aIYdU1gkWJ8rZX8QpCwlgaO5yM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352650; c=relaxed/simple; bh=SW5mPnOzJITL4l0UqpBY1hKBImd1TIHcagRsxcO8Ux8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aen8wSOG4n0GYyCUzYr7y7AabXHTX4QafpvKnxm7M13PzXRrKblQ3X5chdPmMoBaPEbhIELnj+nDqNOZvta5OgYdGZSf6eY6zLj1RvQhojHyusJwjhly9SacdwDgZaqiNSonXpyHxGYM4+dNOCaPbkFSErDTJ7zAPX/zgJ5gz8Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=oa2dFGNh; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="oa2dFGNh" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e4943d713so1254735e9.1 for ; Wed, 01 Oct 2025 14:04:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352647; x=1759957447; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UZKSd5rMdo9tEwoYCoXUQVE6sJT0wmf8X0NATCF37Q4=; b=oa2dFGNhmcvvyhoyP1Si3mFhku55TBbhfKRm+E+pLjSs3LMCkvsVHsx2WfcGA+Ebfo ESQRvHYo0E7mvYGv2xxNa7gYa8/qEWhrDYjSaTvWKfgl5twBdfkXI8atbyr6aOuSpK9i HkARuRCIFJ7UXK/dlTylw4h9LMjzNMoXuvhY5ho4Jclg/HPbeNbXfpxRjXscKixI0blu OtXCMqILnmk6DJ3elccYb4cWGYtykwHGkh7dEr1R4tg/yxBrWVcEq96RzVd8LWJsxS8w 1jrAqhAalr4KqmbEY9u9ZmAYJKu1Dd14oS5MX+zwAy18eb78mUuS27UAqfKcH9kvndEV ZCig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352647; x=1759957447; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UZKSd5rMdo9tEwoYCoXUQVE6sJT0wmf8X0NATCF37Q4=; b=TkDrzvQL4PgeAMlsQ+JBX7Uwiwb86LRXk/slE2NQgs3f/FaPjZx3uIETqan9rXSQX4 IYPr5XLtvrb2w5R/AAj4lg6644+m0QfR46YItao4o0mhzGkKFxvBZvSBo90OaICMzzC0 1E8CYb0t+yPmMuOzt/XY4rSiSFZVWmrWQrxgqMY6M8bN3QZwwD6XYv9ePDqCwWHnvL/C TO973xLMTHFKSsBYtaEFSLlxGXiPuJJ8/6a8tQPD0NI4Fs5P8IV4OrtBkTHBGs7oHisO G9Jp2mqu9wg5DBS/oFQhqThs6oYvvhAODZDwjjiXg9rxYJBERgzyVDQZDH6oreM+ajQt O2hw== X-Forwarded-Encrypted: i=1; AJvYcCXpkDCKZlXZkb7aVZCKh3JBZib/3Yx5kRRcJajn1fFd507hBIu0Jrih7A+oKtMw2AkWCW5SVnWEifhQ1ZM=@vger.kernel.org X-Gm-Message-State: AOJu0YyrbbK1xb/z9cV7W7kF2zPVzG1R7EmVwQP3qaYZuprH8FD53Odj ofhoUyBMDvq0JLiCXlabRFGixu6EOJTAhcBSpB+pqHrgOiC2PKnq7XpThwjJ1m9V1YrG0cgfcg= = X-Google-Smtp-Source: AGHT+IEYxGxSDoioNYClblaAj4vxDm8fokdAPOdazIOTblzm8QXHXPeKE11+MRENrVgxIsmYT+ZQ2SYk X-Received: from wmjs2.prod.google.com ([2002:a7b:c382:0:b0:46e:3422:1fcc]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4fc6:b0:46e:1a07:7bd5 with SMTP id 5b1f17b1804b1-46e61285d7dmr38854685e9.29.1759352647355; Wed, 01 Oct 2025 14:04:07 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:16 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=814; i=ardb@kernel.org; h=from:subject; bh=40tSyXfxoCO4pZAvlNOPxZCROdcuajmQXIsSsXG3TMo=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudNZjToIdV9WeFedtulX01C0kb9ZiTfM1ap1FWzetS rNz+Liho5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzkSAsjw7scuSlTyt+Fs1z8 9Tm6LUE4x8+7eQ0f03c1h6unY6tVFjD8FeRM44laGepRWKlwYnJBPce7w3/PGx652jDr06pfoou aeQE= X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-36-ardb+git@google.com> Subject: [PATCH v2 14/20] crypto/arm64: nhpoly1305 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/nhpoly1305-neon-glue.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/n= hpoly1305-neon-glue.c index e4a0b463f080..013de6ac569a 100644 --- a/arch/arm64/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c @@ -25,9 +25,8 @@ static int nhpoly1305_neon_update(struct shash_desc *desc, do { unsigned int n =3D min_t(unsigned int, srclen, SZ_4K); =20 - kernel_neon_begin(); - crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); - kernel_neon_end(); + scoped_ksimd() + crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); src +=3D n; srclen -=3D n; } while (srclen); --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 743443126C4 for ; Wed, 1 Oct 2025 21:04:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352652; cv=none; b=qde8ummD3tGO65d26uNgI5RS5bdHoNT2f7Z3KD+pIL8N1M7aW6MfIdjT9kLYOVFwSNPNfVFvNpQRjD2jEmmQD4WZ+nRtyEUf6bYhttXm7QUm0YXGaLNI9oj6QDXJ/Z+dt1H9aQ5qfB2k1npV2vmDg8J4VWysfSFzpjpaKHYKHKg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352652; c=relaxed/simple; bh=MPr5IH6hojKeJ31wXvFwJFkUG92wHTMjnSLiOjsIPw0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qJRbciBNeyYypIPwx0Rba8BlkouIYxagmczag8FwtP+QsxmwNcWzT+bvMWzh1ubfsJO3KqdYByWAHfVxWlbCYVm6VuCGp9nguSATqKP8VSG60h4GemrbjlwdYmW3PzEmhVf8/7NWwOuJCvCJjtYS2uQnla2BcY1W3yqp408+gdA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ikrOlFDd; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ikrOlFDd" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e303235e8so8464955e9.1 for ; Wed, 01 Oct 2025 14:04:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352649; x=1759957449; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EAysdzbuIX1TDGcwfJh6CQYboY2P1WuMxFJMLAt4CBc=; b=ikrOlFDdpfhLEtn5BivOx3inGLaHthDnL8TphQ1GbcqQ8WZq/6MawcOtaNrz5mrODZ 2ySpr7D6cpTo+XPTkTpEwobGPxYUgcgr2drxFbOOxPYdmkKa+Q3yvXihs818xRMlTNNk fCoDDazS5ah9R7z8cDhnyE5nfGjtw7ovbtYTQX+Yfe/x5amsWCoWmELzFKV6JbRM2kDr IYtasZPwtXi3wbLEhgCWH9hw3I0vdJvnNdmwdt76+5m8knM+GAyGAQLWnW4wR2DY5tdF jvLqaXfa14TV2H1HZVhKN0eRYhAS8+4zKwiNZP58E60btFUdbo80TgvB2nFpgnnp94wf ZS8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352649; x=1759957449; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EAysdzbuIX1TDGcwfJh6CQYboY2P1WuMxFJMLAt4CBc=; b=dw4om5MPD2anaxvoJEHOi9PQ4XFfiqPdNB+QnXMqPE58+gTEoSZD2jLYvMMLHNkByc MVXTLFFz9SYXfVH6T+lRiX5MifPc54TbXXP2gHRnMBjoJeCTuUnunkXPBA5Ns0k+ZW0X hadUtd4yeC8cla+97seFCeMcVOxmwgmHX9EYjO0jVFFLKy9kFCdhofqlXBiYM9LjNi80 h/hsIwnAGFIsEuO5HzGya6x3vWCCPOZi4rBnulqG8yLmFcsNErIUSgYtp/LctkJbQWEq j8qBpGtmDm4suqbFk77Nhnso9s3i553A6RI427298j6zTjBwtp19HcGNTnAUEmxlOmMb pIUQ== X-Forwarded-Encrypted: i=1; AJvYcCWtBqf7JTQ17n8kYNvKRbW0HeqE5vRYYpRKF3tTGwAbrUkCB6dbZu2y5Q2DLhyn6FhO+roKs28DFgjxk7k=@vger.kernel.org X-Gm-Message-State: AOJu0YyGTfFNhlgel+eLj+SkJuxWCwYjsHoz0FHHdU2KE+GvVUaekR4n nxN9jt2R8fODtQtvz+tkiT2ISLaNHV7msKnAAeUX3WWXfzn0wv3Ahce0Tg9SrDZ/JnUyjDd/2g= = X-Google-Smtp-Source: AGHT+IEjk9J1kTCvnc61ywnt8CFYyxydvdrcsVW9hi2dTiTnWEC79fQZ8w6UNQavQPI1A42BSC/bbmAm X-Received: from wmbz18.prod.google.com ([2002:a05:600c:c092:b0:45d:cfa4:ce19]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d1a:b0:45d:cfee:7058 with SMTP id 5b1f17b1804b1-46e612be155mr35926215e9.22.1759352648678; Wed, 01 Oct 2025 14:04:08 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:17 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1318; i=ardb@kernel.org; h=from:subject; bh=BbTS+F2Hc8gpl9Sr4HMF4TFtbcI93GDZQRNcqkbkuTA=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudC6Lped/zOCRfMt7unjXmYQ+kcVzAvhS/eemiCUfz t6g+T+so5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk4SGG//GN1sFvpTfVrWH9 f3uiyOZP6mo2vN0rTN9+FCjYZLt+6SRGhtdlH6cWzU+SZXQ/xhwiOKOJ4/bkx6xftlrdcBX3Pf5 7KwsA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-37-ardb+git@google.com> Subject: [PATCH v2 15/20] crypto/arm64: polyval - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/polyval-ce-glue.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/crypto/polyval-ce-glue.c b/arch/arm64/crypto/polyva= l-ce-glue.c index c4e653688ea0..51eefbe97885 100644 --- a/arch/arm64/crypto/polyval-ce-glue.c +++ b/arch/arm64/crypto/polyval-ce-glue.c @@ -15,7 +15,7 @@ * ARMv8 Crypto Extensions instructions to implement the finite field oper= ations. */ =20 -#include +#include #include #include #include @@ -45,16 +45,14 @@ asmlinkage void pmull_polyval_mul(u8 *op1, const u8 *op= 2); static void internal_polyval_update(const struct polyval_tfm_ctx *keys, const u8 *in, size_t nblocks, u8 *accumulator) { - kernel_neon_begin(); - pmull_polyval_update(keys, in, nblocks, accumulator); - kernel_neon_end(); + scoped_ksimd() + pmull_polyval_update(keys, in, nblocks, accumulator); } =20 static void internal_polyval_mul(u8 *op1, const u8 *op2) { - kernel_neon_begin(); - pmull_polyval_mul(op1, op2); - kernel_neon_end(); + scoped_ksimd() + pmull_polyval_mul(op1, op2); } =20 static int polyval_arm64_setkey(struct crypto_shash *tfm, --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA47231282D for ; Wed, 1 Oct 2025 21:04:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352653; cv=none; b=jqb3CB2hBhmLek9fUuWY4kK/D5mnmm/MEMqZrTn/6k8djAuvWUx60nU93ah3HnlsdYDYqhla51vrpfChd4Xz/j39d3ect0mDbpSQUK5VQsd0s9QXHwWKzvHlCw6w3oKIKiCq/iXPes121NKS5QJ3kV9caMKzgKUeR5UsTzP80BA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352653; c=relaxed/simple; bh=ZgELTmYgz5fIxY25szP3ldagosc57h1+aL7duMm8keQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KTYe+epunlUjG/yYYYkygax2GkqtDSSVQknhIvc3E7umP1kJTHK9D4Z85HNPEVpSiwPK8DgyMvTGRF/5EOoHc1YQISdJpldoka1RJc8EVqL3Q92kz8vnBXcYoM4wfcglrFyOa2GSWJfxILZDVzTFi/2OHmcNHZpPC/EVYngBxfI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AqV4lIUV; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AqV4lIUV" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3f93db57449so103497f8f.2 for ; Wed, 01 Oct 2025 14:04:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352650; x=1759957450; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AGpAD7So2A8Y91ye2lz/R4t8i6WlG+BPLoLmcvlg798=; b=AqV4lIUV0SLUhZYRWXSW4BndnhYOU63hd9R7TZC6FZvmtEj9iLQmUwwt3ApyYeENEL EHusT6kaccNoC8cZCSNlla52JvAJM2sDD4S+694q6PiR8eznuPdw2BSF2I7rZZ4JEMfE KQf7c8oP91ZSDs3aCBf7WNfFJzw+xAsaidMuoAwA3JNiYH3Z0lEsNrKjM/zFPaE6y5ur FreJBY4p9PTTpSOMNwgK1iLNflvagS4Vo20J8qbMGcBDn7ChnFYetl9HYnJVyqL5meLX j5ccSg+kbSFatYFZl7tB6b03KiNmdNiJ5rv2XYkS8NFNsUywuk8YqItRqKJgpbiAo/1z oxiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352650; x=1759957450; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AGpAD7So2A8Y91ye2lz/R4t8i6WlG+BPLoLmcvlg798=; b=WVDrUnJp5l9b+sJWU6q1l0E7MeSoXMM4cz+MVKgARgy321AHNcZe4rAwdNfcgp443u 3ls1sjEUnCKigeSLSM79vySl7Umbk4LFz2bOY7XivQeLUQJfBiDXOdAC700KsLH1nFoO lwLVD0Ff8j525AzDda8I3jmQCrw+/JOdbg1a61mcKn1O/ZUmDvwu/JbqF604mT3m4T7N S5ln/yYlKqmxBvIvmjCJqhvqdtp5WZgoKv03aDQsEHInjNGTyb5BOIxlQQrb/4TFV4lW GWSrG3sjibS3XhxzTGoMRc6zZpzcvcf89913qWXtBw8dD8N4458PRkdspyGiSNudn0ef zV/Q== X-Forwarded-Encrypted: i=1; AJvYcCX4RuMwMNav6Qr3WSgelDXE+ueHD7PaqVKfd4DrJfme7uzs7XC2gwimArM7vSOIMs4Yw4GO+9EY+ykUDvM=@vger.kernel.org X-Gm-Message-State: AOJu0YwZqITb6AkuucWv4nV5syLoJxAP+I9b1ZjQPH5ahiGKpzGPm+EA +0x3iM0G6SXKweI2yNzA+HsuObDHnrGllauyMeto0yueypuyBjsgzsz4uuxukp9PjyqZ1Zw69g= = X-Google-Smtp-Source: AGHT+IHWWReVgSgZUQEq8XT+L+cxH1DIu/oJIKEaKdRtIQPT1yPlkOb9ch5vfT9GQM1XrYkcIG8pVP20 X-Received: from wrxg11.prod.google.com ([2002:a05:6000:118b:b0:403:1719:fe2]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:5f53:0:b0:3ee:1296:d9e8 with SMTP id ffacd0b85a97d-425577f057cmr3436647f8f.17.1759352650072; Wed, 01 Oct 2025 14:04:10 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:18 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1156; i=ardb@kernel.org; h=from:subject; bh=oFEUnftgBXE3h0R5t48zlJmEk3FO4wNTPpQmOq7/RvQ=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudD65SYn5avG5Dcvubcq+/Oeqwd7/dv8Lvh7dE+bzK E5H5/bpjlIWBjEuBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjARkQRGhu9nlry9Ll7B9c5p S3eg+lWV5s/JX2a+vP7s0xz5d3LhTPsZ/kplbDI8nSli739/azmz6Vbd0ilZXGuOBjtYvLxl614 7jwEA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-38-ardb+git@google.com> Subject: [PATCH v2 16/20] crypto/arm64: sha3 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sha3-ce-glue.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/arm64/crypto/sha3-ce-glue.c b/arch/arm64/crypto/sha3-ce-g= lue.c index b4f1001046c9..22732760edd3 100644 --- a/arch/arm64/crypto/sha3-ce-glue.c +++ b/arch/arm64/crypto/sha3-ce-glue.c @@ -46,9 +46,8 @@ static int sha3_update(struct shash_desc *desc, const u8 = *data, do { int rem; =20 - kernel_neon_begin(); - rem =3D sha3_ce_transform(sctx->st, data, blocks, ds); - kernel_neon_end(); + scoped_ksimd() + rem =3D sha3_ce_transform(sctx->st, data, blocks, ds); data +=3D (blocks - rem) * bs; blocks =3D rem; } while (blocks); @@ -73,9 +72,8 @@ static int sha3_finup(struct shash_desc *desc, const u8 *= src, unsigned int len, memset(block + len, 0, bs - len); block[bs - 1] |=3D 0x80; =20 - kernel_neon_begin(); - sha3_ce_transform(sctx->st, block, 1, ds); - kernel_neon_end(); + scoped_ksimd() + sha3_ce_transform(sctx->st, block, 1, ds); memzero_explicit(block , sizeof(block)); =20 for (i =3D 0; i < ds / 8; i++) --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2B983128C8 for ; Wed, 1 Oct 2025 21:04:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352654; cv=none; b=PhaQJTB9FOh2J2ZgI71ROyVCJyB2Squ0D0vd9UmiL7k2mE/pIbgvgoq+pDVhiR+RHQF0AvuCDVH8pkdg/632u8CjGLhHEC/aLpTtefg/xgyUkIgAlr2osnFjM4Be/sy6slesRX/Rsicd4bIVpt/lWYHh14GB9GxLOCu4e8NexkY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352654; c=relaxed/simple; bh=jNAe/V4zNi6LpOPWatt40NoZhMnHLRvmHGAaaO4Ly1s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=iJcnZivqO5UV32e0FrhiovZXi9lhS6tz0mbnoqvhFlYV47mqTl/StWLy9/xTBrgG1kEvkQHgmrnlrEc9zEICUQsAvHdx2RyMuiIWtU+PTkZvIrNteE3kedBxwLXZp78+IsVfpg0uuI6rNzrkegS7TxluD5lrtlTuaiwj2hwhDvo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=KP/r1dsr; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="KP/r1dsr" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e3a049abaso1239965e9.0 for ; Wed, 01 Oct 2025 14:04:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352651; x=1759957451; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NhER74UBpIhoIb7wn4zkF28t+zv1ftYCaNbghO5GaF0=; b=KP/r1dsryPzrDFOHsD3a8VegYS59PvdVwxOHHDFaHBi+Xe5isVSAHmWemHxR5ph1db F3dBd6fiNpqOL42WlD7XUqT0iUe6jnE2MHoZaILPK921Ql1nH20yhnJEXQ83MSVpy0tS pqIk9MrlnPswG/mgETtk1ke+j716WHlRh5ybBW1MIMuvAIsCcVczLTKZtDPlY8OcEKuk aeHSGbENAZCWz33JFlSI5eC+JSrk0Ippp4ryh1h4RRoM8ZPXeKol76ZArw6M5zjC2NCn AsJUxs0SYXgWmZWuAkOHiE9uSDaOPNLf32tkgYgTHW9aIobxuR7q3+ma5Np5yeR7XlU4 q8Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352651; x=1759957451; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NhER74UBpIhoIb7wn4zkF28t+zv1ftYCaNbghO5GaF0=; b=K+g9U9YA09SIDYcUJzbF7vT/svGY6XflqJbXPqNCXwEY9U23D2a8W4G3mMAvRuM5DE WA8d/YGYIwNFs2Ce1RZjACeUIRC13NzLAziFN7f7zhqW/sjUDbMScf+rOuvk87OzjFED 5BajM3/1xS/Q9AR1nt//lmrCATfXBbEXWnGb59iydh6UWZ2H5AW9D+0jJmbzn05F7gr1 rfwQWEC0lTdqmbs1MrWW8EJ+bEvj/PDIwvsWm6xPJsYvwKgvMXUUarLX4kRBGUFOoXAA e3ug7aJkTn219EU3MM6IC935bRMfiq2/oMMbn2o/6Q6CHmeJrr5iTZrXOP+nimg9kO+/ g7HQ== X-Forwarded-Encrypted: i=1; AJvYcCU+xLts0rukWEz2vM5PskqeVGRdiUb+AyteybM1/MX02s6DmlB4P5ZhE1puery7q/LxwNfVsenyj9b1P8E=@vger.kernel.org X-Gm-Message-State: AOJu0YyZf9L9St3nRtddxAc/DGvQzK8hkj+vWorl6CE+XcERkAfxHAyd RbtCT6NglGXtenvgn7d9k6ol+h8sYpARU02kuTrjoyLL2E1vdOyxthF8a6HcLaEmoh5EbnsIYw= = X-Google-Smtp-Source: AGHT+IHOlHJXU9hXA0ZmAXl6mlRNCJKkb1Gcp+5ssLGIRuoPhjvb1Emmd8X4cPDWubbLQ9YjO7PB9gsQ X-Received: from wmcq27.prod.google.com ([2002:a05:600c:c11b:b0:46e:1f26:9212]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:6692:b0:46e:502c:8d6a with SMTP id 5b1f17b1804b1-46e64641e65mr29454385e9.25.1759352651235; Wed, 01 Oct 2025 14:04:11 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:19 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2860; i=ardb@kernel.org; h=from:subject; bh=nxbcY1qMUcCM5Jp3a99oXHHwAeI3LF7dd8KEKxZdHZ4=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudKH2XdeuGB133c2V2KjV5uU9u09ni9zLgvJw7qCF0 9Z8LXjSUcrCIMbFICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaS0M7I8OxoakasXPVCVm6x mvWzfBYxuO7SO7rQoEMsbF7GoxkqCxj+cNsr7Pt+bLaWpqps/7OPbIU9G7oT7rhfTTy33L9yhcN 5XgA= X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-39-ardb+git@google.com> Subject: [PATCH v2 17/20] crypto/arm64: sm3 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sm3-ce-glue.c | 15 ++++++++------- arch/arm64/crypto/sm3-neon-glue.c | 16 ++++++---------- 2 files changed, 14 insertions(+), 17 deletions(-) diff --git a/arch/arm64/crypto/sm3-ce-glue.c b/arch/arm64/crypto/sm3-ce-glu= e.c index eac6f5fa0abe..24c1fcfae072 100644 --- a/arch/arm64/crypto/sm3-ce-glue.c +++ b/arch/arm64/crypto/sm3-ce-glue.c @@ -5,7 +5,6 @@ * Copyright (C) 2018 Linaro Ltd */ =20 -#include #include #include #include @@ -13,6 +12,8 @@ #include #include =20 +#include + MODULE_DESCRIPTION("SM3 secure hash using ARMv8 Crypto Extensions"); MODULE_AUTHOR("Ard Biesheuvel "); MODULE_LICENSE("GPL v2"); @@ -25,18 +26,18 @@ static int sm3_ce_update(struct shash_desc *desc, const= u8 *data, { int remain; =20 - kernel_neon_begin(); - remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_ce_transform); - kernel_neon_end(); + scoped_ksimd() { + remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_ce_transform); + } return remain; } =20 static int sm3_ce_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - kernel_neon_begin(); - sm3_base_do_finup(desc, data, len, sm3_ce_transform); - kernel_neon_end(); + scoped_ksimd() { + sm3_base_do_finup(desc, data, len, sm3_ce_transform); + } return sm3_base_finish(desc, out); } =20 diff --git a/arch/arm64/crypto/sm3-neon-glue.c b/arch/arm64/crypto/sm3-neon= -glue.c index 6c4611a503a3..15f30cc24f32 100644 --- a/arch/arm64/crypto/sm3-neon-glue.c +++ b/arch/arm64/crypto/sm3-neon-glue.c @@ -5,7 +5,7 @@ * Copyright (C) 2022 Tianjia Zhang */ =20 -#include +#include #include #include #include @@ -20,20 +20,16 @@ asmlinkage void sm3_neon_transform(struct sm3_state *ss= t, u8 const *src, static int sm3_neon_update(struct shash_desc *desc, const u8 *data, unsigned int len) { - int remain; - - kernel_neon_begin(); - remain =3D sm3_base_do_update_blocks(desc, data, len, sm3_neon_transform); - kernel_neon_end(); - return remain; + scoped_ksimd() + return sm3_base_do_update_blocks(desc, data, len, + sm3_neon_transform); } =20 static int sm3_neon_finup(struct shash_desc *desc, const u8 *data, unsigned int len, u8 *out) { - kernel_neon_begin(); - sm3_base_do_finup(desc, data, len, sm3_neon_transform); - kernel_neon_end(); + scoped_ksimd() + sm3_base_do_finup(desc, data, len, sm3_neon_transform); return sm3_base_finish(desc, out); } =20 --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC7D4313280 for ; Wed, 1 Oct 2025 21:04:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352656; cv=none; b=j9PKKR303ZZ1YRDKkBT4PiYJ+/0LclTY9kIXMg5NrGfE51u7nAw/NzUDGeqxC/T4ZesQCJOlKi3+ga+OeuI5yGWG5TkKlzi1zi784rAI1+7ZaM3ScFCWQuXaNrzq277pFDeDybObGlHRgTumgSw6Di2qD/yMoAyVxqFCdnDGAS0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352656; c=relaxed/simple; bh=hzhBBv9qXLsX+gw2P3VFedRwF4INabNGQFsYtmOGp48=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XjNh8yHPOMOrwjeybAnffa67XosDZG64iOtVc4z40bDP7QZhmD9tvIVEGtSUeT9fGNyghan4AxQO9O+FXlcuhsoHPTlskfxvoDM3KslGqJgqjNtTw9P0d+s+i/GuEiy/fUJbbhBmwIYSbthBUr9JTa9CUPehrfRuBqLD6Nonoy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=P/dUx+i4; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="P/dUx+i4" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46b303f6c9cso1572245e9.2 for ; Wed, 01 Oct 2025 14:04:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352652; x=1759957452; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=g+tkM5nzc98sysnyWW7NQQhabfH4eEjn64FzXa1tBKM=; b=P/dUx+i4C+XI8H1y4jWPTpXSwtOtKTdfFnxgzi9wrMTW/c1RVSom0l+yK1+1aEJaDE z3D4kp2QdwIvbuQ40cydlTX0v+htEzMPRLQQqVVCELMYFwp53HdJ4S1B8P5U4IwANLvj X1j49SMdLXQdGCYEUU6QBVBktz8Wn1p3KNVdmCuE/lb6GrkezQq2g2wTyqfLAwwGh9TR nWwJb795/+CEStZFKzpgy6u7nFQ40Lo8xEwSpBj2Yms4A2Loc5lBJVLFQ5xV31Xg9j+K 3ipX3M+/Lak7/Rv/g24V56D1VzquxBla+yWBgro/E1eckVSRAsxDE9CqLn8tlbmNasUX 0XZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352652; x=1759957452; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=g+tkM5nzc98sysnyWW7NQQhabfH4eEjn64FzXa1tBKM=; b=gXCmUi9PwwBMWBCZDTIUpCvFk00y1Xd8KVhW0d0allWuVYP8CUcSMzlXtm/V8kIFwZ CBx0aL+HQFqyUKeN88FucD1EQcZkopbl/3POISiZ55OgWb/DWbaMjxjWIVdsIDEhGfpd 6DgLah/V1+I+vw7VhsZVTh6g/uQ9nuh77bg+Vsy5+2hQIjvgpUsKAQXIya3dnxnB9QZE WPGuWcVBZsaeY/gFNmHW1nKAuKlzwCIYVPn/SHw3gXNgqpV+Ut8GlU0HBklEOZkja/zL TyFx5sZz35IBrrsH8uIfa7dB+m9fAHVAFW5I9AZREMFWK7azb5qi2daCX1m2x1fevD5L fUBQ== X-Forwarded-Encrypted: i=1; AJvYcCWrMba0uesg5uo63mTwjNJ/uwFtZ/grdDKmOW0MwBtH10PPeRNxlT8YZNS930oR4INv/7LGAh7ScOnQsYM=@vger.kernel.org X-Gm-Message-State: AOJu0YzJfeyO2ge8wSpHd8CUhEEseNwKKNwOJAOyUsCLPm1CoaGogfVw ox6qGm6ao7YGf0uVqhGxU8iVO95Asqnxrx+KzLAh68YBnPRrzkMQRNcVIcpESj1unY8+9QLE8g= = X-Google-Smtp-Source: AGHT+IGTbbGeB8Bf6LaYapNslqa0R16CPpjdm36/ci/sQr8CoJzBEl3ACSEhyCvHpEHmBBJvgnVs4JCM X-Received: from wmbjt22.prod.google.com ([2002:a05:600c:5696:b0:46c:f222:92c9]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3553:b0:46e:4b42:1dbe with SMTP id 5b1f17b1804b1-46e612de2eamr35990695e9.32.1759352652363; Wed, 01 Oct 2025 14:04:12 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:20 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=18083; i=ardb@kernel.org; h=from:subject; bh=OgGwE/LQY2w0L4dIZqEdIxr7D0ntO12+gElnEA3Y3HQ=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudNGgTOZFWiUGG2Z/P3Y+Pd/jc5v99Nt67KqxEv8cW FVfXuvsKGVhEONikBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABO5Gcnwz2iC9/LvNlPuflkQ 7RfebliozVPDX7tH31ZQp+POzCctnxkZ/oVtvSzF+urWht+/Bb+fCOGbe/11doTZijLHr58T7Yz WMwIA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-40-ardb+git@google.com> Subject: [PATCH v2 18/20] crypto/arm64: sm4 - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/sm4-ce-ccm-glue.c | 49 +++-- arch/arm64/crypto/sm4-ce-cipher-glue.c | 10 +- arch/arm64/crypto/sm4-ce-gcm-glue.c | 61 +++--- arch/arm64/crypto/sm4-ce-glue.c | 214 +++++++++----------- arch/arm64/crypto/sm4-neon-glue.c | 25 +-- 5 files changed, 158 insertions(+), 201 deletions(-) diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce= -ccm-glue.c index f9771ab2a05f..390facf909a0 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include @@ -35,10 +35,9 @@ static int ccm_setkey(struct crypto_aead *tfm, const u8 = *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 return 0; } @@ -167,35 +166,33 @@ static int ccm_crypt(struct aead_request *req, struct= skcipher_walk *walk, memcpy(ctr0, walk->iv, SM4_BLOCK_SIZE); crypto_inc(walk->iv, SM4_BLOCK_SIZE); =20 - kernel_neon_begin(); + scoped_ksimd() { + if (req->assoclen) + ccm_calculate_auth_mac(req, mac); =20 - if (req->assoclen) - ccm_calculate_auth_mac(req, mac); - - while (walk->nbytes && walk->nbytes !=3D walk->total) { - unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; + while (walk->nbytes && walk->nbytes !=3D walk->total) { + unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; =20 - sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, - walk->src.virt.addr, walk->iv, - walk->nbytes - tail, mac); + sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, + walk->src.virt.addr, walk->iv, + walk->nbytes - tail, mac); =20 - err =3D skcipher_walk_done(walk, tail); - } + err =3D skcipher_walk_done(walk, tail); + } =20 - if (walk->nbytes) { - sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, - walk->src.virt.addr, walk->iv, - walk->nbytes, mac); + if (walk->nbytes) { + sm4_ce_ccm_crypt(rkey_enc, walk->dst.virt.addr, + walk->src.virt.addr, walk->iv, + walk->nbytes, mac); =20 - sm4_ce_ccm_final(rkey_enc, ctr0, mac); + sm4_ce_ccm_final(rkey_enc, ctr0, mac); =20 - err =3D skcipher_walk_done(walk, 0); - } else { - sm4_ce_ccm_final(rkey_enc, ctr0, mac); + err =3D skcipher_walk_done(walk, 0); + } else { + sm4_ce_ccm_final(rkey_enc, ctr0, mac); + } } =20 - kernel_neon_end(); - return err; } =20 diff --git a/arch/arm64/crypto/sm4-ce-cipher-glue.c b/arch/arm64/crypto/sm4= -ce-cipher-glue.c index c31d76fb5a17..bceec833ef4e 100644 --- a/arch/arm64/crypto/sm4-ce-cipher-glue.c +++ b/arch/arm64/crypto/sm4-ce-cipher-glue.c @@ -32,9 +32,8 @@ static void sm4_ce_encrypt(struct crypto_tfm *tfm, u8 *ou= t, const u8 *in) if (!crypto_simd_usable()) { sm4_crypt_block(ctx->rkey_enc, out, in); } else { - kernel_neon_begin(); - sm4_ce_do_crypt(ctx->rkey_enc, out, in); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_do_crypt(ctx->rkey_enc, out, in); } } =20 @@ -45,9 +44,8 @@ static void sm4_ce_decrypt(struct crypto_tfm *tfm, u8 *ou= t, const u8 *in) if (!crypto_simd_usable()) { sm4_crypt_block(ctx->rkey_dec, out, in); } else { - kernel_neon_begin(); - sm4_ce_do_crypt(ctx->rkey_dec, out, in); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_do_crypt(ctx->rkey_dec, out, in); } } =20 diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce= -gcm-glue.c index 170cd0151385..32a6ab669281 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include #include #include #include @@ -48,13 +48,11 @@ static int gcm_setkey(struct crypto_aead *tfm, const u8= *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - sm4_ce_pmull_ghash_setup(ctx->key.rkey_enc, ctx->ghash_table); - - kernel_neon_end(); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); + sm4_ce_pmull_ghash_setup(ctx->key.rkey_enc, ctx->ghash_table); + } return 0; } =20 @@ -149,40 +147,35 @@ static int gcm_crypt(struct aead_request *req, struct= skcipher_walk *walk, memcpy(iv, req->iv, GCM_IV_SIZE); put_unaligned_be32(2, iv + GCM_IV_SIZE); =20 - kernel_neon_begin(); + scoped_ksimd() { + if (req->assoclen) + gcm_calculate_auth_mac(req, ghash); =20 - if (req->assoclen) - gcm_calculate_auth_mac(req, ghash); + while (walk->nbytes) { + unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; + const u8 *src =3D walk->src.virt.addr; + u8 *dst =3D walk->dst.virt.addr; =20 - while (walk->nbytes) { - unsigned int tail =3D walk->nbytes % SM4_BLOCK_SIZE; - const u8 *src =3D walk->src.virt.addr; - u8 *dst =3D walk->dst.virt.addr; + if (walk->nbytes =3D=3D walk->total) { + sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, + walk->nbytes, ghash, + ctx->ghash_table, + (const u8 *)&lengths); + + return skcipher_walk_done(walk, 0); + } =20 - if (walk->nbytes =3D=3D walk->total) { sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, - walk->nbytes, ghash, - ctx->ghash_table, - (const u8 *)&lengths); + walk->nbytes - tail, ghash, + ctx->ghash_table, NULL); =20 - err =3D skcipher_walk_done(walk, 0); - goto out; + err =3D skcipher_walk_done(walk, tail); } =20 - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, dst, src, iv, - walk->nbytes - tail, ghash, - ctx->ghash_table, NULL); - - err =3D skcipher_walk_done(walk, tail); + sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, NULL, NULL, iv, + walk->nbytes, ghash, ctx->ghash_table, + (const u8 *)&lengths); } - - sm4_ce_pmull_gcm_crypt(ctx->key.rkey_enc, NULL, NULL, iv, - walk->nbytes, ghash, ctx->ghash_table, - (const u8 *)&lengths); - -out: - kernel_neon_end(); - return err; } =20 diff --git a/arch/arm64/crypto/sm4-ce-glue.c b/arch/arm64/crypto/sm4-ce-glu= e.c index 7a60e7b559dc..57ae3406257c 100644 --- a/arch/arm64/crypto/sm4-ce-glue.c +++ b/arch/arm64/crypto/sm4-ce-glue.c @@ -8,7 +8,7 @@ * Copyright (C) 2022 Tianjia Zhang */ =20 -#include +#include #include #include #include @@ -74,10 +74,9 @@ static int sm4_setkey(struct crypto_skcipher *tfm, const= u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_expand_key(key, ctx->rkey_enc, ctx->rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); return 0; } =20 @@ -94,12 +93,12 @@ static int sm4_xts_setkey(struct crypto_skcipher *tfm, = const u8 *key, if (ret) return ret; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->key1.rkey_enc, - ctx->key1.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); - sm4_ce_expand_key(&key[SM4_KEY_SIZE], ctx->key2.rkey_enc, - ctx->key2.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key1.rkey_enc, + ctx->key1.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); + sm4_ce_expand_key(&key[SM4_KEY_SIZE], ctx->key2.rkey_enc, + ctx->key2.rkey_dec, crypto_sm4_fk, crypto_sm4_ck); + } =20 return 0; } @@ -117,16 +116,14 @@ static int sm4_ecb_do_crypt(struct skcipher_request *= req, const u32 *rkey) u8 *dst =3D walk.dst.virt.addr; unsigned int nblks; =20 - kernel_neon_begin(); - - nblks =3D BYTES2BLKS(nbytes); - if (nblks) { - sm4_ce_crypt(rkey, dst, src, nblks); - nbytes -=3D nblks * SM4_BLOCK_SIZE; + scoped_ksimd() { + nblks =3D BYTES2BLKS(nbytes); + if (nblks) { + sm4_ce_crypt(rkey, dst, src, nblks); + nbytes -=3D nblks * SM4_BLOCK_SIZE; + } } =20 - kernel_neon_end(); - err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -167,16 +164,14 @@ static int sm4_cbc_crypt(struct skcipher_request *req, =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - if (encrypt) - sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, - walk.iv, nblocks); - else - sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_cbc_enc(ctx->rkey_enc, dst, src, + walk.iv, nblocks); + else + sm4_ce_cbc_dec(ctx->rkey_dec, dst, src, + walk.iv, nblocks); + } } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -249,16 +244,14 @@ static int sm4_cbc_cts_crypt(struct skcipher_request = *req, bool encrypt) if (err) return err; =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_cbc_cts_enc(ctx->rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes); - else - sm4_ce_cbc_cts_dec(ctx->rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_cbc_cts_enc(ctx->rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes); + else + sm4_ce_cbc_cts_dec(ctx->rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes); + } =20 return skcipher_walk_done(&walk, 0); } @@ -288,28 +281,26 @@ static int sm4_ctr_crypt(struct skcipher_request *req) u8 *dst =3D walk.dst.virt.addr; unsigned int nblks; =20 - kernel_neon_begin(); - - nblks =3D BYTES2BLKS(nbytes); - if (nblks) { - sm4_ce_ctr_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); - dst +=3D nblks * SM4_BLOCK_SIZE; - src +=3D nblks * SM4_BLOCK_SIZE; - nbytes -=3D nblks * SM4_BLOCK_SIZE; - } - - /* tail */ - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - u8 keystream[SM4_BLOCK_SIZE]; - - sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); - crypto_inc(walk.iv, SM4_BLOCK_SIZE); - crypto_xor_cpy(dst, src, keystream, nbytes); - nbytes =3D 0; + scoped_ksimd() { + nblks =3D BYTES2BLKS(nbytes); + if (nblks) { + sm4_ce_ctr_enc(ctx->rkey_enc, dst, src, walk.iv, nblks); + dst +=3D nblks * SM4_BLOCK_SIZE; + src +=3D nblks * SM4_BLOCK_SIZE; + nbytes -=3D nblks * SM4_BLOCK_SIZE; + } + + /* tail */ + if (walk.nbytes =3D=3D walk.total && nbytes > 0) { + u8 keystream[SM4_BLOCK_SIZE]; + + sm4_ce_crypt_block(ctx->rkey_enc, keystream, walk.iv); + crypto_inc(walk.iv, SM4_BLOCK_SIZE); + crypto_xor_cpy(dst, src, keystream, nbytes); + nbytes =3D 0; + } } =20 - kernel_neon_end(); - err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -359,18 +350,16 @@ static int sm4_xts_crypt(struct skcipher_request *req= , bool encrypt) if (nbytes < walk.total) nbytes &=3D ~(SM4_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, nbytes, - rkey2_enc); - else - sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, nbytes, - rkey2_enc); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, nbytes, + rkey2_enc); + else + sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, nbytes, + rkey2_enc); + } =20 rkey2_enc =3D NULL; =20 @@ -395,18 +384,16 @@ static int sm4_xts_crypt(struct skcipher_request *req= , bool encrypt) if (err) return err; =20 - kernel_neon_begin(); - - if (encrypt) - sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes, - rkey2_enc); - else - sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, - walk.src.virt.addr, walk.iv, walk.nbytes, - rkey2_enc); - - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes, + rkey2_enc); + else + sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr, + walk.src.virt.addr, walk.iv, walk.nbytes, + rkey2_enc); + } =20 return skcipher_walk_done(&walk, 0); } @@ -510,11 +497,9 @@ static int sm4_cbcmac_setkey(struct crypto_shash *tfm,= const u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - kernel_neon_end(); - + scoped_ksimd() + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); return 0; } =20 @@ -530,15 +515,13 @@ static int sm4_cmac_setkey(struct crypto_shash *tfm, = const u8 *key, =20 memset(consts, 0, SM4_BLOCK_SIZE); =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 - /* encrypt the zero block */ - sm4_ce_crypt_block(ctx->key.rkey_enc, (u8 *)consts, (const u8 *)consts); - - kernel_neon_end(); + /* encrypt the zero block */ + sm4_ce_crypt_block(ctx->key.rkey_enc, (u8 *)consts, (const u8 *)consts); + } =20 /* gf(2^128) multiply zero-ciphertext with u and u^2 */ a =3D be64_to_cpu(consts[0].a); @@ -568,18 +551,16 @@ static int sm4_xcbc_setkey(struct crypto_shash *tfm, = const u8 *key, if (key_len !=3D SM4_KEY_SIZE) return -EINVAL; =20 - kernel_neon_begin(); - - sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); + scoped_ksimd() { + sm4_ce_expand_key(key, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); =20 - sm4_ce_crypt_block(ctx->key.rkey_enc, key2, ks[0]); - sm4_ce_crypt(ctx->key.rkey_enc, ctx->consts, ks[1], 2); + sm4_ce_crypt_block(ctx->key.rkey_enc, key2, ks[0]); + sm4_ce_crypt(ctx->key.rkey_enc, ctx->consts, ks[1], 2); =20 - sm4_ce_expand_key(key2, ctx->key.rkey_enc, ctx->key.rkey_dec, - crypto_sm4_fk, crypto_sm4_ck); - - kernel_neon_end(); + sm4_ce_expand_key(key2, ctx->key.rkey_enc, ctx->key.rkey_dec, + crypto_sm4_fk, crypto_sm4_ck); + } =20 return 0; } @@ -600,10 +581,9 @@ static int sm4_mac_update(struct shash_desc *desc, con= st u8 *p, unsigned int nblocks =3D len / SM4_BLOCK_SIZE; =20 len %=3D SM4_BLOCK_SIZE; - kernel_neon_begin(); - sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, p, - nblocks, false, true); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, p, + nblocks, false, true); return len; } =20 @@ -619,10 +599,9 @@ static int sm4_cmac_finup(struct shash_desc *desc, con= st u8 *src, ctx->digest[len] ^=3D 0x80; consts +=3D SM4_BLOCK_SIZE; } - kernel_neon_begin(); - sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, consts, 1, - false, true); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_mac_update(tctx->key.rkey_enc, ctx->digest, consts, 1, + false, true); memcpy(out, ctx->digest, SM4_BLOCK_SIZE); return 0; } @@ -635,10 +614,9 @@ static int sm4_cbcmac_finup(struct shash_desc *desc, c= onst u8 *src, =20 if (len) { crypto_xor(ctx->digest, src, len); - kernel_neon_begin(); - sm4_ce_crypt_block(tctx->key.rkey_enc, ctx->digest, - ctx->digest); - kernel_neon_end(); + scoped_ksimd() + sm4_ce_crypt_block(tctx->key.rkey_enc, ctx->digest, + ctx->digest); } memcpy(out, ctx->digest, SM4_BLOCK_SIZE); return 0; diff --git a/arch/arm64/crypto/sm4-neon-glue.c b/arch/arm64/crypto/sm4-neon= -glue.c index e3500aca2d18..e944c2a2efb0 100644 --- a/arch/arm64/crypto/sm4-neon-glue.c +++ b/arch/arm64/crypto/sm4-neon-glue.c @@ -48,11 +48,8 @@ static int sm4_ecb_do_crypt(struct skcipher_request *req= , const u32 *rkey) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_crypt(rkey, dst, src, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_crypt(rkey, dst, src, nblocks); } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -126,12 +123,9 @@ static int sm4_cbc_decrypt(struct skcipher_request *re= q) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_cbc_dec(ctx->rkey_dec, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_cbc_dec(ctx->rkey_dec, dst, src, + walk.iv, nblocks); } =20 err =3D skcipher_walk_done(&walk, nbytes % SM4_BLOCK_SIZE); @@ -157,12 +151,9 @@ static int sm4_ctr_crypt(struct skcipher_request *req) =20 nblocks =3D nbytes / SM4_BLOCK_SIZE; if (nblocks) { - kernel_neon_begin(); - - sm4_neon_ctr_crypt(ctx->rkey_enc, dst, src, - walk.iv, nblocks); - - kernel_neon_end(); + scoped_ksimd() + sm4_neon_ctr_crypt(ctx->rkey_enc, dst, src, + walk.iv, nblocks); =20 dst +=3D nblocks * SM4_BLOCK_SIZE; src +=3D nblocks * SM4_BLOCK_SIZE; --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B326313520 for ; Wed, 1 Oct 2025 21:04:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352657; cv=none; b=RJhQmVBw1NjZedfDz2scjQ/j/l0bZztIznVDEWDdIiFHCKLZxymQrUadVTW08XaTM+En83kTs/QWl+aIPF1Q5uXUsc7O8GJW+7g0D7/NVx//k34E2LIAKEKzV6KpMlVRKl0pkGrnjGwlnTQ/R06FoU4Qhb5RaGNHw6XdogZjsck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352657; c=relaxed/simple; bh=ZkL7LWnWXcINBIYfTYLOVlnT/4NvtyeplD+CvXeMct4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bDBCdFY0bbLLv5YutNLESU/u1Nmo7Cm1dFPVpi1gh0yLbYCfVfujuYK2JfA4KRd7UJdxHoDC1QcyyajY/IHnq3AkHgQjny/m4+C599tt+G10z/TLx6JuScTkZnphZKSWKhyplO+uAe+qr9euQH+MIygwj7GWSXZGhyzfEDxPkoo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bbRmD57i; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bbRmD57i" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-63798d4b7caso248239a12.3 for ; Wed, 01 Oct 2025 14:04:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352653; x=1759957453; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ROUK26yKaM3+r4gp21QNYLzmmtvtE/oyrWIw8ssAYWA=; b=bbRmD57iuICNHsucBuggOmjTOlkbnD3+D3p8+3542Ndt2kzEjXC0r9iHE+RPyaK4yA jw0zeEr9rQ2aigkcRYvCSml0DPH9pFZWcpEm2YJKZU2VaLtJ+roceOleYTLbgR1xLGgL pdWiAWO5kDLr/u6RgbdohwZt+kGerI/esnAI57cva60o9vb8gsEyQ0DgzpSU4tUOY6D1 VIXnDlJ8qmcbWlmsP4B8XFJnEtasjurtLF67hXkFDGUey7OeHYsshkBg8PZi2jF/dD4a gc0Y4A3CVbhPnRkdWLCxcj6g9uY986XXGDo5rBE0x5S3io4sjTSkzTMK2EvCFKKKhDwL cpEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352653; x=1759957453; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ROUK26yKaM3+r4gp21QNYLzmmtvtE/oyrWIw8ssAYWA=; b=xTOqHmPTOnQpoCX1Be/+D5CULGn/548lHZCzinkibvJBjUyIaUtlVmgJIhYKVUyN/M BmdAOWdH79Zt0Fi4SWnDMMWcWM+AbCgPhyey1bTFZ7sf/s+UoE5fZBoJ2vaW1JbkwONS vAy4+mY82UF8aLPsCgs9F+TANvXO9A3DwxgFzTAHb1ENM7YAYqwO5rQbZHueoOvJz6Ks 3K52s306JAezEMHvAHwVB7Z+YMaH4g1IMiS9laOzTFwiqGSMsMI5zOVRvFKnY5n2lt1G pg9FdfprEPTLoF/B3uwlLnn6m9Sq41Eq88tauTDD8JJXp+zhyDMwXBl7gXWpfILb7tSp BCAA== X-Forwarded-Encrypted: i=1; AJvYcCUDBsDiZHhEMOX6GpCldjsLdDTjCB8MSJs2gmAooJrzNLq573NiR/ODLs/7K3kv5Pflzx2wmzO+NSQ2gBI=@vger.kernel.org X-Gm-Message-State: AOJu0YxRDx5U9N3VWan5lNHaR8QX/FJ2u9ac5QYwSsGkmTUn7iGLIGYu 53Ro1NAmItll40q8koZhTyS9OZurOEj79PYZPqfqRG9NFwf42X3n0KGbIC2ZJd+AeGC2Yinzew= = X-Google-Smtp-Source: AGHT+IHdBwWuloMVUG2aiVvBgPmXkgGhEuo55jA45O9PblNGid5/E1qaOmRe7tj9K1otJi7AohsCgKdN X-Received: from edrj9.prod.google.com ([2002:aa7:c349:0:b0:633:3402:d21b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:31eb:b0:634:cb54:810e with SMTP id 4fb4d7f45d1cf-63678c7b7f1mr4089985a12.31.1759352653423; Wed, 01 Oct 2025 14:04:13 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:21 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2026; i=ardb@kernel.org; h=from:subject; bh=L5/WB9FoNS6vxFGUhS8qwMQNq/irqgYzuB5f44a5Viw=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudMkvCx4bO1zc0Py8oIBBvm1d5YOu6Ibcg0YiLeezT j1IXbymo5SFQYyLQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzEfQPDPxvRymVXmZzqHHqE H76zk41vn7d4zkU39zc2Hz9Jr/Q8nMzI8KM/3Ek1telkGIvnq2JXr7y+npyFfaHl/y8FMn/7MfE IGwA= X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-41-ardb+git@google.com> Subject: [PATCH v2 19/20] arm64/xorblocks: Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/xor.h | 22 ++++++++------------ 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/xor.h b/arch/arm64/include/asm/xor.h index befcd8a7abc9..c38e3d017a79 100644 --- a/arch/arm64/include/asm/xor.h +++ b/arch/arm64/include/asm/xor.h @@ -9,7 +9,7 @@ #include #include #include -#include +#include =20 #ifdef CONFIG_KERNEL_MODE_NEON =20 @@ -19,9 +19,8 @@ static void xor_neon_2(unsigned long bytes, unsigned long * __restrict p1, const unsigned long * __restrict p2) { - kernel_neon_begin(); - xor_block_inner_neon.do_2(bytes, p1, p2); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_2(bytes, p1, p2); } =20 static void @@ -29,9 +28,8 @@ xor_neon_3(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p2, const unsigned long * __restrict p3) { - kernel_neon_begin(); - xor_block_inner_neon.do_3(bytes, p1, p2, p3); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_3(bytes, p1, p2, p3); } =20 static void @@ -40,9 +38,8 @@ xor_neon_4(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p3, const unsigned long * __restrict p4) { - kernel_neon_begin(); - xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4); } =20 static void @@ -52,9 +49,8 @@ xor_neon_5(unsigned long bytes, unsigned long * __restric= t p1, const unsigned long * __restrict p4, const unsigned long * __restrict p5) { - kernel_neon_begin(); - xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5); - kernel_neon_end(); + scoped_ksimd() + xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5); } =20 static struct xor_block_template xor_block_arm64 =3D { --=20 2.51.0.618.g983fd99d29-goog From nobody Thu Oct 2 10:39:00 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2487E313540 for ; Wed, 1 Oct 2025 21:04:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352658; cv=none; b=MmIdkmrKIg49/OWDch//XxP1JPR0zuaSj5UjLr4FlJP6rBfWzV5Ah3wY4UpdshbTkeFqbRt6gFoKKVd31CEaiIt8j4fnPfHjJhErc94GbCQMdNYAvLagGGLmSZYTmdEhNAc2td9KU4vv3nk0WiDtgPR576lL20cFxYDhuuso29Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352658; c=relaxed/simple; bh=NrkKCf7Vf8E5XPoueeWIQqRDMvrxqVXoVSGlKLkPssM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eIfAEKDJUJo/NP7afjX3HcLetnML8gNvB2e+L1hr8HDn6L6wQXcGOtCuBtAM3s6uX9lWrB+6wHbaCqmgFsoUSZz+Nb+G/WE6I8hSkvVatLQiBNjxIp7iEiFRDF6WRK2A+AAVyWPPwyXibud+tAiXL+9gMFupnIn2e4jHsIQJliY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=K9vtyEeP; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="K9vtyEeP" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e4cc8ed76so727055e9.1 for ; Wed, 01 Oct 2025 14:04:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352654; x=1759957454; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=x1XNfcXLPxTuycG48FvJ57h34RR7WVga1h96zHJSpsk=; b=K9vtyEeP6b+LhqaPIRy3gPlAzw+DEK8+hFVGzJFjV+uXsPeikni2cxgnbo8m9DcO9X vejlkAAfFSYm1Xj5dZZj7+/ccZ9jQSseH0ouun2gVX3U/Jidiw7cuZqc32JBdI2dySnB ACDFwfGFuxSevLXbvUFgJ6bIXX3w3r3EhKltrB0xtIh3BQPiBOrCYZnIObW2aKSFG+8f +AGRb6GA7te/u0oeQhpDK7zdbzPzT4d4CszJdkV7OZEev9LwvK88HaaJCeWMVg+zL7UY EtQU/xOTpWzUunkEzgn1Gqz/UiQGHOdQ6EmjZufFzVovXu/6JH4vj5kizjTlq1Js4BPo z9pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352654; x=1759957454; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=x1XNfcXLPxTuycG48FvJ57h34RR7WVga1h96zHJSpsk=; b=qBJAIoxJSiF6oi1u/SVPtkw9l2x+iKdti5GRAMqhmrEvjOC0sf9S1xUsKxWt/ExKYR g3OLGjikvv0p+Ro2QAp6W/+v6/nvwXsQc/5aDmyXVAr1jov8AvxFc3L2ZPwAAUHfQ8vj MrEt4eo5cDn1Mn0/YM1pYONZ7RjLb8lHOohCwuazFOX3wEKh5LOP7JsiU0KXmN2vjsZ+ 8LpY4hLOJ2VxtGUQfdpSP8p9UBkpy86w84gQn/DrojeNdx9pTyMcKFU6ww7cP84au5iZ 6DWCeUoONv0/CD5p0yX52QvUgmuEWwgATbAy9HcPRCIN/q/Gep49/7Tet7DIn5KOHXPy KMVg== X-Forwarded-Encrypted: i=1; AJvYcCVcsPU9HySHF8crbPHu6cAAaSFl40DmInhCRW+W9npYLBLu3oVuEagYeS6oM2HZscLsCX00LkBIJals/zE=@vger.kernel.org X-Gm-Message-State: AOJu0YyMDFxrjgMDZwAkdSnCPDaTdSDqkE4X6L17IQDRVHLvL3mksswO UQVCom3MxgPpJEaYtaRA+jiNGXfz7NykuIW6wnwHKJNZ2OV0ilLyoSwtFGVEmCaL8js7Wid0rQ= = X-Google-Smtp-Source: AGHT+IF4pAXl5Kz4VKrEzIr4Pw9yb2GP5tMgq1HmwMabQ5THhboAs0iCENwI8BZUxUuDCIiW3R3H9LbE X-Received: from wmbh26.prod.google.com ([2002:a05:600c:a11a:b0:45d:cfa4:ce0d]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:45d0:b0:46e:1d8d:cfa2 with SMTP id 5b1f17b1804b1-46e612bab92mr36541765e9.20.1759352654487; Wed, 01 Oct 2025 14:04:14 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:22 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6908; i=ardb@kernel.org; h=from:subject; bh=AKLCnRRN0mrxopEXea8EITSSalYE601b/mp0OMxmCUA=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePudPllFx24g4Qqp876JueqIbU3bdpm9jenXldn3VCqc 7xwoXx6RykLgxgXg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI/3JGhpfBSQwSSv8eLPfY NFdq3fNCmQ7XN/e1mDiza847V+1/8Izhn2WKrPTZCa8PqC/9+8/01/oEz/e2W6NvM6rtUG1RPc+ VwAIA X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-42-ardb+git@google.com> Subject: [PATCH v2 20/20] arm64/fpsimd: Allocate kernel mode FP/SIMD buffers on the stack From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Commit aefbab8e77eb16b5 ("arm64: fpsimd: Preserve/restore kernel mode NEON at context switch") added a 'kernel_fpsimd_state' field to struct thread_struct, which is the arch-specific portion of struct task_struct, and is allocated for each task in the system. The size of this field is 528 bytes, resulting in non-trivial bloat of task_struct, and the resulting memory overhead may impact performance on systems with many processes. This allocation is only used if the task is scheduled out or interrupted by a softirq while using the FP/SIMD unit in kernel mode, and so it is possible to transparently allocate this buffer on the caller's stack instead. So tweak the 'ksimd' scoped guard implementation so that a stack buffer is allocated and passed to both kernel_neon_begin() and kernel_neon_end(), and record it in the task struct. Passing the address to both functions, and checking the addresses for consistency ensures that callers of the updated bare begin/end API use it in a manner that is consistent with the new context switch semantics. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/neon.h | 4 +-- arch/arm64/include/asm/processor.h | 2 +- arch/arm64/include/asm/simd.h | 7 ++-- arch/arm64/kernel/fpsimd.c | 34 +++++++++++++------- 4 files changed, 31 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/neon.h b/arch/arm64/include/asm/neon.h index d4b1d172a79b..acebee4605b5 100644 --- a/arch/arm64/include/asm/neon.h +++ b/arch/arm64/include/asm/neon.h @@ -13,7 +13,7 @@ =20 #define cpu_has_neon() system_supports_fpsimd() =20 -void kernel_neon_begin(void); -void kernel_neon_end(void); +void kernel_neon_begin(struct user_fpsimd_state *); +void kernel_neon_end(struct user_fpsimd_state *); =20 #endif /* ! __ASM_NEON_H */ diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/pr= ocessor.h index 4f8d677b73ee..93bca4d454d7 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -172,7 +172,7 @@ struct thread_struct { unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ =20 - struct user_fpsimd_state kernel_fpsimd_state; + struct user_fpsimd_state *kernel_fpsimd_state; unsigned int kernel_fpsimd_cpu; #ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user; diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h index d9f83c478736..7ddb25df5c98 100644 --- a/arch/arm64/include/asm/simd.h +++ b/arch/arm64/include/asm/simd.h @@ -43,8 +43,11 @@ static __must_check inline bool may_use_simd(void) { =20 #endif /* ! CONFIG_KERNEL_MODE_NEON */ =20 -DEFINE_LOCK_GUARD_0(ksimd, kernel_neon_begin(), kernel_neon_end()) +DEFINE_LOCK_GUARD_1(ksimd, + struct user_fpsimd_state, + kernel_neon_begin(_T->lock), + kernel_neon_end(_T->lock)) =20 -#define scoped_ksimd() scoped_guard(ksimd) +#define scoped_ksimd() scoped_guard(ksimd, &(struct user_fpsimd_state){}) =20 #endif diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index c37f02d7194e..ea9192a180aa 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1488,21 +1488,23 @@ static void fpsimd_load_kernel_state(struct task_st= ruct *task) * Elide the load if this CPU holds the most recent kernel mode * FPSIMD context of the current task. */ - if (last->st =3D=3D &task->thread.kernel_fpsimd_state && + if (last->st =3D=3D task->thread.kernel_fpsimd_state && task->thread.kernel_fpsimd_cpu =3D=3D smp_processor_id()) return; =20 - fpsimd_load_state(&task->thread.kernel_fpsimd_state); + fpsimd_load_state(task->thread.kernel_fpsimd_state); } =20 static void fpsimd_save_kernel_state(struct task_struct *task) { struct cpu_fp_state cpu_fp_state =3D { - .st =3D &task->thread.kernel_fpsimd_state, + .st =3D task->thread.kernel_fpsimd_state, .to_save =3D FP_STATE_FPSIMD, }; =20 - fpsimd_save_state(&task->thread.kernel_fpsimd_state); + BUG_ON(!cpu_fp_state.st); + + fpsimd_save_state(task->thread.kernel_fpsimd_state); fpsimd_bind_state_to_cpu(&cpu_fp_state); =20 task->thread.kernel_fpsimd_cpu =3D smp_processor_id(); @@ -1773,6 +1775,7 @@ void fpsimd_update_current_state(struct user_fpsimd_s= tate const *state) void fpsimd_flush_task_state(struct task_struct *t) { t->thread.fpsimd_cpu =3D NR_CPUS; + t->thread.kernel_fpsimd_state =3D NULL; /* * If we don't support fpsimd, bail out after we have * reset the fpsimd_cpu for this task and clear the @@ -1833,7 +1836,7 @@ void fpsimd_save_and_flush_cpu_state(void) * The caller may freely use the FPSIMD registers until kernel_neon_end() = is * called. */ -void kernel_neon_begin(void) +void kernel_neon_begin(struct user_fpsimd_state *s) { if (WARN_ON(!system_supports_fpsimd())) return; @@ -1866,8 +1869,16 @@ void kernel_neon_begin(void) * mode in task context. So in this case, setting the flag here * is always appropriate. */ - if (IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()) + if (IS_ENABLED(CONFIG_PREEMPT_RT) || !in_serving_softirq()) { + /* + * Record the caller provided buffer as the kernel mode + * FP/SIMD buffer for this task, so that the state can + * be preserved and restored on a context switch. + */ + if (cmpxchg(¤t->thread.kernel_fpsimd_state, NULL, s)) + BUG(); set_thread_flag(TIF_KERNEL_FPSTATE); + } } =20 /* Invalidate any task state remaining in the fpsimd regs: */ @@ -1886,7 +1897,7 @@ EXPORT_SYMBOL_GPL(kernel_neon_begin); * The caller must not use the FPSIMD registers after this function is cal= led, * unless kernel_neon_begin() is called again in the meantime. */ -void kernel_neon_end(void) +void kernel_neon_end(struct user_fpsimd_state *s) { if (!system_supports_fpsimd()) return; @@ -1899,8 +1910,9 @@ void kernel_neon_end(void) if (!IS_ENABLED(CONFIG_PREEMPT_RT) && in_serving_softirq() && test_thread_flag(TIF_KERNEL_FPSTATE)) fpsimd_load_kernel_state(current); - else - clear_thread_flag(TIF_KERNEL_FPSTATE); + else if (test_and_clear_thread_flag(TIF_KERNEL_FPSTATE)) + if (cmpxchg(¤t->thread.kernel_fpsimd_state, s, NULL) !=3D s) + BUG(); } EXPORT_SYMBOL_GPL(kernel_neon_end); =20 @@ -1936,7 +1948,7 @@ void __efi_fpsimd_begin(void) WARN_ON(preemptible()); =20 if (may_use_simd()) { - kernel_neon_begin(); + kernel_neon_begin(&efi_fpsimd_state); } else { /* * If !efi_sve_state, SVE can't be in use yet and doesn't need @@ -1985,7 +1997,7 @@ void __efi_fpsimd_end(void) return; =20 if (!efi_fpsimd_state_used) { - kernel_neon_end(); + kernel_neon_end(&efi_fpsimd_state); } else { if (system_supports_sve() && efi_sve_state_used) { bool ffr =3D true; --=20 2.51.0.618.g983fd99d29-goog