From nobody Tue Oct 7 08:37:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA36D2367C0; Sat, 12 Jul 2025 23:26:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752362783; cv=none; b=nA7gM04uNrWaYzxfnVZuL1f+tN1IpvaSz1DYfo/+7oM3RWLrb5CakwX2e99Ya6QbJ3cRBa8scw8cKtr1jOB4vJzASuwi9f4JiNg9I6B3U82FkpKs0UKvIrX8MFk8IzwSEqRVy0RLuLaQYTmR4GVAGRW8YZJqLc9nA7q1SuBHDQw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752362783; c=relaxed/simple; bh=bpokzlR5aRrtAf8xhNRcFoywovQO/qQZG9FKuitvbCw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FLmuHzcb1LOw9f2tAHTiUxtmp7LBNMusIoNj2kY0I03jZUpvbEXrkYUyahBGEhIe+svGEzsiW8y02K/NlM8N3cnpmr3AWtxO0f/yw9ozFGDFVr+Z5DgUmdMVHXZAqQnQatzUcA9BQGxoiU3aokKNKXkuD/VbICsUw8X/0EjjM5A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VHp+oJL4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VHp+oJL4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3C9FBC4CEF7; Sat, 12 Jul 2025 23:26:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1752362783; bh=bpokzlR5aRrtAf8xhNRcFoywovQO/qQZG9FKuitvbCw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VHp+oJL4lww+HdUgf7FStuudDaEDe4dNiT/XClKNlG+hNGTx3nTDtMRzMd79RvUAd HwebO+Scx1RN0yl+EavxREVlCgi0yunL5T+KDzGRcaBzPJucUQ61pK3nZp3k4bcrHo B4T01D1397+RVSmZY9cGBfKhOkR65hNiE6gHEukV5u2yIw/VQTHffDnqfIAJSYpaa9 LCnzRFnURojiEODMeMQr8Ai/7CAYsKOnDnHKmX0gmrUQ3YtrvZd/Bwm2hym+38atew qKMOlBrFqCv1I5eL+wfRHpQSNTYlpRLNtjr9dEQZs22Dw8Xfke2XInPk55SYESOx/I 0V9QcqcOg8o5A== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, Eric Biggers Subject: [PATCH 07/26] lib/crypto: arm/sha1: Migrate optimized code into library Date: Sat, 12 Jul 2025 16:22:58 -0700 Message-ID: <20250712232329.818226-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250712232329.818226-1-ebiggers@kernel.org> References: <20250712232329.818226-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Instead of exposing the arm-optimized SHA-1 code via arm-specific crypto_shash algorithms, instead just implement the sha1_blocks() library function. This is much simpler, it makes the SHA-1 library functions be arm-optimized, and it fixes the longstanding issue where the arm-optimized SHA-1 code was disabled by default. SHA-1 still remains available through crypto_shash, but individual architectures no longer need to handle it. To match sha1_blocks(), change the type of the nblocks parameter of the assembly functions from int to size_t. The assembly functions actually already treated it as size_t. Signed-off-by: Eric Biggers --- arch/arm/configs/exynos_defconfig | 1 - arch/arm/configs/milbeaut_m10v_defconfig | 2 - arch/arm/configs/multi_v7_defconfig | 2 - arch/arm/configs/omap2plus_defconfig | 1 - arch/arm/configs/pxa_defconfig | 1 - arch/arm/crypto/Kconfig | 31 ------- arch/arm/crypto/Makefile | 6 -- arch/arm/crypto/sha1-ce-glue.c | 72 ---------------- arch/arm/crypto/sha1_glue.c | 75 ----------------- arch/arm/crypto/sha1_neon_glue.c | 83 ------------------- lib/crypto/Kconfig | 1 + lib/crypto/Makefile | 5 ++ .../crypto/arm}/sha1-armv4-large.S | 0 .../crypto/arm}/sha1-armv7-neon.S | 13 ++- .../crypto =3D> lib/crypto/arm}/sha1-ce-core.S | 4 +- lib/crypto/arm/sha1.h | 46 ++++++++++ 16 files changed, 60 insertions(+), 283 deletions(-) delete mode 100644 arch/arm/crypto/sha1-ce-glue.c delete mode 100644 arch/arm/crypto/sha1_glue.c delete mode 100644 arch/arm/crypto/sha1_neon_glue.c rename {arch/arm/crypto =3D> lib/crypto/arm}/sha1-armv4-large.S (100%) rename {arch/arm/crypto =3D> lib/crypto/arm}/sha1-armv7-neon.S (98%) rename {arch/arm/crypto =3D> lib/crypto/arm}/sha1-ce-core.S (96%) create mode 100644 lib/crypto/arm/sha1.h diff --git a/arch/arm/configs/exynos_defconfig b/arch/arm/configs/exynos_de= fconfig index d58e300693045..6915c766923a2 100644 --- a/arch/arm/configs/exynos_defconfig +++ b/arch/arm/configs/exynos_defconfig @@ -361,11 +361,10 @@ CONFIG_CRYPTO_LZO=3Dm CONFIG_CRYPTO_LZ4=3Dm CONFIG_CRYPTO_USER_API_HASH=3Dm CONFIG_CRYPTO_USER_API_SKCIPHER=3Dm CONFIG_CRYPTO_USER_API_RNG=3Dm CONFIG_CRYPTO_USER_API_AEAD=3Dm -CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm CONFIG_CRYPTO_DEV_EXYNOS_RNG=3Dy CONFIG_CRYPTO_DEV_S5P=3Dy CONFIG_DMA_CMA=3Dy diff --git a/arch/arm/configs/milbeaut_m10v_defconfig b/arch/arm/configs/mi= lbeaut_m10v_defconfig index 8ebf8bd872fe8..a3be0b2ede09c 100644 --- a/arch/arm/configs/milbeaut_m10v_defconfig +++ b/arch/arm/configs/milbeaut_m10v_defconfig @@ -96,12 +96,10 @@ CONFIG_KEYS=3Dy CONFIG_CRYPTO_SELFTESTS=3Dy # CONFIG_CRYPTO_ECHAINIV is not set CONFIG_CRYPTO_AES=3Dy CONFIG_CRYPTO_SEQIV=3Dm CONFIG_CRYPTO_GHASH_ARM_CE=3Dm -CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm -CONFIG_CRYPTO_SHA1_ARM_CE=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_AES_ARM_CE=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm # CONFIG_CRYPTO_HW is not set diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v= 7_defconfig index 3fd07e864ca85..fb63f487a6232 100644 --- a/arch/arm/configs/multi_v7_defconfig +++ b/arch/arm/configs/multi_v7_defconfig @@ -1278,12 +1278,10 @@ CONFIG_CRYPTO_USER=3Dm CONFIG_CRYPTO_USER_API_HASH=3Dm CONFIG_CRYPTO_USER_API_SKCIPHER=3Dm CONFIG_CRYPTO_USER_API_RNG=3Dm CONFIG_CRYPTO_USER_API_AEAD=3Dm CONFIG_CRYPTO_GHASH_ARM_CE=3Dm -CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm -CONFIG_CRYPTO_SHA1_ARM_CE=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_AES_ARM_CE=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm CONFIG_CRYPTO_DEV_SUN4I_SS=3Dm diff --git a/arch/arm/configs/omap2plus_defconfig b/arch/arm/configs/omap2p= lus_defconfig index 530dfb8338c98..0464676379013 100644 --- a/arch/arm/configs/omap2plus_defconfig +++ b/arch/arm/configs/omap2plus_defconfig @@ -702,11 +702,10 @@ CONFIG_ROOT_NFS=3Dy CONFIG_NLS_CODEPAGE_437=3Dy CONFIG_NLS_ISO8859_1=3Dy CONFIG_SECURITY=3Dy CONFIG_CRYPTO_MICHAEL_MIC=3Dy CONFIG_CRYPTO_GHASH_ARM_CE=3Dm -CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm CONFIG_CRYPTO_DEV_OMAP=3Dm CONFIG_CRYPTO_DEV_OMAP_SHAM=3Dm diff --git a/arch/arm/configs/pxa_defconfig b/arch/arm/configs/pxa_defconfig index eaa44574d4a64..1a80602c12845 100644 --- a/arch/arm/configs/pxa_defconfig +++ b/arch/arm/configs/pxa_defconfig @@ -656,11 +656,10 @@ CONFIG_CRYPTO_TGR192=3Dm CONFIG_CRYPTO_WP512=3Dm CONFIG_CRYPTO_ANUBIS=3Dm CONFIG_CRYPTO_XCBC=3Dm CONFIG_CRYPTO_DEFLATE=3Dy CONFIG_CRYPTO_LZO=3Dy -CONFIG_CRYPTO_SHA1_ARM=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_FONTS=3Dy CONFIG_FONT_8x8=3Dy CONFIG_FONT_8x16=3Dy CONFIG_FONT_6x11=3Dy diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig index a18f97f1597cb..1e5f3cdf691c4 100644 --- a/arch/arm/crypto/Kconfig +++ b/arch/arm/crypto/Kconfig @@ -60,41 +60,10 @@ config CRYPTO_BLAKE2B_NEON On ARM processors that have NEON support but not the ARMv8 Crypto Extensions, typically this BLAKE2b implementation is much faster than the SHA-2 family and slightly faster than SHA-1. =20 -config CRYPTO_SHA1_ARM - tristate "Hash functions: SHA-1" - select CRYPTO_SHA1 - select CRYPTO_HASH - help - SHA-1 secure hash algorithm (FIPS 180) - - Architecture: arm - -config CRYPTO_SHA1_ARM_NEON - tristate "Hash functions: SHA-1 (NEON)" - depends on KERNEL_MODE_NEON - select CRYPTO_SHA1_ARM - select CRYPTO_SHA1 - select CRYPTO_HASH - help - SHA-1 secure hash algorithm (FIPS 180) - - Architecture: arm using - - NEON (Advanced SIMD) extensions - -config CRYPTO_SHA1_ARM_CE - tristate "Hash functions: SHA-1 (ARMv8 Crypto Extensions)" - depends on KERNEL_MODE_NEON - select CRYPTO_SHA1_ARM - select CRYPTO_HASH - help - SHA-1 secure hash algorithm (FIPS 180) - - Architecture: arm using ARMv8 Crypto Extensions - config CRYPTO_AES_ARM tristate "Ciphers: AES" select CRYPTO_ALGAPI select CRYPTO_AES help diff --git a/arch/arm/crypto/Makefile b/arch/arm/crypto/Makefile index 78a4042d8761c..4f23999ae17df 100644 --- a/arch/arm/crypto/Makefile +++ b/arch/arm/crypto/Makefile @@ -3,25 +3,19 @@ # Arch-specific CryptoAPI modules. # =20 obj-$(CONFIG_CRYPTO_AES_ARM) +=3D aes-arm.o obj-$(CONFIG_CRYPTO_AES_ARM_BS) +=3D aes-arm-bs.o -obj-$(CONFIG_CRYPTO_SHA1_ARM) +=3D sha1-arm.o -obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) +=3D sha1-arm-neon.o obj-$(CONFIG_CRYPTO_BLAKE2B_NEON) +=3D blake2b-neon.o obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) +=3D nhpoly1305-neon.o obj-$(CONFIG_CRYPTO_CURVE25519_NEON) +=3D curve25519-neon.o =20 obj-$(CONFIG_CRYPTO_AES_ARM_CE) +=3D aes-arm-ce.o -obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) +=3D sha1-arm-ce.o obj-$(CONFIG_CRYPTO_GHASH_ARM_CE) +=3D ghash-arm-ce.o =20 aes-arm-y :=3D aes-cipher-core.o aes-cipher-glue.o aes-arm-bs-y :=3D aes-neonbs-core.o aes-neonbs-glue.o -sha1-arm-y :=3D sha1-armv4-large.o sha1_glue.o -sha1-arm-neon-y :=3D sha1-armv7-neon.o sha1_neon_glue.o blake2b-neon-y :=3D blake2b-neon-core.o blake2b-neon-glue.o -sha1-arm-ce-y :=3D sha1-ce-core.o sha1-ce-glue.o aes-arm-ce-y :=3D aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y :=3D ghash-ce-core.o ghash-ce-glue.o nhpoly1305-neon-y :=3D nh-neon-core.o nhpoly1305-neon-glue.o curve25519-neon-y :=3D curve25519-core.o curve25519-glue.o diff --git a/arch/arm/crypto/sha1-ce-glue.c b/arch/arm/crypto/sha1-ce-glue.c deleted file mode 100644 index fac07a4799de6..0000000000000 --- a/arch/arm/crypto/sha1-ce-glue.c +++ /dev/null @@ -1,72 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * sha1-ce-glue.c - SHA-1 secure hash using ARMv8 Crypto Extensions - * - * Copyright (C) 2015 Linaro Ltd - */ - -#include -#include -#include -#include -#include -#include -#include - -MODULE_DESCRIPTION("SHA1 secure hash using ARMv8 Crypto Extensions"); -MODULE_AUTHOR("Ard Biesheuvel "); -MODULE_LICENSE("GPL v2"); - -asmlinkage void sha1_ce_transform(struct sha1_state *sst, u8 const *src, - int blocks); - -static int sha1_ce_update(struct shash_desc *desc, const u8 *data, - unsigned int len) -{ - int remain; - - kernel_neon_begin(); - remain =3D sha1_base_do_update_blocks(desc, data, len, sha1_ce_transform); - kernel_neon_end(); - - return remain; -} - -static int sha1_ce_finup(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - kernel_neon_begin(); - sha1_base_do_finup(desc, data, len, sha1_ce_transform); - kernel_neon_end(); - - return sha1_base_finish(desc, out); -} - -static struct shash_alg alg =3D { - .init =3D sha1_base_init, - .update =3D sha1_ce_update, - .finup =3D sha1_ce_finup, - .descsize =3D SHA1_STATE_SIZE, - .digestsize =3D SHA1_DIGEST_SIZE, - .base =3D { - .cra_name =3D "sha1", - .cra_driver_name =3D "sha1-ce", - .cra_priority =3D 200, - .cra_flags =3D CRYPTO_AHASH_ALG_BLOCK_ONLY, - .cra_blocksize =3D SHA1_BLOCK_SIZE, - .cra_module =3D THIS_MODULE, - } -}; - -static int __init sha1_ce_mod_init(void) -{ - return crypto_register_shash(&alg); -} - -static void __exit sha1_ce_mod_fini(void) -{ - crypto_unregister_shash(&alg); -} - -module_cpu_feature_match(SHA1, sha1_ce_mod_init); -module_exit(sha1_ce_mod_fini); diff --git a/arch/arm/crypto/sha1_glue.c b/arch/arm/crypto/sha1_glue.c deleted file mode 100644 index 255da00c7d98a..0000000000000 --- a/arch/arm/crypto/sha1_glue.c +++ /dev/null @@ -1,75 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * Cryptographic API. - * Glue code for the SHA1 Secure Hash Algorithm assembler implementation - * - * This file is based on sha1_generic.c and sha1_ssse3_glue.c - * - * Copyright (c) Alan Smithee. - * Copyright (c) Andrew McDonald - * Copyright (c) Jean-Francois Dive - * Copyright (c) Mathias Krause - */ - -#include -#include -#include -#include -#include - -asmlinkage void sha1_block_data_order(struct sha1_state *digest, - const u8 *data, int rounds); - -static int sha1_update_arm(struct shash_desc *desc, const u8 *data, - unsigned int len) -{ - /* make sure signature matches sha1_block_fn() */ - BUILD_BUG_ON(offsetof(struct sha1_state, state) !=3D 0); - - return sha1_base_do_update_blocks(desc, data, len, - sha1_block_data_order); -} - -static int sha1_finup_arm(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - sha1_base_do_finup(desc, data, len, sha1_block_data_order); - return sha1_base_finish(desc, out); -} - -static struct shash_alg alg =3D { - .digestsize =3D SHA1_DIGEST_SIZE, - .init =3D sha1_base_init, - .update =3D sha1_update_arm, - .finup =3D sha1_finup_arm, - .descsize =3D SHA1_STATE_SIZE, - .base =3D { - .cra_name =3D "sha1", - .cra_driver_name=3D "sha1-asm", - .cra_priority =3D 150, - .cra_flags =3D CRYPTO_AHASH_ALG_BLOCK_ONLY, - .cra_blocksize =3D SHA1_BLOCK_SIZE, - .cra_module =3D THIS_MODULE, - } -}; - - -static int __init sha1_mod_init(void) -{ - return crypto_register_shash(&alg); -} - - -static void __exit sha1_mod_fini(void) -{ - crypto_unregister_shash(&alg); -} - - -module_init(sha1_mod_init); -module_exit(sha1_mod_fini); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("SHA1 Secure Hash Algorithm (ARM)"); -MODULE_ALIAS_CRYPTO("sha1"); -MODULE_AUTHOR("David McCullough "); diff --git a/arch/arm/crypto/sha1_neon_glue.c b/arch/arm/crypto/sha1_neon_g= lue.c deleted file mode 100644 index d321850f22a6d..0000000000000 --- a/arch/arm/crypto/sha1_neon_glue.c +++ /dev/null @@ -1,83 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * Glue code for the SHA1 Secure Hash Algorithm assembler implementation u= sing - * ARM NEON instructions. - * - * Copyright =C2=A9 2014 Jussi Kivilinna - * - * This file is based on sha1_generic.c and sha1_ssse3_glue.c: - * Copyright (c) Alan Smithee. - * Copyright (c) Andrew McDonald - * Copyright (c) Jean-Francois Dive - * Copyright (c) Mathias Krause - * Copyright (c) Chandramouli Narayanan - */ - -#include -#include -#include -#include -#include -#include - -asmlinkage void sha1_transform_neon(struct sha1_state *state_h, - const u8 *data, int rounds); - -static int sha1_neon_update(struct shash_desc *desc, const u8 *data, - unsigned int len) -{ - int remain; - - kernel_neon_begin(); - remain =3D sha1_base_do_update_blocks(desc, data, len, - sha1_transform_neon); - kernel_neon_end(); - - return remain; -} - -static int sha1_neon_finup(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - kernel_neon_begin(); - sha1_base_do_finup(desc, data, len, sha1_transform_neon); - kernel_neon_end(); - - return sha1_base_finish(desc, out); -} - -static struct shash_alg alg =3D { - .digestsize =3D SHA1_DIGEST_SIZE, - .init =3D sha1_base_init, - .update =3D sha1_neon_update, - .finup =3D sha1_neon_finup, - .descsize =3D SHA1_STATE_SIZE, - .base =3D { - .cra_name =3D "sha1", - .cra_driver_name =3D "sha1-neon", - .cra_priority =3D 250, - .cra_flags =3D CRYPTO_AHASH_ALG_BLOCK_ONLY, - .cra_blocksize =3D SHA1_BLOCK_SIZE, - .cra_module =3D THIS_MODULE, - } -}; - -static int __init sha1_neon_mod_init(void) -{ - if (!cpu_has_neon()) - return -ENODEV; - - return crypto_register_shash(&alg); -} - -static void __exit sha1_neon_mod_fini(void) -{ - crypto_unregister_shash(&alg); -} - -module_init(sha1_neon_mod_init); -module_exit(sha1_neon_mod_fini); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("SHA1 Secure Hash Algorithm, NEON accelerated"); -MODULE_ALIAS_CRYPTO("sha1"); diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig index ff54c9a631f86..5c1bfa02fa349 100644 --- a/lib/crypto/Kconfig +++ b/lib/crypto/Kconfig @@ -144,10 +144,11 @@ config CRYPTO_LIB_SHA1 the functions from . =20 config CRYPTO_LIB_SHA1_ARCH bool depends on CRYPTO_LIB_SHA1 && !UML + default y if ARM =20 config CRYPTO_LIB_SHA256 tristate help Enable the SHA-256 library interface. This interface may be fulfilled diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile index c96abfc4b9d3b..e10a84a6dda6a 100644 --- a/lib/crypto/Makefile +++ b/lib/crypto/Makefile @@ -71,10 +71,15 @@ libpoly1305-generic-y +=3D poly1305-generic.o =20 obj-$(CONFIG_CRYPTO_LIB_SHA1) +=3D libsha1.o libsha1-y :=3D sha1.o ifeq ($(CONFIG_CRYPTO_LIB_SHA1_ARCH),y) CFLAGS_sha1.o +=3D -I$(src)/$(SRCARCH) +ifeq ($(CONFIG_ARM),y) +libsha1-y +=3D arm/sha1-armv4-large.o +libsha1-$(CONFIG_KERNEL_MODE_NEON) +=3D arm/sha1-armv7-neon.o \ + arm/sha1-ce-core.o +endif endif # CONFIG_CRYPTO_LIB_SHA1_ARCH =20 ##########################################################################= ###### =20 obj-$(CONFIG_CRYPTO_LIB_SHA256) +=3D libsha256.o diff --git a/arch/arm/crypto/sha1-armv4-large.S b/lib/crypto/arm/sha1-armv4= -large.S similarity index 100% rename from arch/arm/crypto/sha1-armv4-large.S rename to lib/crypto/arm/sha1-armv4-large.S diff --git a/arch/arm/crypto/sha1-armv7-neon.S b/lib/crypto/arm/sha1-armv7-= neon.S similarity index 98% rename from arch/arm/crypto/sha1-armv7-neon.S rename to lib/crypto/arm/sha1-armv7-neon.S index 28d816a6a5307..6edba3ab62e8b 100644 --- a/arch/arm/crypto/sha1-armv7-neon.S +++ b/lib/crypto/arm/sha1-armv7-neon.S @@ -280,22 +280,21 @@ #define WPRECALC_32_79_9(i,W,W_m04,W_m08,W_m12,W_m16,W_m20,W_m24,W_m28) \ vst1.32 {tmp0}, [RWK]; =20 =20 /* - * Transform nblks*64 bytes (nblks*16 32-bit words) at DATA. + * Transform nblocks*64 bytes (nblocks*16 32-bit words) at DATA. * - * unsigned int - * sha1_transform_neon (void *ctx, const unsigned char *data, - * unsigned int nblks) + * void sha1_transform_neon(struct sha1_block_state *state, + * const u8 *data, size_t nblocks); */ .align 3 ENTRY(sha1_transform_neon) /* input: - * r0: ctx, CTX - * r1: data (64*nblks bytes) - * r2: nblks + * r0: state + * r1: data (64*nblocks bytes) + * r2: nblocks */ =20 cmp RNBLKS, #0; beq .Ldo_nothing; =20 diff --git a/arch/arm/crypto/sha1-ce-core.S b/lib/crypto/arm/sha1-ce-core.S similarity index 96% rename from arch/arm/crypto/sha1-ce-core.S rename to lib/crypto/arm/sha1-ce-core.S index 8a702e051738a..2de40dd25e47e 100644 --- a/arch/arm/crypto/sha1-ce-core.S +++ b/lib/crypto/arm/sha1-ce-core.S @@ -57,12 +57,12 @@ .word 0x6ed9eba1, 0x6ed9eba1, 0x6ed9eba1, 0x6ed9eba1 .word 0x8f1bbcdc, 0x8f1bbcdc, 0x8f1bbcdc, 0x8f1bbcdc .word 0xca62c1d6, 0xca62c1d6, 0xca62c1d6, 0xca62c1d6 =20 /* - * void sha1_ce_transform(struct sha1_state *sst, u8 const *src, - * int blocks); + * void sha1_ce_transform(struct sha1_block_state *state, + * const u8 *data, size_t nblocks); */ ENTRY(sha1_ce_transform) /* load round constants */ adr ip, .Lsha1_rcon vld1.32 {k0-k1}, [ip, :128]! diff --git a/lib/crypto/arm/sha1.h b/lib/crypto/arm/sha1.h new file mode 100644 index 0000000000000..b177b71f5530a --- /dev/null +++ b/lib/crypto/arm/sha1.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * SHA-1 optimized for ARM + * + * Copyright 2025 Google LLC + */ +#include +#include + +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_ce); + +asmlinkage void sha1_block_data_order(struct sha1_block_state *state, + const u8 *data, size_t nblocks); +asmlinkage void sha1_transform_neon(struct sha1_block_state *state, + const u8 *data, size_t nblocks); +asmlinkage void sha1_ce_transform(struct sha1_block_state *state, + const u8 *data, size_t nblocks); + +static void sha1_blocks(struct sha1_block_state *state, + const u8 *data, size_t nblocks) +{ + if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && + static_branch_likely(&have_neon) && may_use_simd()) { + kernel_neon_begin(); + if (static_branch_likely(&have_ce)) + sha1_ce_transform(state, data, nblocks); + else + sha1_transform_neon(state, data, nblocks); + kernel_neon_end(); + } else { + sha1_block_data_order(state, data, nblocks); + } +} + +#ifdef CONFIG_KERNEL_MODE_NEON +#define sha1_mod_init_arch sha1_mod_init_arch +static inline void sha1_mod_init_arch(void) +{ + if (elf_hwcap & HWCAP_NEON) { + static_branch_enable(&have_neon); + if (elf_hwcap2 & HWCAP2_SHA1) + static_branch_enable(&have_ce); + } +} +#endif /* CONFIG_KERNEL_MODE_NEON */ --=20 2.50.1