From nobody Mon Feb 9 03:35:47 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 708A21A238E; Wed, 11 Jun 2025 02:12:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749607976; cv=none; b=Ru9tJ1g1T2MbqNLZf0Cd2ntD9jQe/4lM565mmb5OTWS+t6U9kKAyybTFuerPrPFuLdkLf1gHFwGWwjGOaHCyn3/zDL4BJMqAupb3j7SbssChyOVK4JAjzpZqlt3W4wVa9zUo7N9RNtVDx8a1xNNTBCEHXm2suV1cvX2+kSlah5M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749607976; c=relaxed/simple; bh=6+kbm73zE6tAcXQGb7C8c8diT3Co3HmNcJy1QJKMO9Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EjP9JVOQeD2P+7cRfyjoGCiDuU1rlSq0lDvK5M/um4i4MACwyBX+Ha1PO7cWlWtL4OBuwOW8AXI/CF7jjtSZ19a4qMFh3rOxOtn1iS5v4lSuGI2gpd5o0JkzFrsXuzxvTYSmz9YIDhK8bNReRlpa+F72LSINJ/UbDI2XsJldex4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DcJsO//4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DcJsO//4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A54EFC4CEF7; Wed, 11 Jun 2025 02:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1749607976; bh=6+kbm73zE6tAcXQGb7C8c8diT3Co3HmNcJy1QJKMO9Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DcJsO//4X/t8dZG1V7huwnUYCeIQdMdbiX05zGEkTihOmVazQYBo+USUTpUdsAOpy p2JHfjVhBmi7itENOlptKzLBAk9dfZFILBZWvjPlncGTkgr+qxi7wr47JtiwFAdoHx Z9nvNxEWEuxR39/4Zjb55xaqK213o5uro+EkGdb/D/qX9xeY0ShX9jjMWGfDVXLX/Q Q0fn24GqEQ4uMlfiUClli9eZdxLp37f/HbJKNhxEKza6sN0VH3DXy2O5utnkCH9pni D/mvYm6l3bwYvyn9yXA0D1oIY5ZGWgj7Rh28Cau9ivzP5sKJYrSqETyLXSpPUUzcrM V8uTm9RngR/DA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, x86@kernel.org, Ard Biesheuvel , "Jason A . Donenfeld " , Linus Torvalds Subject: [PATCH 08/16] lib/crypto/sha512: migrate arm-optimized SHA-512 code to library Date: Tue, 10 Jun 2025 19:09:15 -0700 Message-ID: <20250611020923.1482701-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250611020923.1482701-1-ebiggers@kernel.org> References: <20250611020923.1482701-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Instead of exposing the arm-optimized SHA-512 code via arm-specific crypto_shash algorithms, instead just implement the sha512_blocks() library function. This is much simpler, it makes the SHA-512 (and SHA-384) library functions be arm-optimized, and it fixes the longstanding issue where the arm-optimized SHA-512 code was disabled by default. SHA-512 still remains available through crypto_shash, but individual architectures no longer need to handle it. To match sha512_blocks(), change the type of the nblocks parameter of the assembly functions from int to size_t. The assembly functions actually already treated it as size_t. Signed-off-by: Eric Biggers --- arch/arm/configs/exynos_defconfig | 1 - arch/arm/configs/milbeaut_m10v_defconfig | 1 - arch/arm/configs/multi_v7_defconfig | 1 - arch/arm/configs/omap2plus_defconfig | 1 - arch/arm/configs/pxa_defconfig | 1 - arch/arm/crypto/Kconfig | 10 -- arch/arm/crypto/Makefile | 15 --- arch/arm/crypto/sha512-glue.c | 110 ------------------ arch/arm/crypto/sha512-neon-glue.c | 75 ------------ arch/arm/crypto/sha512.h | 3 - lib/crypto/Kconfig | 1 + lib/crypto/Makefile | 14 +++ lib/crypto/arm/.gitignore | 2 + .../crypto =3D> lib/crypto/arm}/sha512-armv4.pl | 0 lib/crypto/arm/sha512.h | 38 ++++++ 15 files changed, 55 insertions(+), 218 deletions(-) delete mode 100644 arch/arm/crypto/sha512-glue.c delete mode 100644 arch/arm/crypto/sha512-neon-glue.c delete mode 100644 arch/arm/crypto/sha512.h create mode 100644 lib/crypto/arm/.gitignore rename {arch/arm/crypto =3D> lib/crypto/arm}/sha512-armv4.pl (100%) create mode 100644 lib/crypto/arm/sha512.h diff --git a/arch/arm/configs/exynos_defconfig b/arch/arm/configs/exynos_de= fconfig index f71af368674cf..d58e300693045 100644 --- a/arch/arm/configs/exynos_defconfig +++ b/arch/arm/configs/exynos_defconfig @@ -362,11 +362,10 @@ CONFIG_CRYPTO_LZ4=3Dm CONFIG_CRYPTO_USER_API_HASH=3Dm CONFIG_CRYPTO_USER_API_SKCIPHER=3Dm CONFIG_CRYPTO_USER_API_RNG=3Dm CONFIG_CRYPTO_USER_API_AEAD=3Dm CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm -CONFIG_CRYPTO_SHA512_ARM=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm CONFIG_CRYPTO_DEV_EXYNOS_RNG=3Dy CONFIG_CRYPTO_DEV_S5P=3Dy CONFIG_DMA_CMA=3Dy diff --git a/arch/arm/configs/milbeaut_m10v_defconfig b/arch/arm/configs/mi= lbeaut_m10v_defconfig index 242e7d5a3f682..8ebf8bd872fe8 100644 --- a/arch/arm/configs/milbeaut_m10v_defconfig +++ b/arch/arm/configs/milbeaut_m10v_defconfig @@ -98,11 +98,10 @@ CONFIG_CRYPTO_SELFTESTS=3Dy CONFIG_CRYPTO_AES=3Dy CONFIG_CRYPTO_SEQIV=3Dm CONFIG_CRYPTO_GHASH_ARM_CE=3Dm CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm CONFIG_CRYPTO_SHA1_ARM_CE=3Dm -CONFIG_CRYPTO_SHA512_ARM=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_AES_ARM_CE=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm # CONFIG_CRYPTO_HW is not set diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v= 7_defconfig index 50c170b4619f7..3fd07e864ca85 100644 --- a/arch/arm/configs/multi_v7_defconfig +++ b/arch/arm/configs/multi_v7_defconfig @@ -1280,11 +1280,10 @@ CONFIG_CRYPTO_USER_API_SKCIPHER=3Dm CONFIG_CRYPTO_USER_API_RNG=3Dm CONFIG_CRYPTO_USER_API_AEAD=3Dm CONFIG_CRYPTO_GHASH_ARM_CE=3Dm CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm CONFIG_CRYPTO_SHA1_ARM_CE=3Dm -CONFIG_CRYPTO_SHA512_ARM=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_AES_ARM_CE=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm CONFIG_CRYPTO_DEV_SUN4I_SS=3Dm diff --git a/arch/arm/configs/omap2plus_defconfig b/arch/arm/configs/omap2p= lus_defconfig index 9f9780c8e62aa..530dfb8338c98 100644 --- a/arch/arm/configs/omap2plus_defconfig +++ b/arch/arm/configs/omap2plus_defconfig @@ -703,11 +703,10 @@ CONFIG_NLS_CODEPAGE_437=3Dy CONFIG_NLS_ISO8859_1=3Dy CONFIG_SECURITY=3Dy CONFIG_CRYPTO_MICHAEL_MIC=3Dy CONFIG_CRYPTO_GHASH_ARM_CE=3Dm CONFIG_CRYPTO_SHA1_ARM_NEON=3Dm -CONFIG_CRYPTO_SHA512_ARM=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_CRYPTO_AES_ARM_BS=3Dm CONFIG_CRYPTO_CHACHA20_NEON=3Dm CONFIG_CRYPTO_DEV_OMAP=3Dm CONFIG_CRYPTO_DEV_OMAP_SHAM=3Dm diff --git a/arch/arm/configs/pxa_defconfig b/arch/arm/configs/pxa_defconfig index ff29c5b0e9c93..eaa44574d4a64 100644 --- a/arch/arm/configs/pxa_defconfig +++ b/arch/arm/configs/pxa_defconfig @@ -657,11 +657,10 @@ CONFIG_CRYPTO_WP512=3Dm CONFIG_CRYPTO_ANUBIS=3Dm CONFIG_CRYPTO_XCBC=3Dm CONFIG_CRYPTO_DEFLATE=3Dy CONFIG_CRYPTO_LZO=3Dy CONFIG_CRYPTO_SHA1_ARM=3Dm -CONFIG_CRYPTO_SHA512_ARM=3Dm CONFIG_CRYPTO_AES_ARM=3Dm CONFIG_FONTS=3Dy CONFIG_FONT_8x8=3Dy CONFIG_FONT_8x16=3Dy CONFIG_FONT_6x11=3Dy diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig index 7efb9a8596e4e..a18f97f1597cb 100644 --- a/arch/arm/crypto/Kconfig +++ b/arch/arm/crypto/Kconfig @@ -91,20 +91,10 @@ config CRYPTO_SHA1_ARM_CE help SHA-1 secure hash algorithm (FIPS 180) =20 Architecture: arm using ARMv8 Crypto Extensions =20 -config CRYPTO_SHA512_ARM - tristate "Hash functions: SHA-384 and SHA-512 (NEON)" - select CRYPTO_HASH - depends on !CPU_V7M - help - SHA-384 and SHA-512 secure hash algorithms (FIPS 180) - - Architecture: arm using - - NEON (Advanced SIMD) extensions - config CRYPTO_AES_ARM tristate "Ciphers: AES" select CRYPTO_ALGAPI select CRYPTO_AES help diff --git a/arch/arm/crypto/Makefile b/arch/arm/crypto/Makefile index 8479137c6e800..78a4042d8761c 100644 --- a/arch/arm/crypto/Makefile +++ b/arch/arm/crypto/Makefile @@ -5,11 +5,10 @@ =20 obj-$(CONFIG_CRYPTO_AES_ARM) +=3D aes-arm.o obj-$(CONFIG_CRYPTO_AES_ARM_BS) +=3D aes-arm-bs.o obj-$(CONFIG_CRYPTO_SHA1_ARM) +=3D sha1-arm.o obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) +=3D sha1-arm-neon.o -obj-$(CONFIG_CRYPTO_SHA512_ARM) +=3D sha512-arm.o obj-$(CONFIG_CRYPTO_BLAKE2B_NEON) +=3D blake2b-neon.o obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) +=3D nhpoly1305-neon.o obj-$(CONFIG_CRYPTO_CURVE25519_NEON) +=3D curve25519-neon.o =20 obj-$(CONFIG_CRYPTO_AES_ARM_CE) +=3D aes-arm-ce.o @@ -18,25 +17,11 @@ obj-$(CONFIG_CRYPTO_GHASH_ARM_CE) +=3D ghash-arm-ce.o =20 aes-arm-y :=3D aes-cipher-core.o aes-cipher-glue.o aes-arm-bs-y :=3D aes-neonbs-core.o aes-neonbs-glue.o sha1-arm-y :=3D sha1-armv4-large.o sha1_glue.o sha1-arm-neon-y :=3D sha1-armv7-neon.o sha1_neon_glue.o -sha512-arm-neon-$(CONFIG_KERNEL_MODE_NEON) :=3D sha512-neon-glue.o -sha512-arm-y :=3D sha512-core.o sha512-glue.o $(sha512-arm-neon-y) blake2b-neon-y :=3D blake2b-neon-core.o blake2b-neon-glue.o sha1-arm-ce-y :=3D sha1-ce-core.o sha1-ce-glue.o aes-arm-ce-y :=3D aes-ce-core.o aes-ce-glue.o ghash-arm-ce-y :=3D ghash-ce-core.o ghash-ce-glue.o nhpoly1305-neon-y :=3D nh-neon-core.o nhpoly1305-neon-glue.o curve25519-neon-y :=3D curve25519-core.o curve25519-glue.o - -quiet_cmd_perl =3D PERL $@ - cmd_perl =3D $(PERL) $(<) > $(@) - -$(obj)/%-core.S: $(src)/%-armv4.pl - $(call cmd,perl) - -clean-files +=3D sha512-core.S - -aflags-thumb2-$(CONFIG_THUMB2_KERNEL) :=3D -U__thumb2__ -D__thumb2__=3D1 - -AFLAGS_sha512-core.o +=3D $(aflags-thumb2-y) diff --git a/arch/arm/crypto/sha512-glue.c b/arch/arm/crypto/sha512-glue.c deleted file mode 100644 index f8a6480889b1b..0000000000000 --- a/arch/arm/crypto/sha512-glue.c +++ /dev/null @@ -1,110 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * sha512-glue.c - accelerated SHA-384/512 for ARM - * - * Copyright (C) 2015 Linaro Ltd - */ - -#include -#include -#include -#include -#include -#include -#include - -#include "sha512.h" - -MODULE_DESCRIPTION("Accelerated SHA-384/SHA-512 secure hash for ARM"); -MODULE_AUTHOR("Ard Biesheuvel "); -MODULE_LICENSE("GPL v2"); - -MODULE_ALIAS_CRYPTO("sha384"); -MODULE_ALIAS_CRYPTO("sha512"); -MODULE_ALIAS_CRYPTO("sha384-arm"); -MODULE_ALIAS_CRYPTO("sha512-arm"); - -asmlinkage void sha512_block_data_order(struct sha512_state *state, - u8 const *src, int blocks); - -static int sha512_arm_update(struct shash_desc *desc, const u8 *data, - unsigned int len) -{ - return sha512_base_do_update_blocks(desc, data, len, - sha512_block_data_order); -} - -static int sha512_arm_finup(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - sha512_base_do_finup(desc, data, len, sha512_block_data_order); - return sha512_base_finish(desc, out); -} - -static struct shash_alg sha512_arm_algs[] =3D { { - .init =3D sha384_base_init, - .update =3D sha512_arm_update, - .finup =3D sha512_arm_finup, - .descsize =3D SHA512_STATE_SIZE, - .digestsize =3D SHA384_DIGEST_SIZE, - .base =3D { - .cra_name =3D "sha384", - .cra_driver_name =3D "sha384-arm", - .cra_priority =3D 250, - .cra_flags =3D CRYPTO_AHASH_ALG_BLOCK_ONLY | - CRYPTO_AHASH_ALG_FINUP_MAX, - .cra_blocksize =3D SHA512_BLOCK_SIZE, - .cra_module =3D THIS_MODULE, - } -}, { - .init =3D sha512_base_init, - .update =3D sha512_arm_update, - .finup =3D sha512_arm_finup, - .descsize =3D SHA512_STATE_SIZE, - .digestsize =3D SHA512_DIGEST_SIZE, - .base =3D { - .cra_name =3D "sha512", - .cra_driver_name =3D "sha512-arm", - .cra_priority =3D 250, - .cra_flags =3D CRYPTO_AHASH_ALG_BLOCK_ONLY | - CRYPTO_AHASH_ALG_FINUP_MAX, - .cra_blocksize =3D SHA512_BLOCK_SIZE, - .cra_module =3D THIS_MODULE, - } -} }; - -static int __init sha512_arm_mod_init(void) -{ - int err; - - err =3D crypto_register_shashes(sha512_arm_algs, - ARRAY_SIZE(sha512_arm_algs)); - if (err) - return err; - - if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && cpu_has_neon()) { - err =3D crypto_register_shashes(sha512_neon_algs, - ARRAY_SIZE(sha512_neon_algs)); - if (err) - goto err_unregister; - } - return 0; - -err_unregister: - crypto_unregister_shashes(sha512_arm_algs, - ARRAY_SIZE(sha512_arm_algs)); - - return err; -} - -static void __exit sha512_arm_mod_fini(void) -{ - crypto_unregister_shashes(sha512_arm_algs, - ARRAY_SIZE(sha512_arm_algs)); - if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && cpu_has_neon()) - crypto_unregister_shashes(sha512_neon_algs, - ARRAY_SIZE(sha512_neon_algs)); -} - -module_init(sha512_arm_mod_init); -module_exit(sha512_arm_mod_fini); diff --git a/arch/arm/crypto/sha512-neon-glue.c b/arch/arm/crypto/sha512-ne= on-glue.c deleted file mode 100644 index bd528077fefbf..0000000000000 --- a/arch/arm/crypto/sha512-neon-glue.c +++ /dev/null @@ -1,75 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * sha512-neon-glue.c - accelerated SHA-384/512 for ARM NEON - * - * Copyright (C) 2015 Linaro Ltd - */ - -#include -#include -#include -#include -#include -#include - -#include "sha512.h" - -MODULE_ALIAS_CRYPTO("sha384-neon"); -MODULE_ALIAS_CRYPTO("sha512-neon"); - -asmlinkage void sha512_block_data_order_neon(struct sha512_state *state, - const u8 *src, int blocks); - -static int sha512_neon_update(struct shash_desc *desc, const u8 *data, - unsigned int len) -{ - int remain; - - kernel_neon_begin(); - remain =3D sha512_base_do_update_blocks(desc, data, len, - sha512_block_data_order_neon); - kernel_neon_end(); - return remain; -} - -static int sha512_neon_finup(struct shash_desc *desc, const u8 *data, - unsigned int len, u8 *out) -{ - kernel_neon_begin(); - sha512_base_do_finup(desc, data, len, sha512_block_data_order_neon); - kernel_neon_end(); - return sha512_base_finish(desc, out); -} - -struct shash_alg sha512_neon_algs[] =3D { { - .init =3D sha384_base_init, - .update =3D sha512_neon_update, - .finup =3D sha512_neon_finup, - .descsize =3D SHA512_STATE_SIZE, - .digestsize =3D SHA384_DIGEST_SIZE, - .base =3D { - .cra_name =3D "sha384", - .cra_driver_name =3D "sha384-neon", - .cra_priority =3D 300, - .cra_flags =3D CRYPTO_AHASH_ALG_BLOCK_ONLY | - CRYPTO_AHASH_ALG_FINUP_MAX, - .cra_blocksize =3D SHA384_BLOCK_SIZE, - .cra_module =3D THIS_MODULE, - - } -}, { - .init =3D sha512_base_init, - .update =3D sha512_neon_update, - .finup =3D sha512_neon_finup, - .descsize =3D SHA512_STATE_SIZE, - .digestsize =3D SHA512_DIGEST_SIZE, - .base =3D { - .cra_name =3D "sha512", - .cra_driver_name =3D "sha512-neon", - .cra_priority =3D 300, - .cra_flags =3D CRYPTO_AHASH_ALG_BLOCK_ONLY | - CRYPTO_AHASH_ALG_FINUP_MAX, - .cra_blocksize =3D SHA512_BLOCK_SIZE, - .cra_module =3D THIS_MODULE, - } -} }; diff --git a/arch/arm/crypto/sha512.h b/arch/arm/crypto/sha512.h deleted file mode 100644 index eeaee52cda69b..0000000000000 --- a/arch/arm/crypto/sha512.h +++ /dev/null @@ -1,3 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ - -extern struct shash_alg sha512_neon_algs[2]; diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig index 34b249ca3db23..83054a496f9cd 100644 --- a/lib/crypto/Kconfig +++ b/lib/crypto/Kconfig @@ -177,10 +177,11 @@ config CRYPTO_LIB_SHA512 . =20 config CRYPTO_LIB_SHA512_ARCH bool depends on CRYPTO_LIB_SHA512 + default y if ARM && !CPU_V7M =20 config CRYPTO_LIB_SM3 tristate =20 if !KMSAN # avoid false positives from assembly diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile index 7df76ab5fe692..41513bc29a5e6 100644 --- a/lib/crypto/Makefile +++ b/lib/crypto/Makefile @@ -1,9 +1,14 @@ # SPDX-License-Identifier: GPL-2.0 =20 obj-y +=3D tests/ =20 +aflags-thumb2-$(CONFIG_THUMB2_KERNEL) :=3D -U__thumb2__ -D__thumb2__=3D1 + +quiet_cmd_perlasm =3D PERLASM $@ + cmd_perlasm =3D $(PERL) $(<) > $(@) + obj-$(CONFIG_CRYPTO_LIB_UTILS) +=3D libcryptoutils.o libcryptoutils-y :=3D memneq.o utils.o =20 # chacha is used by the /dev/random driver which is always builtin obj-y +=3D chacha.o @@ -64,10 +69,19 @@ libsha256-generic-y :=3D sha256-generic.o =20 obj-$(CONFIG_CRYPTO_LIB_SHA512) +=3D libsha512.o libsha512-y :=3D sha512.o ifeq ($(CONFIG_CRYPTO_LIB_SHA512_ARCH),y) CFLAGS_sha512.o +=3D -I$(src)/$(SRCARCH) + +ifeq ($(CONFIG_ARM),y) +libsha512-y +=3D arm/sha512-core.o +$(obj)/arm/sha512-core.S: $(src)/arm/sha512-armv4.pl + $(call cmd,perlasm) +clean-files +=3D arm/sha512-core.S +AFLAGS_arm/sha512-core.o +=3D $(aflags-thumb2-y) +endif + endif # CONFIG_CRYPTO_LIB_SHA512_ARCH =20 obj-$(CONFIG_MPILIB) +=3D mpi/ =20 obj-$(CONFIG_CRYPTO_SELFTESTS) +=3D simd.o diff --git a/lib/crypto/arm/.gitignore b/lib/crypto/arm/.gitignore new file mode 100644 index 0000000000000..670a4d97b5684 --- /dev/null +++ b/lib/crypto/arm/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +sha512-core.S diff --git a/arch/arm/crypto/sha512-armv4.pl b/lib/crypto/arm/sha512-armv4.= pl similarity index 100% rename from arch/arm/crypto/sha512-armv4.pl rename to lib/crypto/arm/sha512-armv4.pl diff --git a/lib/crypto/arm/sha512.h b/lib/crypto/arm/sha512.h new file mode 100644 index 0000000000000..f147b6490d6cd --- /dev/null +++ b/lib/crypto/arm/sha512.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * arm32-optimized SHA-512 block function + * + * Copyright 2025 Google LLC + */ + +#include +#include + +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon); + +asmlinkage void sha512_block_data_order(struct sha512_block_state *state, + const u8 *data, size_t nblocks); +asmlinkage void sha512_block_data_order_neon(struct sha512_block_state *st= ate, + const u8 *data, size_t nblocks); + +static void sha512_blocks(struct sha512_block_state *state, + const u8 *data, size_t nblocks) +{ + if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && + static_branch_likely(&have_neon) && likely(crypto_simd_usable())) { + kernel_neon_begin(); + sha512_block_data_order_neon(state, data, nblocks); + kernel_neon_end(); + } else { + sha512_block_data_order(state, data, nblocks); + } +} + +#ifdef CONFIG_KERNEL_MODE_NEON +#define sha512_mod_init_arch sha512_mod_init_arch +static inline void sha512_mod_init_arch(void) +{ + if (cpu_has_neon()) + static_branch_enable(&have_neon); +} +#endif /* CONFIG_KERNEL_MODE_NEON */ --=20 2.49.0