From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F21653537FE; Mon, 21 Sep 2026 05:16:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967765; cv=none; b=RzKehFglIcYv3zF0kXM0sD0PNfKODH2ri2OSDoEYSzMdXobMXh5Q0VrnCJyXgcYrU38bAh+p6jfvmLcBVX4beP4L9pFIMvOfDd/K6rMEw0i43KdC9n4VOcVlJ6QMhSgmfaLJYyRlMD8QplqMNwIHjGFXJ0cPTeosNmsk3cWoWA0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967765; c=relaxed/simple; bh=auzSsUIFvJG2EtKpLFfnXxHUxfbBKvMt0g7pXMc4uvk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nq7DaxoSemlYLyGeWqpnxUrcyPhgex1ojnXiUranvKBnMNWLZytEfh6pM6ZwIgzRz6nFo2xgj52IPsl/Y/1R6WM1Cza4CASZMLxzBb2nPYfvPWLFQiAfOFdDFWHM9r6Rjf1gk2AS7M+7iK2ld+m9G8Ly/6m0S7tBnw9D205MK8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KzCfrKiN; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KzCfrKiN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8024D1F0089A; Mon, 21 Sep 2026 05:16:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967763; bh=AmO1iCgpFBbB/kyTIAKSaYWoXnvuk8nhmmPYC4t0U3A=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=KzCfrKiN5xbJknA8Z0bxouu+q1rvnE+ICUiYxL1KzsYDKbiThtDeVNj6Pug2wVWh8 jvwoWf7N+iOtxzCecOjlRNAtu+GBhAlTo5I/AB//qkzjSt3+o5bRwJH9Fo6/ggqSET UWvp0IP55uxzUJugC7Gnv8xWCLhPa1UyxWMWa9fESruk15RUksMcoGCT+Y/AGDefHO gA5qf06iF45qh3WU3q3EfS0iRJtYiPwoWoAg6w1FCP1FfirkOLeOmCU4N/GjhKivxJ mfJoP1wInFh1wph6XccqJSKEgEJjAeUDLfxXvlS2VlMNGZMtq1JNLxDpFnpscTtJRi /GVM6fPV+CRiQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 01/20] crypto: aes - Fix undesired override of some optimized AES modes Date: Sun, 20 Sep 2026 22:08:47 -0700 Message-ID: <20260921050910.296144-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The new library APIs for AES encryption modes were wired up to the traditional crypto API via crypto/aes.c. However, for now the kernel is still in a transitional state where various architectures still have architecture-optimized implementations of AES modes in arch/*/crypto/, wired up to the traditional crypto API only. Because of that, the crypto/aes.c algorithms were given a cra_priority of only 110 to prevent them from overriding arch/*/crypto/ in the traditional crypto API. However, because of how the traditional crypto API works, the cra_priority trick doesn't work in cases where the relevant algorithm isn't directly implemented by arch/*/crypto/ but rather is provided by a template instance using other code in arch/*/crypto/. For example, x86 doesn't have its own "ccm(aes)" but rather relies on the "ccm" template constructing it from the x86-optimized "ctr(aes)". The existence of the library-based "ccm(aes)" prevents that, even though its priority is lower than what the template would produce. Thus, "ccm(aes)" ends up using the slower single-block AES code. Of course, this problem will go away as architecture-optimized implementations of AES modes are migrated into the library. But until then, we need to ensure that code continues to be used. Therefore, skip wiring up the relevant library-based code to the traditional crypto API on architectures where this problem can occur, as determined by what exists in arch/*/crypto/ for each architecture. Note: "xts(aes)" is left alone. Though the "xts" template can use "ecb(aes)" as an inner algorithm, in practice this isn't very efficient and a dedicated "xts(aes)" is already provided in all the important cases anyway. (This omission is also consistent with the fact that the library isn't planned to provide a similar ECB-to-XTS "adapter".) Fixes: 20df21a482aa ("crypto: aes - Add CBC and CBC-CTS support using libra= ry") Fixes: 8ca62072faa1 ("crypto: aes - Add GCM support using library") Fixes: f70ad727d1d6 ("crypto: aes - Add CCM support using library") Signed-off-by: Eric Biggers --- crypto/aes.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/crypto/aes.c b/crypto/aes.c index 94791f481e98..e6ba3899d868 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -637,7 +637,17 @@ static struct skcipher_alg skcipher_algs[] =3D { .decrypt =3D crypto_aes_cbc_decrypt, }, #endif -#if IS_ENABLED(CONFIG_CRYPTO_CTS) +#if IS_ENABLED(CONFIG_CRYPTO_CTS) && \ + /* + * Skip registering this when it might block a "better" implementation + * from being instantiated via the "cts" template wrapping an arch- + * optimized "cbc(aes)" that hasn't yet been migrated into the library. + */ \ + !(IS_ENABLED(CONFIG_ARM) || \ + IS_ENABLED(CONFIG_ARM64) || \ + IS_ENABLED(CONFIG_POWERPC) || \ + IS_ENABLED(CONFIG_S390) || \ + IS_ENABLED(CONFIG_SPARC)) { .base.cra_name =3D "cts(cbc(aes))", .base.cra_driver_name =3D "cts-cbc-aes-lib", @@ -980,7 +990,18 @@ static __maybe_unused int crypto_aes_ccm_decrypt(struc= t aead_request *req) } =20 static struct aead_alg aead_algs[] =3D { -#if IS_ENABLED(CONFIG_CRYPTO_GCM) +#if IS_ENABLED(CONFIG_CRYPTO_GCM) && \ + /* + * Skip registering these when they might block "better" implementations + * from being instantiated via the corresponding templates using + * arch-optimized code that hasn't yet been migrated into the library. + */ \ + !(IS_ENABLED(CONFIG_ARM) || \ + IS_ENABLED(CONFIG_ARM64) || \ + IS_ENABLED(CONFIG_POWERPC) || \ + IS_ENABLED(CONFIG_RISCV) || \ + IS_ENABLED(CONFIG_S390) || \ + IS_ENABLED(CONFIG_SPARC)) { .base.cra_name =3D "gcm(aes)", .base.cra_driver_name =3D "gcm-aes-lib", @@ -1012,7 +1033,19 @@ static struct aead_alg aead_algs[] =3D { .chunksize =3D AES_BLOCK_SIZE, }, #endif /* CONFIG_CRYPTO_GCM */ -#if IS_ENABLED(CONFIG_CRYPTO_CCM) +#if IS_ENABLED(CONFIG_CRYPTO_CCM) && \ + /* + * Skip registering this when it might block a "better" implementation + * from being instantiated via the "ccm" template wrapping an arch- + * optimized "ctr(aes)" that hasn't yet been migrated into the library. + */ \ + !(IS_ENABLED(CONFIG_ARM) || \ + IS_ENABLED(CONFIG_ARM64) || \ + IS_ENABLED(CONFIG_POWERPC) || \ + IS_ENABLED(CONFIG_RISCV) || \ + IS_ENABLED(CONFIG_S390) || \ + IS_ENABLED(CONFIG_SPARC) || \ + IS_ENABLED(CONFIG_X86)) { .base.cra_name =3D "ccm(aes)", .base.cra_driver_name =3D "ccm-aes-lib", --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BAA4352C5B; Mon, 21 Sep 2026 05:16:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967765; cv=none; b=roNQz8ea1jYhNmglSBlbQitTpF/q9K3LbURlZAuOwwpBnPtDKzGMKFdxAEWsCyqpfI89uuHsot2JmtL28WOIt2OLG6BzDBTQKrjeh/e2wnIHMNwv2cpsk+H1ei5zN4rFqGIbb3qbGaGt/PpZlzhKKCCz/ZaTtjnqolGozezfTtE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967765; c=relaxed/simple; bh=cJ/f3I9nHV0BnFI5TMqcZcGC8JGPk0cuKcXko1DNkrM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DV7uBCoIcmtwqxr8D+9fxEhMn78JSbK6WgkL/5ziREs/DbTj0Qg2t3QeZ7yqE5KJ+8oBF3CCTg6e7J4pnhJDEAla7DBcWq3IEdUYoHX9kmF90Idx8yDWOa/1D9cer9nMlMstkIAVxquJLanbQz/e2HIbFeZV6rogMweNQ7gzpVU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jZeziMwR; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jZeziMwR" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D97D01F00899; Mon, 21 Sep 2026 05:16:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967764; bh=DmrQnWAp+PfddUGRztDV+bpGLhcW7RlUL0r9TlxAYCg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=jZeziMwRSQcDUOCfNw+OPJIdK57/XDgsSnwJ4Fd/PYUQjXmeqejrpt7IoS3FiPndS BOq+HV3nZak15huEPJZYE+fGb3RYFZ+MCFdQ9rioFuawemy/lsBhFeDwWhdIIWDx4h ec67PmRHQ80qrWtwD62fAm+Fn0Eg+bc+BpHGOLjBbrvT/EGJLRK+rbj19+eTRvnnBm R1Mxd7KJUz4AB9S1rBNg2Z9Jteb0T0GiEcyHWRw4MfsDBkooMZnJK18XqD6aDunqZx dVL90HiO6iJauPKn+wGfxtaSmn+xA9cE6C05VSivQbRFWKOl98iV6aqEwQLh4Dr3S0 51yo1VPjN6BHQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 02/20] lib/crypto: aes-xctr: Pass counter by value to aes_xctr_arch() Date: Sun, 20 Sep 2026 22:08:48 -0700 Message-ID: <20260921050910.296144-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Update the calling convention for aes_xctr_arch() to pass the counter by value, then do the increment in generic code. This aligns better with the x86_64 and arm64 assembly code for XCTR, which takes the counter by value and thus has to be paired with an increment in C code anyway. Signed-off-by: Eric Biggers Reviewed-by: Thomas Huth --- lib/crypto/aes.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c index f1549839b3de..22f096c50242 100644 --- a/lib/crypto/aes.c +++ b/lib/crypto/aes.c @@ -1094,7 +1094,7 @@ static bool aes_ctr_arch(u8 *dst, const u8 *src, size= _t len, } #endif #ifndef aes_xctr_arch -static bool aes_xctr_arch(u8 *dst, const u8 *src, size_t len, u64 *ctr, +static bool aes_xctr_arch(u8 *dst, const u8 *src, size_t len, u64 ctr, const u8 iv[AES_BLOCK_SIZE], const struct aes_enckey *key) { @@ -1150,8 +1150,10 @@ void aes_xctr(u8 *dst, const u8 *src, size_t len, u6= 4 *ctr, __le64 aes_input[2]; u8 keystream[AES_BLOCK_SIZE] __aligned(__alignof__(long)); =20 - if (likely(aes_xctr_arch(dst, src, len, ctr, iv, key.enc_key))) + if (likely(aes_xctr_arch(dst, src, len, *ctr, iv, key.enc_key))) { + *ctr +=3D DIV_ROUND_UP(len, AES_BLOCK_SIZE); return; + } =20 aes_input[1] =3D get_unaligned((const __le64 *)&iv[8]); /* Handle the full blocks. */ --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8B49350A05; Mon, 21 Sep 2026 05:16:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967767; cv=none; b=Y72NyEIKbUnngUtcLckIxDXGYSGHC7d6/T6Kdfl2fo6YqFYPMwsNpST2k048pCvC4JYq8wDxZcdRkMAw8kuj/mFvQ/8S/kEe3VOszrkvVsahh9SCQxYMJw5L8Vbp2YpZqENvEbkvCDz8BYZkCNLcT7lPTMkDDhrwmOBWrUCZKUs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967767; c=relaxed/simple; bh=dwR3M5J35wTBrGyhCwiRv0p021dVC+yo5FnYeR/w+E8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UdEwPbYTT9CEX8oJVSuijzfQIA2vCx8m7FpIO/3KpsUq7SlPa8l5f1ldfBO+LQ+oeTXx7NIHpHYOQEo/tIJGh+msAzH7CEc4RlBz1zdOCzFbwMnSbkHMuGwescFkfJlGWgrMioLTv/aIFC21qLQnNu2oiLGF5MvsUJsgxheatmg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VgTXIipP; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VgTXIipP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3F3831F00898; Mon, 21 Sep 2026 05:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967764; bh=l2e5aS3P16JVyDKjq90b4GgBp+uz5p5ORC8v/uhLgc0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=VgTXIipPBuyIzUxt4YB8QdAuFiMZVdDBHQqS5I37l6gCAoBuz1sYrIRv9mp5a7CFs 7BSknCugDixGBc6ncSypV6r55fcR08MRcNq0/JG7Tf9o87OFQSOnfgCSl8XwL4feYq flSvX5ZIwGYsZOR9TttT+Ypqy2Mjop5+r5DDgLn/2d2rb5mYwBuLWyb9axBu1MpxG1 y7hhhepchWWxXY9R6KTd+/Lmt+x6pD+IqlYsm0+NH4HHBSYcxJCw2vj0p6PqAt+zWi lVRFy2c97BPGP672gD4dLZnkvAnfSbhdXgBdyGZJvjmRaPKM4TvP7217WSDga32xX5 YO/Sv2wAj1xvA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 03/20] lib/crypto: x86/aes: Clean up aes-aesni.S in preparation for AES modes Date: Sun, 20 Sep 2026 22:08:49 -0700 Message-ID: <20260921050910.296144-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Various miscellaneous updates to the assembly code in preparation for adding implementations of AES modes to the same file: - Define macros for the function argument registers, function prologues, and function epilogues to centralize some of the handling of 32-bit vs 64-bit. - Refactor the actual AES encryption and AES decryption logic into macros _do_aes and _do_aes_ecb so that some of the modes can reuse it. - Rename mask to expandkey_mask to differentiate it from the bswap_mask that will be added. - Update the prototypes of aes_encrypt_aesni() and aes_decrypt_aesni() to be dst, src, key so that they will match the mode functions. Note that this means passing a pointer to the key struct instead of a (nrounds, rndkeys) pair, similar to what arch/x86/crypto/aes*.S do. Although this makes the assembly code depend on the format of the key struct, having one fewer argument makes it easier to accommodate 32-bit mode, and the C glue code becomes slightly simpler. Signed-off-by: Eric Biggers --- lib/crypto/x86/aes-aesni.S | 249 +++++++++++++++++++++++++------------ lib/crypto/x86/aes.h | 28 +++-- 2 files changed, 185 insertions(+), 92 deletions(-) diff --git a/lib/crypto/x86/aes-aesni.S b/lib/crypto/x86/aes-aesni.S index b8c3e104a3be..90a3765d35b8 100644 --- a/lib/crypto/x86/aes-aesni.S +++ b/lib/crypto/x86/aes-aesni.S @@ -8,25 +8,126 @@ // AVX. It does use up to SSE4.1, which all CPUs with AES-NI have. #include =20 -.section .rodata #ifdef __x86_64__ #define RODATA(label) label(%rip) -#else + +#define ARG0 %rdi +#define ARG1 %rsi +#define ARG2 %rdx +#define ARG2_32 %edx +#define ARG3 %rcx +#define ARG4 %r8 +#define TMP %rax +#define TMP_32 %eax +#define TMP_16 %ax +#define TMP_8 %al + +#else // __x86_64__ + #define RODATA(label) label -#endif =20 +// Caller-save GPRs and the first 3 function arguments, assuming -mregparm= =3D3 +#define ARG0 %eax +#define ARG1 %edx +#define ARG2 %ecx +#define ARG2_32 %ecx + +// *Callee*-save GPRs. +#define ARG3 %edi +#define ARG3_32 %edi +#define ARG4 %esi +#define TMP %ebx +#define TMP_32 %ebx +#define TMP_16 %bx +#define TMP_8 %bl +#endif // !__x86_64__ + +// Offsets in struct aes_key +#define OFFSETOF_NROUNDS 4 +#define OFFSETOF_ROUNDKEYS 16 +#define OFFSETOF_INVROUNDKEYS 256 + +.section .rodata +.p2align 4 +.Lexpandkey_mask: // A mask for pshufb that extracts the last dword, rotates it right by 8 // bits, and copies the result to all four dwords. -.p2align 4 -.Lmask: .byte 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 12, 13, 14, 15, 12 =20 - // The AES round constants, used during key expansion .Lrcon: + // The AES round constants, used during key expansion .long 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36 =20 .text =20 +// In 32-bit mode, push certain callee-saved GPRs and optionally load func= tion +// arguments from the stack into them. Do nothing in 64-bit mode. +// +// TMP is always made available as a temporary register. +// +// \uses_arg3 is 0 to not touch ARG3, 1 to make it available as a temporary +// register only, or 2 to actually load it as an argument from the stack. +// Likewise for \uses_arg4 and ARG4. +.macro _prologue uses_arg3=3D0, uses_arg4=3D0 +#ifdef __i386__ + .set ARG3_OFFSET, 4 + .set ARG3_PUSHED, \uses_arg3 + .set ARG4_PUSHED, \uses_arg4 +.if \uses_arg3 + push ARG3 + .set ARG3_OFFSET, ARG3_OFFSET + 4 + .if \uses_arg3 =3D=3D 2 + mov ARG3_OFFSET(%esp), ARG3 + .endif +.endif +.if \uses_arg4 + push ARG4 + .set ARG3_OFFSET, ARG3_OFFSET + 4 + .if \uses_arg4 =3D=3D 2 + mov ARG3_OFFSET+4(%esp), ARG4 + .endif +.endif + push TMP + .set ARG3_OFFSET, ARG3_OFFSET + 4 +#endif // __i386__ +.endm + +.macro _reload_arg3 +#ifdef __i386__ + mov ARG3_OFFSET(%esp), ARG3 +#endif +.endm + +// Undo any pushes that _prologue did, then return. +.macro _epilogue +#ifdef __i386__ + pop TMP +.if ARG4_PUSHED + pop ARG4 +.endif +.if ARG3_PUSHED + pop ARG3 +.endif +#endif + RET +.endm + +.macro _aesenc enc, rndkey, data +.if \enc + aesenc \rndkey, \data +.else + aesdec \rndkey, \data +.endif +.endm + +.macro _aesenclast enc, rndkeylast, data +.if \enc + aesenclast \rndkeylast, \data +.else + aesdeclast \rndkeylast, \data +.endif +.endm + // Transform four dwords [a0, a1, a2, a3] in \a into // [a0, a0^a1, a0^a1^a2, a0^a1^a2^a3]. \tmp is a temporary xmm register. // @@ -71,32 +172,18 @@ .endm =20 .macro _aes_expandkey_aesni is_aes128 -#ifdef __x86_64__ // Arguments - .set RNDKEYS, %rdi - .set INV_RNDKEYS, %rsi - .set IN_KEY, %rdx + .set RNDKEYS, ARG0 + .set INV_RNDKEYS, ARG1 + .set IN_KEY, ARG2 =20 // Other local variables - .set RCON_PTR, %rcx - .set COUNTER, %eax -#else - // Arguments, assuming -mregparm=3D3 - .set RNDKEYS, %eax - .set INV_RNDKEYS, %edx - .set IN_KEY, %ecx - - // Other local variables - .set RCON_PTR, %ebx - .set COUNTER, %esi -#endif + .set RCON_PTR, ARG3 + .set COUNTER, TMP_32 .set RCON, %xmm6 .set MASK, %xmm7 =20 -#ifdef __i386__ - push %ebx - push %esi -#endif + _prologue uses_arg3=3D1 =20 .if \is_aes128 // AES-128: the first round key is simply a copy of the raw key. @@ -112,7 +199,7 @@ .endif =20 // Generate the remaining round keys. - movdqa RODATA(.Lmask), MASK + movdqa RODATA(.Lexpandkey_mask), MASK .if \is_aes128 lea RODATA(.Lrcon), RCON_PTR mov $10, COUNTER @@ -176,11 +263,7 @@ movdqu %xmm0, 16(INV_RNDKEYS) // =3D> Last inverse round key =20 .Ldone\@: -#ifdef __i386__ - pop %esi - pop %ebx -#endif - RET + _epilogue .endm =20 // void aes128_expandkey_aesni(u32 rndkeys[], u32 *inv_rndkeys, @@ -195,67 +278,73 @@ SYM_FUNC_START(aes256_expandkey_aesni) _aes_expandkey_aesni 0 SYM_FUNC_END(aes256_expandkey_aesni) =20 -.macro _aes_crypt_aesni enc -#ifdef __x86_64__ - .set RNDKEYS, %rdi - .set NROUNDS, %esi - .set OUT, %rdx - .set IN, %rcx -#else - // Assuming -mregparm=3D3 - .set RNDKEYS, %eax - .set NROUNDS, %edx - .set OUT, %ecx - .set IN, %ebx // Passed on stack -#endif - -#ifdef __i386__ - push %ebx - mov 8(%esp), %ebx -#endif - - // Zero-th round - movdqu (IN), %xmm0 - movdqu (RNDKEYS), %xmm1 - pxor %xmm1, %xmm0 - - // Normal rounds - add $16, RNDKEYS +// AES-encrypt (\enc=3D1) or decrypt (\enc=3D0) the AESDATA registers spec= ified in +// \vecs using the aes_enckey or aes_key pointed to by KEY. RNDKEY must b= e set +// to a temporary XMM register, NROUNDS to a temporary 32-bit GPR, and +// RNDKEY_PTR to a temporary full-size GPR. +.macro _do_aes enc, vecs:vararg + movl OFFSETOF_NROUNDS(KEY), NROUNDS dec NROUNDS -.Lnext_round\@: - movdqu (RNDKEYS), %xmm1 .if \enc - aesenc %xmm1, %xmm0 + .set rndkey0_offs, OFFSETOF_ROUNDKEYS .else - aesdec %xmm1, %xmm0 + .set rndkey0_offs, OFFSETOF_INVROUNDKEYS .endif - add $16, RNDKEYS + + // Do the zero-th AES round. + movdqu rndkey0_offs(KEY), RNDKEY +.irp i, \vecs + pxor RNDKEY, AESDATA\i +.endr + // Do the regular AES rounds. + lea rndkey0_offs+16(KEY), RNDKEY_PTR +.Lnext_round\@: + movdqu (RNDKEY_PTR), RNDKEY + add $16, RNDKEY_PTR +.irp i, \vecs + _aesenc \enc, RNDKEY, AESDATA\i +.endr dec NROUNDS - jne .Lnext_round\@ + jnz .Lnext_round\@ + // Do the last AES round. + movdqu (RNDKEY_PTR), RNDKEY +.irp i, \vecs + _aesenclast \enc, RNDKEY, AESDATA\i +.endr +.endm =20 - // Last round - movdqu (RNDKEYS), %xmm1 -.if \enc - aesenclast %xmm1, %xmm0 -.else - aesdeclast %xmm1, %xmm0 -.endif - movdqu %xmm0, (OUT) +.macro _do_aes_ecb enc, vecs:vararg +.irp i, \vecs + movdqu \i*16(SRC), AESDATA\i +.endr + _do_aes \enc, \vecs +.irp i, \vecs + movdqu AESDATA\i, \i*16(DST) +.endr +.endm =20 -#ifdef __i386__ - pop %ebx -#endif - RET +.macro _aes_crypt_aesni enc + .set DST, ARG0 + .set SRC, ARG1 + .set KEY, ARG2 + .set RNDKEY_PTR, ARG3 // Temporary register for _do_aes + .set NROUNDS, TMP_32 // Temporary register for _do_aes + .set AESDATA0, %xmm0 + .set RNDKEY, %xmm1 // Temporary register for _do_aes + + _prologue uses_arg3=3D1 + _do_aes_ecb \enc, 0 + _epilogue .endm =20 -// void aes_encrypt_aesni(const u32 rndkeys[], int nrounds, -// u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); +// void aes_encrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_S= IZE], +// const struct aes_enckey *key); SYM_FUNC_START(aes_encrypt_aesni) _aes_crypt_aesni 1 SYM_FUNC_END(aes_encrypt_aesni) =20 -// void aes_decrypt_aesni(const u32 inv_rndkeys[], int nrounds, -// u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); +// void aes_decrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_S= IZE], +// const struct aes_key *key); SYM_FUNC_START(aes_decrypt_aesni) _aes_crypt_aesni 0 SYM_FUNC_END(aes_decrypt_aesni) diff --git a/lib/crypto/x86/aes.h b/lib/crypto/x86/aes.h index b047dee94f57..06146fef06be 100644 --- a/lib/crypto/x86/aes.h +++ b/lib/crypto/x86/aes.h @@ -7,16 +7,21 @@ =20 #include =20 -static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aes); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aesni); + +/* The assembly code assumes the following offsets. */ +static_assert(offsetof(struct aes_enckey, nrounds) =3D=3D 4); +static_assert(offsetof(struct aes_enckey, k.rndkeys) =3D=3D 16); +static_assert(offsetof(struct aes_key, inv_k.inv_rndkeys) =3D=3D 256); =20 void aes128_expandkey_aesni(u32 rndkeys[], u32 *inv_rndkeys, const u8 in_key[AES_KEYSIZE_128]); void aes256_expandkey_aesni(u32 rndkeys[], u32 *inv_rndkeys, const u8 in_key[AES_KEYSIZE_256]); -void aes_encrypt_aesni(const u32 rndkeys[], int nrounds, - u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); -void aes_decrypt_aesni(const u32 inv_rndkeys[], int nrounds, - u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); +void aes_encrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_SIZE= ], + const struct aes_enckey *key); +void aes_decrypt_aesni(u8 dst[AES_BLOCK_SIZE], const u8 src[AES_BLOCK_SIZE= ], + const struct aes_key *key); =20 /* * Expand an AES key using AES-NI if supported and usable or generic code @@ -36,7 +41,7 @@ static void aes_preparekey_arch(union aes_enckey_arch *k, u32 *rndkeys =3D k->rndkeys; u32 *inv_rndkeys =3D inv_k ? inv_k->inv_rndkeys : NULL; =20 - if (static_branch_likely(&have_aes) && key_len !=3D AES_KEYSIZE_192 && + if (static_branch_likely(&have_aesni) && key_len !=3D AES_KEYSIZE_192 && irq_fpu_usable()) { kernel_fpu_begin(); if (key_len =3D=3D AES_KEYSIZE_128) @@ -53,9 +58,9 @@ static void aes_encrypt_arch(const struct aes_enckey *key, u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]) { - if (static_branch_likely(&have_aes) && irq_fpu_usable()) { + if (static_branch_likely(&have_aesni) && irq_fpu_usable()) { kernel_fpu_begin(); - aes_encrypt_aesni(key->k.rndkeys, key->nrounds, out, in); + aes_encrypt_aesni(out, in, key); kernel_fpu_end(); } else { aes_encrypt_generic(key->k.rndkeys, key->nrounds, out, in); @@ -66,10 +71,9 @@ static void aes_decrypt_arch(const struct aes_key *key, u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]) { - if (static_branch_likely(&have_aes) && irq_fpu_usable()) { + if (static_branch_likely(&have_aesni) && irq_fpu_usable()) { kernel_fpu_begin(); - aes_decrypt_aesni(key->inv_k.inv_rndkeys, key->nrounds, - out, in); + aes_decrypt_aesni(out, in, key); kernel_fpu_end(); } else { aes_decrypt_generic(key->inv_k.inv_rndkeys, key->nrounds, @@ -81,5 +85,5 @@ static void aes_decrypt_arch(const struct aes_key *key, static void aes_mod_init_arch(void) { if (boot_cpu_has(X86_FEATURE_AES)) - static_branch_enable(&have_aes); + static_branch_enable(&have_aesni); } --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BA8E353A73; Mon, 21 Sep 2026 05:16:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967767; cv=none; b=PbzR/gDfKObJ6F38/NSSPfGj5Br8O28Hvz8xX6oy8LJrUYQXlflnOgh9FOsY+XkZ+HgIfMjb2mHMqjVD7bdBhR2+hpn/Cib93cz0BCqoQ5/wd8yH2AnxqrI5BrTfFlOD7/vWMzUqyyxypf3mfLIP6OBupoBa3ZGeUyIhkPVrdAU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967767; c=relaxed/simple; bh=QSGclspbdogYLNXd2D/CGLmcjTmFgs8Ctczq1o/lxCc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kN4v3KswfrNA+AhfDiqjyjDtroG5MN5Tu2vUx/HFFFrcUhYIFvYm+k7epuKmsTfUhS5jHovhMwVyV0sl9RhVkV5eOQwlPA1oUQRldNSJEOH2QfNkkAoP4WN5R+GtKafDIKcbsLI9VraImrCoXt9U2klcdvC+wXTa2LEngCTMOPk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=P7l4EyBC; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="P7l4EyBC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 986741F0089B; Mon, 21 Sep 2026 05:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967764; bh=fcL/+D9vU7MNcdq0qTxGI6yjDteWwVjrF1tAzNkzGWc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=P7l4EyBCuYa26EerE8ZpkxwLraZtPIt3oIip9jfGw/C6iMPifu36hmD1ZWVtDV+7l XvbfzLBZpRw19V/zitmPV9xvaEQUmPSVwWi6IUF/7w5bRQN2KYEtHCycd45bkmFZpA DYcYvCbc08FPBgVF3Y4/CJhklroVKAcciqIAgDRg37bI5ZNRi7iJgqVUTqhWip+MFO OdhAxZGoSKot0M8YoPiu0uVvanWqFabS8O17Ueu/LI8aaM+uDmQmVgvDKbnUw0ZaF3 3KScqEdD7RiHV94XAVdcxY9oNfZSJzdVPfDDHiSWVjNbSHiuQo2vTjOm0Q0CJwP477 +t0SwP2m6w5+g== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 04/20] lib/crypto: x86/aes-ecb: Add AES-NI optimization Date: Sun, 20 Sep 2026 22:08:50 -0700 Message-ID: <20260921050910.296144-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Optimize the crypto library's AES-ECB support with AES-NI, making its performance be at least at parity with the "ecb-aes-aesni" skcipher algorithm that it will supersede. The new assembly functions are written from scratch to fit well into the crypto library. However, they are functionally very similar to the functions in arch/x86/crypto/aesni-intel_asm.S that they will supersede and are intended to provide parity with those -- including supporting 32-bit mode, having the inner loops do 4 AES blocks per iteration, etc. Signed-off-by: Eric Biggers --- crypto/aes.c | 2 +- lib/crypto/x86/aes-aesni.S | 54 ++++++++++++++++++++++++++++++++++++++ lib/crypto/x86/aes.h | 33 +++++++++++++++++++++++ 3 files changed, 88 insertions(+), 1 deletion(-) diff --git a/crypto/aes.c b/crypto/aes.c index e6ba3899d868..5a97dc812e8b 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -610,7 +610,7 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "ecb(aes)", .base.cra_driver_name =3D "ecb-aes-lib", - .base.cra_priority =3D 110, + .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_key), .base.cra_module =3D THIS_MODULE, diff --git a/lib/crypto/x86/aes-aesni.S b/lib/crypto/x86/aes-aesni.S index 90a3765d35b8..fdb2917deb59 100644 --- a/lib/crypto/x86/aes-aesni.S +++ b/lib/crypto/x86/aes-aesni.S @@ -348,3 +348,57 @@ SYM_FUNC_END(aes_encrypt_aesni) SYM_FUNC_START(aes_decrypt_aesni) _aes_crypt_aesni 0 SYM_FUNC_END(aes_decrypt_aesni) + +.macro _ecb_crypt enc + // Arguments + .set DST, ARG0 + .set SRC, ARG1 + .set NBLOCKS, ARG2 + .set NBLOCKS32, ARG2_32 // Used for improved code density + .set KEY, ARG3 + + // Other local variables + .set RNDKEY_PTR, ARG4 // Temporary register for _do_aes + .set NROUNDS, TMP_32 // Temporary register for _do_aes + .set AESDATA0, %xmm0 + .set AESDATA1, %xmm1 + .set AESDATA2, %xmm2 + .set AESDATA3, %xmm3 + .set RNDKEY, %xmm4 // Temporary register for _do_aes + _prologue uses_arg3=3D2, uses_arg4=3D1 + + sub $4, NBLOCKS + jl .Lecb_loop4_done\@ +.p2align 5 +.Lecb_loop4\@: + _do_aes_ecb \enc, 0,1,2,3 + add $64, DST + add $64, SRC + sub $4, NBLOCKS + jge .Lecb_loop4\@ +.Lecb_loop4_done\@: + add $4, NBLOCKS32 + jz .Lecb_done\@ + +.Lecb_loop1\@: + _do_aes_ecb \enc, 0 + add $16, DST + add $16, SRC + dec NBLOCKS32 + jnz .Lecb_loop1\@ + +.Lecb_done\@: + _epilogue +.endm + +// void aes_ecb_encrypt_aesni(u8 *dst, const u8 *src, long nblocks, +// const struct aes_enckey *key); +SYM_FUNC_START(aes_ecb_encrypt_aesni) + _ecb_crypt 1 +SYM_FUNC_END(aes_ecb_encrypt_aesni) + +// void aes_ecb_decrypt_aesni(u8 *dst, const u8 *src, long nblocks, +// const struct aes_key *key); +SYM_FUNC_START(aes_ecb_decrypt_aesni) + _ecb_crypt 0 +SYM_FUNC_END(aes_ecb_decrypt_aesni) diff --git a/lib/crypto/x86/aes.h b/lib/crypto/x86/aes.h index 06146fef06be..9ad4a84f0378 100644 --- a/lib/crypto/x86/aes.h +++ b/lib/crypto/x86/aes.h @@ -81,6 +81,39 @@ static void aes_decrypt_arch(const struct aes_key *key, } } =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_ECB) +void aes_ecb_encrypt_aesni(u8 *dst, const u8 *src, long nblocks, + const struct aes_enckey *key); +void aes_ecb_decrypt_aesni(u8 *dst, const u8 *src, long nblocks, + const struct aes_key *key); + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_ecb_encrypt_arch aes_ecb_encrypt_arch +static bool aes_ecb_encrypt_arch(u8 *dst, const u8 *src, size_t len, + const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + kernel_fpu_begin(); + aes_ecb_encrypt_aesni(dst, src, len / AES_BLOCK_SIZE, key); + kernel_fpu_end(); + return true; +} + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_ecb_decrypt_arch aes_ecb_decrypt_arch +static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src, size_t len, + const struct aes_key *key) +{ + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + kernel_fpu_begin(); + aes_ecb_decrypt_aesni(dst, src, len / AES_BLOCK_SIZE, key); + kernel_fpu_end(); + return true; +} +#endif /* CONFIG_CRYPTO_LIB_AES_ECB */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D46A3546D7; Mon, 21 Sep 2026 05:16:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967767; cv=none; b=JfdlzPv/21ZjQ6z/Xprrk2dtuX40zkJqaJA0CB0Y8zm0llax9gqO3SbiqTPKqmHjBtjdta2JvKWzT+04+q8e33RatypQHSPD3JQ7deVdQCa5Zj4KohqXYk1h1UI2N9/vBxWAQJRwHmjsmqudg/vy8jLxTvlWJFjh5dsoIKFMgro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967767; c=relaxed/simple; bh=zlkL4O2KlClb50FnJ5YALUCJCu2y++H0Wp/pk7st6Oo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bEGFa/7wl78pgAYqtAlcHMgz7p4oOtefTPzAw8afwO97t0JSmHw7Xo/vvBZD39MVSHe4/osxnTAlfALI0gPOqeCPQSfzNo94GXIsSehZwoTZ3n1yWAbwawsL53wcfcNFguxrue1KyPqiTFORgDokvYLVe7yoN6HrbsI5xviZlRg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hRQANaMg; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hRQANaMg" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1A6E1F00893; Mon, 21 Sep 2026 05:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967765; bh=lwSKS6W6HhL1Ql1jfyigEqHpJBGpQYQ2rJqm5NzIuGE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=hRQANaMgs17/6VSF0JU4mhvDouBHwJpPuIat0NoyyNtJhno0dv5U8+QUVYIeurOIo uZpVEU8Z+JkWDZAA9LMWzUchYta29mlppSkgjjIMk9ibnDqbCAvOp4XDQZEY9hv0zx zvOlojmxBE7PB4fUc0RrCcOaUjdX2cfpY1UB5tS81yCv26ylNp5BvlxMueHdLG7kjQ CF933MMbv7U4nBIhbLyUumJL8V1AgORpBwoY++X9gofyZdsuzlru2cqD9If805e5c7 31fBSeABXpPJOX2HhtcvaUqS3jNmmfrlw7jnfa2IZGK6y4XAXQFutwXG+EEkiHSveU MsRM6mshCk1Gw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 05/20] lib/crypto: x86/aes-cbc: Add AES-NI optimization Date: Sun, 20 Sep 2026 22:08:51 -0700 Message-ID: <20260921050910.296144-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Optimize the crypto library's AES-CBC and AES-CBC-CTS support with AES-NI, making their performance be at least at parity with the "cbc-aes-aesni" and "cts-cbc-aes-aesni" skcipher algorithms that they will supersede. The new assembly functions are written from scratch to fit well into the crypto library. However, they are functionally very similar to the functions in arch/x86/crypto/aesni-intel_asm.S that they will supersede and are intended to provide parity with those -- including supporting 32-bit mode, having the inner loops do 4 AES blocks per iteration, etc. Signed-off-by: Eric Biggers --- crypto/aes.c | 4 +- lib/crypto/x86/aes-aesni.S | 246 +++++++++++++++++++++++++++++++++++++ lib/crypto/x86/aes.h | 89 ++++++++++++++ 3 files changed, 337 insertions(+), 2 deletions(-) diff --git a/crypto/aes.c b/crypto/aes.c index 5a97dc812e8b..cc2cd6b08eee 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -625,7 +625,7 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "cbc(aes)", .base.cra_driver_name =3D "cbc-aes-lib", - .base.cra_priority =3D 110, + .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_key), .base.cra_module =3D THIS_MODULE, @@ -651,7 +651,7 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "cts(cbc(aes))", .base.cra_driver_name =3D "cts-cbc-aes-lib", - .base.cra_priority =3D 110, + .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_key), .base.cra_module =3D THIS_MODULE, diff --git a/lib/crypto/x86/aes-aesni.S b/lib/crypto/x86/aes-aesni.S index fdb2917deb59..17da4d710574 100644 --- a/lib/crypto/x86/aes-aesni.S +++ b/lib/crypto/x86/aes-aesni.S @@ -58,6 +58,14 @@ // The AES round constants, used during key expansion .long 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36 =20 +.Lcts_permute_table: + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 + .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 + .text =20 // In 32-bit mode, push certain callee-saved GPRs and optionally load func= tion @@ -402,3 +410,241 @@ SYM_FUNC_END(aes_ecb_encrypt_aesni) SYM_FUNC_START(aes_ecb_decrypt_aesni) _ecb_crypt 0 SYM_FUNC_END(aes_ecb_decrypt_aesni) + +// void aes_cbc_encrypt_aesni(u8 *dst, const u8 *src, long nblocks, +// u8 iv[AES_BLOCK_SIZE], +// const struct aes_enckey *key); +SYM_FUNC_START(aes_cbc_encrypt_aesni) + // Arguments + .set DST, ARG0 + .set SRC, ARG1 + .set NBLOCKS, ARG2 + .set IV_PTR, ARG3 + .set KEY, ARG4 + + // Other local variables +#ifdef __x86_64__ + .set RNDKEY_PTR, %r9 +#else + .set RNDKEY_PTR, IV_PTR // IV_PTR is clobbered and reloaded later +#endif + .set NROUNDS, TMP_32 + .set AESDATA0, %xmm0 + .set PTEXT, %xmm1 + _prologue uses_arg3=3D2, uses_arg4=3D2 + + movdqu (IV_PTR), AESDATA0 +.p2align 5 +.Lcbc_enc_loop: + movdqu (SRC), PTEXT + pxor PTEXT, AESDATA0 + _do_aes 1, 0 + movdqu AESDATA0, (DST) + add $16, DST + add $16, SRC + dec NBLOCKS + jnz .Lcbc_enc_loop + + // Store the next IV. On 32-bit, reload IV_PTR from stack first. + _reload_arg3 + movdqu AESDATA0, (IV_PTR) + _epilogue +SYM_FUNC_END(aes_cbc_encrypt_aesni) + +// void aes_cbc_decrypt_aesni(u8 *dst, const u8 *src, long nblocks, +// u8 iv[AES_BLOCK_SIZE], +// const struct aes_key *key); +SYM_FUNC_START(aes_cbc_decrypt_aesni) + // Arguments + .set DST, ARG0 + .set SRC, ARG1 + .set NBLOCKS, ARG2 + .set NBLOCKS32, ARG2_32 // Used for improved code density + .set IV_PTR, ARG3 + .set KEY, ARG4 + + // Other local variables +#ifdef __x86_64__ + .set RNDKEY_PTR, %r9 +#else + .set RNDKEY_PTR, IV_PTR // IV_PTR is clobbered and reloaded later +#endif + .set NROUNDS, TMP_32 + .set AESDATA0, %xmm0 + .set AESDATA1, %xmm1 + .set AESDATA2, %xmm2 + .set AESDATA3, %xmm3 + .set RNDKEY, %xmm4 + .set IV, %xmm5 + .set CTEXT0, %xmm6 + .set CTEXT1, %xmm7 +#ifdef __x86_64__ + .set CTEXT2, %xmm8 + .set CTEXT3, %xmm9 +#endif + _prologue uses_arg3=3D2, uses_arg4=3D2 + + movdqu (IV_PTR), IV + + sub $4, NBLOCKS + jl .Lcbc_dec_loop4_done + +.p2align 5 +.Lcbc_dec_loop4: + movdqu 0(SRC), AESDATA0 + movdqu 16(SRC), AESDATA1 + movdqu 32(SRC), AESDATA2 + movdqu 48(SRC), AESDATA3 + movdqa AESDATA0, CTEXT0 + movdqa AESDATA1, CTEXT1 +#ifdef __x86_64__ + movdqa AESDATA2, CTEXT2 + movdqa AESDATA3, CTEXT3 +#endif + _do_aes 0, 0,1,2,3 + pxor IV, AESDATA0 + pxor CTEXT0, AESDATA1 + pxor CTEXT1, AESDATA2 +#ifdef __x86_64__ + pxor CTEXT2, AESDATA3 + movdqa CTEXT3, IV +#else + movdqu 32(SRC), CTEXT0 + pxor CTEXT0, AESDATA3 + movdqu 48(SRC), IV +#endif + movdqu AESDATA0, 0(DST) + movdqu AESDATA1, 16(DST) + movdqu AESDATA2, 32(DST) + movdqu AESDATA3, 48(DST) + add $64, DST + add $64, SRC + sub $4, NBLOCKS + jge .Lcbc_dec_loop4 +.Lcbc_dec_loop4_done: + add $4, NBLOCKS32 + jz .Lcbc_dec_done + +.Lcbc_dec_loop1: + movdqu (SRC), AESDATA0 + movdqa AESDATA0, CTEXT0 + _do_aes 0, 0 + pxor IV, AESDATA0 + movdqa CTEXT0, IV + movdqu AESDATA0, (DST) + add $16, DST + add $16, SRC + dec NBLOCKS32 + jnz .Lcbc_dec_loop1 + +.Lcbc_dec_done: + // Store the next IV. On 32-bit, reload IV_PTR from stack first. + _reload_arg3 + movdqu IV, (IV_PTR) + _epilogue +SYM_FUNC_END(aes_cbc_decrypt_aesni) + +// void aes_cbc_cts_encrypt_aesni(u8 *dst, const u8 *src, long pn_len, +// const u8 iv[AES_BLOCK_SIZE], +// const struct aes_enckey *key); +// +// Encrypt the last two blocks using the CS3 variant of ciphertext stealin= g. +// 1 <=3D pn_len <=3D 16 gives the length of the last plaintext block (i.e= . P_n) in +// bytes, so in total this processes 17 to 32 bytes inclusive. +SYM_FUNC_START(aes_cbc_cts_encrypt_aesni) + .set DST, ARG0 + .set SRC, ARG1 + .set PN_LEN, ARG2 + .set IV_PTR, ARG3 + .set KEY, ARG4 + .set RNDKEY_PTR, IV_PTR // Temporary register for _do_aes + .set NROUNDS, TMP_32 // Temporary register for _do_aes + .set AESDATA0, %xmm0 + .set AESDATA1, %xmm1 + .set RNDKEY, %xmm2 + .set LSHIFT_MASK, %xmm3 // [0x80, 0x80, ...] + range(PN_LEN) + .set RSHIFT_MASK, %xmm4 // range(16-PN_LEN,16) + [0x80, 0x80, ...] + .set IV, %xmm5 + + _prologue uses_arg3=3D2, uses_arg4=3D2 + + lea RODATA(.Lcts_permute_table), TMP + movdqu (TMP,PN_LEN), LSHIFT_MASK + sub PN_LEN, TMP + movdqu 32(TMP), RSHIFT_MASK + + // Load the last two plaintext blocks. Last one is left-aligned. + movdqu (SRC), AESDATA0 + movdqu (SRC,PN_LEN), AESDATA1 + + // Encrypt the second-from-last block. + movdqu (IV_PTR), IV + pxor IV, AESDATA0 + _do_aes 1, 0 + + // Right-align the last block, then encrypt it. + pshufb RSHIFT_MASK, AESDATA1 + pxor AESDATA0, AESDATA1 + _do_aes 1, 1 + + // Store the last two ciphertext blocks. + pshufb LSHIFT_MASK, AESDATA0 + movdqu AESDATA0, (DST,PN_LEN) + movdqu AESDATA1, (DST) + + _epilogue +SYM_FUNC_END(aes_cbc_cts_encrypt_aesni) + +// void aes_cbc_cts_decrypt_aesni(u8 *dst, const u8 *src, long pn_len, +// const u8 iv[AES_BLOCK_SIZE], +// const struct aes_key *key); +// +// Decrypt the last two blocks using the CS3 variant of ciphertext stealin= g. +// 1 <=3D pn_len <=3D 16 gives the length of the last plaintext block (i.e= . P_n) in +// bytes, so in total this processes 17 to 32 bytes inclusive. +SYM_FUNC_START(aes_cbc_cts_decrypt_aesni) + .set DST, ARG0 + .set SRC, ARG1 + .set PN_LEN, ARG2 + .set IV_PTR, ARG3 + .set KEY, ARG4 + .set RNDKEY_PTR, IV_PTR // Temporary register for _do_aes + .set NROUNDS, TMP_32 // Temporary register for _do_aes + .set RSHIFT_MASK, %xmm0 // range(16-PN_LEN,16) + [0x80, 0x80, ...] + .set LSHIFT_MASK, %xmm1 // [0x80, 0x80, ...] + range(PN_LEN) + .set AESDATA0, %xmm2 + .set AESDATA1, %xmm3 + .set RNDKEY, %xmm4 + .set IV, %xmm5 + + _prologue uses_arg3=3D2, uses_arg4=3D2 + + lea RODATA(.Lcts_permute_table), TMP + movdqu (TMP,PN_LEN), LSHIFT_MASK + sub PN_LEN, TMP + movdqu 32(TMP), RSHIFT_MASK + movdqu (IV_PTR), IV + + // Load the last two ciphertext blocks. Last one is left-aligned. + movdqu (SRC), AESDATA0 + movdqu (SRC,PN_LEN), AESDATA1 + + // Decrypt the second-from-last ciphertext block. + _do_aes 0, 0 + + // Recover and store the last plaintext block, left-aligned. + movdqa AESDATA0, %xmm6 + pshufb LSHIFT_MASK, %xmm6 + pxor AESDATA1, %xmm6 + movdqu %xmm6, (DST,PN_LEN) + + // Recover and store the second-from-last plaintext block. + // Note that pblendvb uses %xmm0 (RSHIFT_MASK) as an implicit operand. + pshufb RSHIFT_MASK, AESDATA1 + pblendvb AESDATA0, AESDATA1 + _do_aes 0, 1 + pxor IV, AESDATA1 + movdqu AESDATA1, (DST) + + _epilogue +SYM_FUNC_END(aes_cbc_cts_decrypt_aesni) diff --git a/lib/crypto/x86/aes.h b/lib/crypto/x86/aes.h index 9ad4a84f0378..67a4178b7acd 100644 --- a/lib/crypto/x86/aes.h +++ b/lib/crypto/x86/aes.h @@ -114,6 +114,95 @@ static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *sr= c, size_t len, } #endif /* CONFIG_CRYPTO_LIB_AES_ECB */ =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CBC) +void aes_cbc_encrypt_aesni(u8 *dst, const u8 *src, long nblocks, + u8 iv[AES_BLOCK_SIZE], const struct aes_enckey *key); +void aes_cbc_decrypt_aesni(u8 *dst, const u8 *src, long nblocks, + u8 iv[AES_BLOCK_SIZE], const struct aes_key *key); +void aes_cbc_cts_encrypt_aesni(u8 *dst, const u8 *src, long pn_len, + const u8 iv[AES_BLOCK_SIZE], + const struct aes_enckey *key); +void aes_cbc_cts_decrypt_aesni(u8 *dst, const u8 *src, long pn_len, + const u8 iv[AES_BLOCK_SIZE], + const struct aes_key *key); + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_cbc_encrypt_arch aes_cbc_encrypt_arch +static bool aes_cbc_encrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + kernel_fpu_begin(); + aes_cbc_encrypt_aesni(dst, src, len / AES_BLOCK_SIZE, iv, key); + kernel_fpu_end(); + return true; +} + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_cbc_decrypt_arch aes_cbc_decrypt_arch +static bool aes_cbc_decrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_key *key) +{ + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + kernel_fpu_begin(); + aes_cbc_decrypt_aesni(dst, src, len / AES_BLOCK_SIZE, iv, key); + kernel_fpu_end(); + return true; +} + +/* len can be any value greater than AES_BLOCK_SIZE here. */ +#define aes_cbc_cts_encrypt_arch aes_cbc_cts_encrypt_arch +static bool aes_cbc_cts_encrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_enckey *key) +{ + const size_t cbc_blocks =3D (len - AES_BLOCK_SIZE - 1) / AES_BLOCK_SIZE; + const size_t pn_len =3D ((len - 1) % AES_BLOCK_SIZE) + 1; + + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + + kernel_fpu_begin(); + if (cbc_blocks) { + aes_cbc_encrypt_aesni(dst, src, cbc_blocks, iv, key); + dst +=3D cbc_blocks * AES_BLOCK_SIZE; + src +=3D cbc_blocks * AES_BLOCK_SIZE; + } + /* This part handles the final 17 to 32 bytes. */ + aes_cbc_cts_encrypt_aesni(dst, src, pn_len, iv, key); + kernel_fpu_end(); + return true; +} + +/* len can be any value greater than AES_BLOCK_SIZE here. */ +#define aes_cbc_cts_decrypt_arch aes_cbc_cts_decrypt_arch +static bool aes_cbc_cts_decrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_key *key) +{ + const size_t cbc_blocks =3D (len - AES_BLOCK_SIZE - 1) / AES_BLOCK_SIZE; + const size_t pn_len =3D ((len - 1) % AES_BLOCK_SIZE) + 1; + + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + + kernel_fpu_begin(); + if (cbc_blocks) { + aes_cbc_decrypt_aesni(dst, src, cbc_blocks, iv, key); + dst +=3D cbc_blocks * AES_BLOCK_SIZE; + src +=3D cbc_blocks * AES_BLOCK_SIZE; + } + /* This part handles the final 17 to 32 bytes. */ + aes_cbc_cts_decrypt_aesni(dst, src, pn_len, iv, key); + kernel_fpu_end(); + return true; +} +#endif /* CONFIG_CRYPTO_LIB_AES_CBC */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDCD8356770; Mon, 21 Sep 2026 05:16:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967768; cv=none; b=GpHmmlJGsVIvMNDBtJVucDGiPLZY4abuyjAG0MjmwRqGpNSFZcPRcq2UIsIvFMQZQScDNtZ/VQR+Qsj08ff/s2jb1x8K5fwep1vL+l9jQm5sEWxUDeCmZrzte2hFT7q/hvSEiQZWxJQRYHSCuYxlMa3wwE1EHflEcTIvFaWvR20= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967768; c=relaxed/simple; bh=xZIvNRXxYbj2iqtmLmxjpLgXQra/KLHdMVFsaXJTJw4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fONuyNp41yT8zVS6REFGM5od2jEd3LmsLrVupve0akdukT4Qgkjw7TjTCrZkMlVJfpUPDUjeIaCjGPrKd3KP8tKpJ/g+hZLq1ik1OR1ckbLUp9xUhOj1l+BMR/YxTvfiEVim+LqmH/E5WzgQomKkcXoDzi+gEfUBbzR5ihnbQ1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=az5o6khp; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="az5o6khp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 57A251F0089A; Mon, 21 Sep 2026 05:16:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967765; bh=WqHwU9wc0nBnAXv14Y3oqjMJhQv94lehmswMUrPQrGY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=az5o6khpTOFV+kkbYyE7+68Rax9Em3N73CTxrNC4UYkI/O3jRfAcP/+4xFB3W4TZ8 LqxCVE73PeO9mk9c9ief4q9jxpolFf2vMD2xYzVjNLEjSilxzlxGQr4qCe5EE/udYw SSxqechPBt6jy+uXwSAGngxe+qhsY+1nSng+lEFM4B+MUL8R8xRhGG/kryK04ugQRF yM8/N/qpa2axkzQr5pybXXtylziNa1nuESqoqjYxZpkkeYJ5rKy8F3H5lgiVKdozxZ 5JkAHke1f+3C8Z0ok898xEy2WkCJ4KrPCQzoQVR8fG74JRZBRhrNQUUc8iAeJZp2fG wZuQ8+ATrsahQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 06/20] lib/crypto: x86/aes-ctr: Add AES-NI optimization Date: Sun, 20 Sep 2026 22:08:52 -0700 Message-ID: <20260921050910.296144-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Optimize the crypto library's AES-CTR support with AES-NI, making its performance be at least at parity with the "ctr-aes-aesni" skcipher algorithm that it will supersede. The new assembly function is written from scratch to fit well into the crypto library and to be more consistent with aes-ctr-avx-x86_64.S than the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede. That includes using the "ctr64" convention, where the assembly code is simplified by making the C code handle incrementing the high 64 bits of the counter. Unlike the ECB, CBC, and XTS code, 32-bit support is *not* included for this one, as the existing CTR code didn't have it. Note: the priority of ctr-aes-lib is left unchanged at 110 temporarily. It will be increased when the AVX-optimized code is migrated too. Signed-off-by: Eric Biggers --- lib/crypto/x86/aes-aesni.S | 130 ++++++++++++++++++++++++++++++++++++- lib/crypto/x86/aes.h | 54 +++++++++++++++ 2 files changed, 183 insertions(+), 1 deletion(-) diff --git a/lib/crypto/x86/aes-aesni.S b/lib/crypto/x86/aes-aesni.S index 17da4d710574..24c53f1a144b 100644 --- a/lib/crypto/x86/aes-aesni.S +++ b/lib/crypto/x86/aes-aesni.S @@ -4,7 +4,8 @@ // // Copyright 2026 Google LLC // -// The code in this file supports 32-bit and 64-bit CPUs, and it doesn't r= equire +// The code in this file supports 32-bit and 64-bit CPUs (except for +// aes_ctr64_crypt_aesni() which supports 64-bit only), and it doesn't req= uire // AVX. It does use up to SSE4.1, which all CPUs with AES-NI have. #include =20 @@ -49,6 +50,12 @@ =20 .section .rodata .p2align 4 +#ifdef __x86_64__ +.Lbswap_mask: + // A mask for pshufb that byte-reflects the value. + .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 +#endif + .Lexpandkey_mask: // A mask for pshufb that extracts the last dword, rotates it right by 8 // bits, and copies the result to all four dwords. @@ -648,3 +655,124 @@ SYM_FUNC_START(aes_cbc_cts_decrypt_aesni) =20 _epilogue SYM_FUNC_END(aes_cbc_cts_decrypt_aesni) + +#ifdef __x86_64__ +// void aes_ctr64_crypt_aesni(u8 *dst, const u8 *src, s64 len, +// const u64 le_ctr[2], +// const struct aes_enckey *key); +SYM_FUNC_START(aes_ctr64_crypt_aesni) + // Arguments + .set DST, ARG0 + .set SRC, ARG1 + .set LEN, ARG2 + .set LEN32, ARG2_32 // Used for improved code density + .set LE_CTR_PTR, ARG3 // Used as temp reg after LE_CTR is loaded + .set KEY, ARG4 + + // Other local variables + .set AESDATA0, %xmm0 + .set AESDATA1, %xmm1 + .set AESDATA2, %xmm2 + .set AESDATA3, %xmm3 + .set LE_CTR, %xmm4 // Current 128-bit little endian counter + .set LE_CTR_INC, %xmm5 // Initialized to (u64[])[1, 0] + .set BSWAP_MASK, %xmm6 + .set RNDKEY, %xmm7 + .set RNDKEY_PTR, LE_CTR_PTR // Temporary register for _do_aes + .set NROUNDS, TMP_32 // Temporary register for _do_aes + + // Initialize LE_CTR, BSWAP_MASK, and LE_CTR_INC. + movdqu (LE_CTR_PTR), LE_CTR + movdqa RODATA(.Lbswap_mask), BSWAP_MASK + mov $1, TMP_32 + movd TMP_32, LE_CTR_INC + + // Encrypt and XOR four blocks (64 bytes) at a time. + sub $64, LEN + jl .Lctr_loop4_done +.p2align 5 +.Lctr_loop4: +.irp i, 0,1,2,3 + movdqa LE_CTR, AESDATA\i + pshufb BSWAP_MASK, AESDATA\i // =3D> big endian counter + paddq LE_CTR_INC, LE_CTR +.endr + _do_aes 1, 0,1,2,3 + // AESDATA[0-3] now contain four keystream blocks. +.irp i, 0,1,2,3 + movdqu \i*16(SRC), RNDKEY // Use RNDKEY as temp register. + pxor RNDKEY, AESDATA\i + movdqu AESDATA\i, \i*16(DST) +.endr + add $64, DST + add $64, SRC + sub $64, LEN + jge .Lctr_loop4 +.Lctr_loop4_done: + add $64, LEN + jz .Lctr_done + + // 1 <=3D LEN <=3D 63 bytes remain. Prepare four more keystream blocks. +.irp i, 0,1,2,3 + movdqa LE_CTR, AESDATA\i + pshufb BSWAP_MASK, AESDATA\i // =3D> big endian counter + .if \i !=3D 3 + paddq LE_CTR_INC, LE_CTR + .endif +.endr + _do_aes 1, 0,1,2,3 + // AESDATA[0-3] now contain four keystream blocks. + + // XOR one block (16 bytes) at a time. + sub $16, LEN32 + jl .Lctr_partial +.Lctr_xor1: + movdqu (SRC), RNDKEY // Use RNDKEY as temp register. + pxor RNDKEY, AESDATA0 + movdqu AESDATA0, (DST) + movdqa AESDATA1, AESDATA0 + movdqa AESDATA2, AESDATA1 + movdqa AESDATA3, AESDATA2 + add $16, SRC + add $16, DST + sub $16, LEN32 + jge .Lctr_xor1 + + // XOR the remaining LEN mod 16 bytes. +.Lctr_partial: + test $8, LEN32 + jz 1f + movq AESDATA0, TMP + xor (SRC), TMP // XOR 8 bytes. + mov TMP, (DST) + add $8, SRC + add $8, DST + psrldq $8, AESDATA0 +1: + test $4, LEN32 + jz 2f + movd AESDATA0, TMP_32 + xor (SRC), TMP_32 // XOR 4 bytes. + mov TMP_32, (DST) + add $4, SRC + add $4, DST + psrldq $4, AESDATA0 +2: + test $2, LEN32 + jz 3f + movd AESDATA0, TMP_32 + xor (SRC), TMP_16 // XOR 2 bytes. + mov TMP_16, (DST) + add $2, SRC + add $2, DST + psrldq $2, AESDATA0 +3: + test $1, LEN32 + jz .Lctr_done + movd AESDATA0, TMP_32 + xor (SRC), TMP_8 // XOR 1 byte. + mov TMP_8, (DST) +.Lctr_done: + RET +SYM_FUNC_END(aes_ctr64_crypt_aesni) +#endif // __x86_64__ diff --git a/lib/crypto/x86/aes.h b/lib/crypto/x86/aes.h index 67a4178b7acd..685b43ce6ef0 100644 --- a/lib/crypto/x86/aes.h +++ b/lib/crypto/x86/aes.h @@ -203,6 +203,60 @@ static bool aes_cbc_cts_decrypt_arch(u8 *dst, const u8= *src, size_t len, } #endif /* CONFIG_CRYPTO_LIB_AES_CBC */ =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CTR) && IS_ENABLED(CONFIG_X86_64) +void aes_ctr64_crypt_aesni(u8 *dst, const u8 *src, s64 len, const u64 le_c= tr[2], + const struct aes_enckey *key); + +static void aes_ctr64_x86(u8 *dst, const u8 *src, size_t len, + const u64 le_ctr[2], const struct aes_enckey *key) +{ + aes_ctr64_crypt_aesni(dst, src, len, le_ctr, key); +} + +#define aes_ctr_arch aes_ctr_arch +static bool aes_ctr_arch(u8 *dst, const u8 *src, size_t len, + u8 ctr[AES_BLOCK_SIZE], const struct aes_enckey *key) +{ + u64 le_ctr[2]; + u64 ctr64; + size_t nblocks; + size_t part1_len; + + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + + ctr64 =3D le_ctr[0] =3D get_unaligned_be64(&ctr[8]); + le_ctr[1] =3D get_unaligned_be64(&ctr[0]); + + kernel_fpu_begin(); + + nblocks =3D DIV_ROUND_UP(len, AES_BLOCK_SIZE); + ctr64 +=3D nblocks; + + if (likely(ctr64 >=3D nblocks)) { + /* The low 64 bits of the counter won't overflow. */ + aes_ctr64_x86(dst, src, len, le_ctr, key); + } else { + /* + * The low 64 bits of the counter will overflow. The + * assembly doesn't handle this case, so split the + * operation into two at the point where the overflow + * will occur. After the first part, add the carry bit. + */ + part1_len =3D min(len, (nblocks - ctr64) * AES_BLOCK_SIZE); + aes_ctr64_x86(dst, src, part1_len, le_ctr, key); + le_ctr[0] =3D 0; + le_ctr[1]++; + aes_ctr64_x86(dst + part1_len, src + part1_len, len - part1_len, + le_ctr, key); + } + kernel_fpu_end(); + put_unaligned_be64(ctr64, &ctr[8]); + put_unaligned_be64(le_ctr[1], &ctr[0]); + return true; +} +#endif /* CONFIG_CRYPTO_LIB_AES_CTR && CONFIG_X86_64 */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6990934F24E; Mon, 21 Sep 2026 05:16:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967768; cv=none; b=W/4ps9zRg+b2lx3eCnT6Ns2eHJm11vFZ9E1NkbHGxvNfk9Xqu6wjOH/voUCBkY302n2UE/IaC7ByeX49ZFyZvZVTvnrpNg1HmSS8a/Gek0xV2O7mIkzA5lTXTOSo4GbEyW87G2W+50B4HVJIxLUK2aiGLp2DGRmDzWeWOxLKSbQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967768; c=relaxed/simple; bh=vqZeM33mhFWvjBrHHGcT7w+sbnduuHJ6FS6RuzD3N4Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nU/7iwiByOUg3x5+LFa0mFbWgT6jPPguyfUYi3zYSFODP8uDc6NHKuZZe+ezIj3xxrQ9/tdC0XHzNNIefycDyXXGFtflvS2NyHXHGy/FF8TEszjAndk8WRt4UH9F00HR2ry7zsn7KYa+tNTELqTo86NNkowWeGZINTrgF0krkvs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=X48SLngo; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="X48SLngo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B26FB1F000FF; Mon, 21 Sep 2026 05:16:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967766; bh=xrbxpAIudFXLQj0+9acngXmOgr6mUBsVSCepUOvq0KU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=X48SLngoUJu+xvQlChDPm/opx1y8tBFRk2E8KAGZVoWao1w0+eZvYd/EhB5W2FUxN pFtHV/Z2PRjWXxQ03LIpqaRJVOpJ9M5hmnemuZyob5jOBmQNKuJY+8lMbfS5LsHATL JdhgbSZXOa6i4Td8mM3Nz2V0rmeCuMaDhmdEJBRcM9r5DblbFXiw734yuD1fIF7V6a aQFkPvv65XWZW4jbHYuPMV5xMB1I/CpVUZznO0Ywc2L7PewYrU4ZvqdO0jRKJkGh6a MMi0/5kbZzSV7Z3AeE2uT7Z2y5b0IzOFBA8rrs0fGUlV3bIWQutYqE72sunSTOXKqb EK3EWhWT517Yw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 07/20] lib/crypto: x86/aes-xts: Add AES-NI optimization Date: Sun, 20 Sep 2026 22:08:53 -0700 Message-ID: <20260921050910.296144-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Optimize the crypto library's AES-XTS support with AES-NI, making its performance be at least at parity with the "xts-aes-aesni" skcipher algorithm that it will supersede. The new assembly function is written from scratch to fit well into the crypto library and to be more consistent with aes-xts-avx-x86_64.S than the code in arch/x86/crypto/aesni-intel_asm.S that it will supersede. At a high level it is quite similar though, including doing 4 blocks per iteration and supporting 32-bit mode for parity with the old code. Note: the priority of xts-aes-lib is left unchanged at 110 temporarily. It will be increased when the AVX-optimized code is migrated too. Signed-off-by: Eric Biggers --- lib/crypto/x86/aes-aesni.S | 146 +++++++++++++++++++++++++++++++++++++ lib/crypto/x86/aes.h | 44 +++++++++++ 2 files changed, 190 insertions(+) diff --git a/lib/crypto/x86/aes-aesni.S b/lib/crypto/x86/aes-aesni.S index 24c53f1a144b..50f09ac6ef34 100644 --- a/lib/crypto/x86/aes-aesni.S +++ b/lib/crypto/x86/aes-aesni.S @@ -50,6 +50,17 @@ =20 .section .rodata .p2align 4 +.Lxts_gf_poly: + // For XTS: a constant used when advancing the tweak by one block by + // multiplying by the polynomial 'x' in GF(2^128). The low 64 bits of + // this value represent the polynomial x^7 + x^2 + x + 1; it is the + // value that must be XOR'd into the low 64 bits of the tweak each time + // a 1 is carried out of the high 64 bits. + // + // The high 64 bits of this value is just the internal carry bit that + // exists when there's a carry out of the low 64 bits of the tweak. + .quad 0x87, 1 + #ifdef __x86_64__ .Lbswap_mask: // A mask for pshufb that byte-reflects the value. @@ -776,3 +787,138 @@ SYM_FUNC_START(aes_ctr64_crypt_aesni) RET SYM_FUNC_END(aes_ctr64_crypt_aesni) #endif // __x86_64__ + +// Given a 128-bit XTS tweak in the xmm register \tweak, compute the next = tweak +// (by multiplying by the polynomial 'x') and write it back to \tweak. +.macro _next_tweak tweak, tmp + pshufd $0x13, \tweak, \tmp + paddq \tweak, \tweak + psrad $31, \tmp + pand GF_POLY, \tmp + pxor \tmp, \tweak +.endm + +.macro _aes_xts_crypt enc + // Arguments + .set DST, ARG0 + .set SRC, ARG1 + .set NBLOCKS, ARG2 + .set NBLOCKS32, ARG2_32 // Used for improved code density + .set TWEAK_PTR, ARG3 + .set KEY, ARG4 + + // Other local variables +#ifdef __x86_64__ + .set RNDKEY_PTR, %r9 +#else + .set RNDKEY_PTR, TWEAK_PTR // TWEAK_PTR is clobbered and reloaded later. +#endif + .set NROUNDS, TMP_32 + .set AESDATA0, %xmm0 + .set AESDATA1, %xmm1 + .set AESDATA2, %xmm2 + .set AESDATA3, %xmm3 + .set GF_POLY, %xmm4 + .set RNDKEY, %xmm5 + .set TWEAK, %xmm6 + .set SAVED_TWEAK0, %xmm7 +#ifdef __x86_64__ + .set SAVED_TWEAK1, %xmm8 + .set SAVED_TWEAK2, %xmm9 +#endif + + _prologue uses_arg3=3D2, uses_arg4=3D2 + + movdqu (TWEAK_PTR), TWEAK + movdqa RODATA(.Lxts_gf_poly), GF_POLY + + sub $4, NBLOCKS + jl .Lxts_loop4_done\@ +.p2align 5 +.Lxts_loop4\@: + // Load the next four source blocks into AESDATA[0-3] and XOR them with + // their tweaks, advancing the tweak three times in order to do so. + // Save the four tweaks for later; on 64-bit they all fit into + // registers, while on 32-bit two tweaks are spilled to DST. +.irp i, 0,1,2,3 + movdqu \i*16(SRC), AESDATA\i + pxor TWEAK, AESDATA\i + .if \i !=3D 3 +#ifdef __x86_64__ + movdqa TWEAK, SAVED_TWEAK\i +#else + .if \i =3D=3D 0 + movdqa TWEAK, SAVED_TWEAK0 + .else + movdqu TWEAK, (\i-1)*16(DST) + .endif +#endif + _next_tweak TWEAK, RNDKEY + .endif +.endr + + // Encrypt or decrypt the blocks. + _do_aes \enc, 0,1,2,3 + + // XOR the blocks with the saved tweaks. + pxor SAVED_TWEAK0, AESDATA0 +#ifdef __x86_64__ + pxor SAVED_TWEAK1, AESDATA1 + pxor SAVED_TWEAK2, AESDATA2 +#else + movdqu 0(DST), RNDKEY + pxor RNDKEY, AESDATA1 + movdqu 16(DST), RNDKEY + pxor RNDKEY, AESDATA2 +#endif + pxor TWEAK, AESDATA3 + + // Store the encrypted or decrypted blocks. +.irp i, 0,1,2,3 + movdqu AESDATA\i, \i*16(DST) +.endr + + _next_tweak TWEAK, RNDKEY + add $64, DST + add $64, SRC + sub $4, NBLOCKS + jge .Lxts_loop4\@ +.Lxts_loop4_done\@: + add $4, NBLOCKS32 + jz .Lxts_done\@ + +.Lxts_loop1\@: + movdqu (SRC), AESDATA0 + pxor TWEAK, AESDATA0 + _do_aes \enc, 0 + pxor TWEAK, AESDATA0 + movdqu AESDATA0, (DST) + _next_tweak TWEAK, RNDKEY + add $16, DST + add $16, SRC + dec NBLOCKS32 + jnz .Lxts_loop1\@ + +.Lxts_done\@: + // Store the next tweak. On 32-bit, reload TWEAK_PTR from stack first. + _reload_arg3 + movdqu TWEAK, (TWEAK_PTR) + _epilogue +.endm + +// void aes_xts_encrypt_aesni(u8 *dst, const u8 *src, long nblocks, +// u8 tweak[AES_BLOCK_SIZE], +// const struct aes_key *key); +// void aes_xts_decrypt_aesni(u8 *dst, const u8 *src, long nblocks, +// u8 tweak[AES_BLOCK_SIZE], +// const struct aes_key *key); +// +// `tweak` must have already been encrypted by the tweak key; `key` is jus= t the +// main key. To allow incremental computation, `tweak` is updated to cont= ain +// the next tweak. +SYM_FUNC_START(aes_xts_encrypt_aesni) + _aes_xts_crypt 1 +SYM_FUNC_END(aes_xts_encrypt_aesni) +SYM_FUNC_START(aes_xts_decrypt_aesni) + _aes_xts_crypt 0 +SYM_FUNC_END(aes_xts_decrypt_aesni) diff --git a/lib/crypto/x86/aes.h b/lib/crypto/x86/aes.h index 685b43ce6ef0..def9799302c1 100644 --- a/lib/crypto/x86/aes.h +++ b/lib/crypto/x86/aes.h @@ -257,6 +257,50 @@ static bool aes_ctr_arch(u8 *dst, const u8 *src, size_= t len, } #endif /* CONFIG_CRYPTO_LIB_AES_CTR && CONFIG_X86_64 */ =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_XTS) +void aes_xts_encrypt_aesni(u8 *dst, const u8 *src, long nblocks, + u8 tweak[AES_BLOCK_SIZE], const struct aes_key *key); +void aes_xts_decrypt_aesni(u8 *dst, const u8 *src, long nblocks, + u8 tweak[AES_BLOCK_SIZE], const struct aes_key *key); + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +static __always_inline bool +aes_xts_crypt_x86(u8 *dst, const u8 *src, size_t len, u8 tweak[AES_BLOCK_S= IZE], + const struct aes_xts_key *key, bool cont, bool enc) +{ + const long nblocks =3D len / AES_BLOCK_SIZE; + + if (!static_branch_likely(&have_aesni) || unlikely(!irq_fpu_usable())) + return false; + + kernel_fpu_begin(); + if (!cont) + aes_encrypt_aesni(tweak, tweak, &key->tweak_key); + if (enc) + aes_xts_encrypt_aesni(dst, src, nblocks, tweak, &key->main_key); + else + aes_xts_decrypt_aesni(dst, src, nblocks, tweak, &key->main_key); + kernel_fpu_end(); + return true; +} + +#define aes_xts_encrypt_arch aes_xts_encrypt_arch +static bool aes_xts_encrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 tweak[AES_BLOCK_SIZE], + const struct aes_xts_key *key, bool cont) +{ + return aes_xts_crypt_x86(dst, src, len, tweak, key, cont, true); +} + +#define aes_xts_decrypt_arch aes_xts_decrypt_arch +static bool aes_xts_decrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 tweak[AES_BLOCK_SIZE], + const struct aes_xts_key *key, bool cont) +{ + return aes_xts_crypt_x86(dst, src, len, tweak, key, cont, false); +} +#endif /* CONFIG_CRYPTO_LIB_AES_XTS */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE8B935AC3E; Mon, 21 Sep 2026 05:16:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967768; cv=none; b=lPQoXVf7rNA2cfxHf2WZx3oVAjEt9WEPemPF6X+b9fQsOC0QVhfeIQeIVZyqMqNB9BpzhKjB3N7YUwynOmS3jXmiI7HYBiJycl5CLVdAFK9sXgiK2ApaWA1qD2Cm6Ulr9Bm4QVLV3/ir5/bd2Rm58KVudM5URz4/yv704H8+qPI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967768; c=relaxed/simple; bh=m8It4hVcmkfuLc/bJZoRNi3ysLXZIOXJfH879b0LBMk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NiduznrTXfyVH/hyh4uax1d5hA97u0F8YbNjOxY4Jzp//H2QrUeXOmrj4FgPRPV2SlrKgRYgQqlX5eBoVWROrkoH5/jB0s40TIlqt4w2RpJ1V0xmfpjW3Ixe3SbznsBpVzjBzOFbREc29BvVXI70uOJBKl6WSUaRcRZHHjTw6vs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aOGxvqZT; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aOGxvqZT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1884B1F00899; Mon, 21 Sep 2026 05:16:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967766; bh=mlQyMsJiv5qUwYGxCdI53YiBa6XzV8Yve52rPyxXedA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=aOGxvqZT5zaFfD40iwgE0np9/MJ+a+abKSfw/YV1o1zaMPyGd5SspbV5/FKIRFiik XWQk8MZK9PKBpHs9wLCbL72stKyd/yTKJ4mEB1hQL8HPBX9YSjCWa88W79tjsP+flm lFrlpNQr5KuwFHNYNO6PmSiVDGa7LiGFEiLKJ8KzrKUwZlNHARjwgLzSrAUwLI6pet GAjYJTxXw0ButJpoedXTZuZh4nt9IZIxDU96uYoxWbN9eYWk5KEuIF2F+BnTSVe2TD hMVoiuxYSH675w3M+zVMF6H5wI95m6JAW/tlflKeDsijCraawhNMCrqXJuTChhjs1m +4EoubPJiwvVg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 08/20] crypto: x86/aes-ecb - Remove superseded ECB skcipher Date: Sun, 20 Sep 2026 22:08:54 -0700 Message-ID: <20260921050910.296144-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the AES-ECB library functions are optimized with dedicated AES-NI code and exposed via the skcipher API via crypto/aes.c, the similar implementation in aesni-intel is redundant. Remove it. Signed-off-by: Eric Biggers --- arch/x86/crypto/Kconfig | 4 +- arch/x86/crypto/aesni-intel_asm.S | 121 ----------------------------- arch/x86/crypto/aesni-intel_glue.c | 62 --------------- 3 files changed, 2 insertions(+), 185 deletions(-) diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index f65d7b83702f..ecd4931557a3 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -3,14 +3,14 @@ menu "Accelerated Cryptographic Algorithms for CPU (x86)" =20 config CRYPTO_AES_NI_INTEL - tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XCTR, XTS, GCM (AES-NI= /VAES)" + tristate "Ciphers: AES, modes: CBC, CTS, CTR, XCTR, XTS, GCM (AES-NI/VAES= )" select CRYPTO_AEAD select CRYPTO_LIB_AES select CRYPTO_LIB_GF128MUL select CRYPTO_SKCIPHER help AEAD cipher: AES with GCM - Length-preserving ciphers: AES with ECB, CBC, CTS, CTR, XCTR, XTS + Length-preserving ciphers: AES with CBC, CTS, CTR, XCTR, XTS =20 Architecture: x86 (32-bit and 64-bit) using: - AES-NI (AES new instructions) diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-inte= l_asm.S index 6abe5e38a6d7..db7f259ee8ab 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -601,127 +601,6 @@ SYM_FUNC_START_LOCAL(_aesni_dec4) RET SYM_FUNC_END(_aesni_dec4) =20 -/* - * void aesni_ecb_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len) - */ -SYM_FUNC_START(aesni_ecb_enc) - FRAME_BEGIN -#ifndef __x86_64__ - pushl LEN - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+16)(%esp), KEYP # ctx - movl (FRAME_OFFSET+20)(%esp), OUTP # dst - movl (FRAME_OFFSET+24)(%esp), INP # src - movl (FRAME_OFFSET+28)(%esp), LEN # len -#endif - test LEN, LEN # check length - jz .Lecb_enc_ret - mov 480(KEYP), KLEN - cmp $16, LEN - jb .Lecb_enc_ret - cmp $64, LEN - jb .Lecb_enc_loop1 -.align 4 -.Lecb_enc_loop4: - movups (INP), STATE1 - movups 0x10(INP), STATE2 - movups 0x20(INP), STATE3 - movups 0x30(INP), STATE4 - call _aesni_enc4 - movups STATE1, (OUTP) - movups STATE2, 0x10(OUTP) - movups STATE3, 0x20(OUTP) - movups STATE4, 0x30(OUTP) - sub $64, LEN - add $64, INP - add $64, OUTP - cmp $64, LEN - jge .Lecb_enc_loop4 - cmp $16, LEN - jb .Lecb_enc_ret -.align 4 -.Lecb_enc_loop1: - movups (INP), STATE1 - call _aesni_enc1 - movups STATE1, (OUTP) - sub $16, LEN - add $16, INP - add $16, OUTP - cmp $16, LEN - jge .Lecb_enc_loop1 -.Lecb_enc_ret: -#ifndef __x86_64__ - popl KLEN - popl KEYP - popl LEN -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_ecb_enc) - -/* - * void aesni_ecb_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len); - */ -SYM_FUNC_START(aesni_ecb_dec) - FRAME_BEGIN -#ifndef __x86_64__ - pushl LEN - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+16)(%esp), KEYP # ctx - movl (FRAME_OFFSET+20)(%esp), OUTP # dst - movl (FRAME_OFFSET+24)(%esp), INP # src - movl (FRAME_OFFSET+28)(%esp), LEN # len -#endif - test LEN, LEN - jz .Lecb_dec_ret - mov 480(KEYP), KLEN - add $240, KEYP - cmp $16, LEN - jb .Lecb_dec_ret - cmp $64, LEN - jb .Lecb_dec_loop1 -.align 4 -.Lecb_dec_loop4: - movups (INP), STATE1 - movups 0x10(INP), STATE2 - movups 0x20(INP), STATE3 - movups 0x30(INP), STATE4 - call _aesni_dec4 - movups STATE1, (OUTP) - movups STATE2, 0x10(OUTP) - movups STATE3, 0x20(OUTP) - movups STATE4, 0x30(OUTP) - sub $64, LEN - add $64, INP - add $64, OUTP - cmp $64, LEN - jge .Lecb_dec_loop4 - cmp $16, LEN - jb .Lecb_dec_ret -.align 4 -.Lecb_dec_loop1: - movups (INP), STATE1 - call _aesni_dec1 - movups STATE1, (OUTP) - sub $16, LEN - add $16, INP - add $16, OUTP - cmp $16, LEN - jge .Lecb_dec_loop1 -.Lecb_dec_ret: -#ifndef __x86_64__ - popl KLEN - popl KEYP - popl LEN -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_ecb_dec) - /* * void aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, * size_t len, u8 *iv) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index f522fff9231e..f3b9cfb0b813 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -61,10 +61,6 @@ static inline void *aes_align_addr(void *addr) asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in); -asmlinkage void aesni_ecb_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len); -asmlinkage void aesni_ecb_dec(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len); asmlinkage void aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); asmlinkage void aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, @@ -119,50 +115,6 @@ static int aesni_skcipher_setkey(struct crypto_skciphe= r *tfm, const u8 *key, return aes_set_key_common(aes_ctx(crypto_skcipher_ctx(tfm)), key, len); } =20 -static int ecb_encrypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx =3D aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - - while ((nbytes =3D walk.nbytes)) { - kernel_fpu_begin(); - aesni_ecb_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK); - kernel_fpu_end(); - nbytes &=3D AES_BLOCK_SIZE - 1; - err =3D skcipher_walk_done(&walk, nbytes); - } - - return err; -} - -static int ecb_decrypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx =3D aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - - while ((nbytes =3D walk.nbytes)) { - kernel_fpu_begin(); - aesni_ecb_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK); - kernel_fpu_end(); - nbytes &=3D AES_BLOCK_SIZE - 1; - err =3D skcipher_walk_done(&walk, nbytes); - } - - return err; -} - static int cbc_encrypt(struct skcipher_request *req) { struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); @@ -513,20 +465,6 @@ static int xts_decrypt_aesni(struct skcipher_request *= req) =20 static struct skcipher_alg aesni_skciphers[] =3D { { - .base =3D { - .cra_name =3D "ecb(aes)", - .cra_driver_name =3D "ecb-aes-aesni", - .cra_priority =3D 400, - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D CRYPTO_AES_CTX_SIZE, - .cra_module =3D THIS_MODULE, - }, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .setkey =3D aesni_skcipher_setkey, - .encrypt =3D ecb_encrypt, - .decrypt =3D ecb_decrypt, - }, { .base =3D { .cra_name =3D "cbc(aes)", .cra_driver_name =3D "cbc-aes-aesni", --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E942635C689; Mon, 21 Sep 2026 05:16:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967770; cv=none; b=S+hIi2za5HwYeGOeKBgjfCORbuzb64IZ3t80GJ/+wRdnw24VJ2k0PAhPsLkgQz0/aegvPHDHa7zL54GR5R6qrrf/udHyBOsghltdji7oTTTlTQJkeUtaYLWIcyPT4Rp8TMhvSdZt46daw+ek58XjNJPIt6gOAAuRTtoctqldDag= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967770; c=relaxed/simple; bh=TvLg+dq279b46+A4Gq7t5hNia4AK/Oag2NDKCoX56A0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lkRgW3Ifrxz8LGufPxre6O2iNOZy7jDuTurQcmJWq1WYbwytc7xoRJDDBo8iLoAo6Y09AyJiv+WtTOO1C1q6v2UJuulIZ+vJUHm7xgnFkUH84zPNxBeAHWWHWnwGYf7gzxHqPIaEdBELr6KWtc7aQa/IGjXL+ZESf/GZ3FQIMgw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=f4twR0Dp; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="f4twR0Dp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7367B1F0089F; Mon, 21 Sep 2026 05:16:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967766; bh=C4Sum1Eo67MmuYPujINFCyTxdf5JAlfxhIYWhng7ASo=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=f4twR0Dpt6hubSLnA+MUarIVMkfLkmGdDKjE8d/AABHLr+UqEaknCOa0iRm7FuQfe ZNwAaO6LWHpt0Z4Lz04myMYW8bvd+6FKZ+3PKMineMOxRg8s0FcQZMIoSZcnxLLm/E QZ8CNVEX62k0p67SVcdxGR06pcZCiWTc07YdlWz3tFsQYLUWJxuva2osdzVtaoqziI vn1vm2d0Z9Di8Gi4UMxBCzlFx/1p7lSHw2kFNolBTe1WWv73BvHUaB9gO8F/kYK1eV f1KLcMLF+DctppE9ie0tidAKx4BPyvMwuqh3TCVwXIGc4xuXJ1SweiOgkTXhFHv/HU bcAEKgHbp729g== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 09/20] crypto: x86/aes-cbc - Remove superseded CBC skciphers Date: Sun, 20 Sep 2026 22:08:55 -0700 Message-ID: <20260921050910.296144-10-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the AES-CBC (and AES-CBC-CTS) library functions are optimized with dedicated AES-NI code and exposed via the skcipher API via crypto/aes.c, the similar implementation in aesni-intel is redundant. Remove it. Signed-off-by: Eric Biggers --- arch/x86/crypto/Kconfig | 4 +- arch/x86/crypto/aesni-intel_asm.S | 255 ----------------------------- arch/x86/crypto/aesni-intel_glue.c | 200 +--------------------- 3 files changed, 5 insertions(+), 454 deletions(-) diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index ecd4931557a3..6dbf5e083966 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -3,14 +3,14 @@ menu "Accelerated Cryptographic Algorithms for CPU (x86)" =20 config CRYPTO_AES_NI_INTEL - tristate "Ciphers: AES, modes: CBC, CTS, CTR, XCTR, XTS, GCM (AES-NI/VAES= )" + tristate "Ciphers: AES, modes: CTR, XCTR, XTS, GCM (AES-NI/VAES)" select CRYPTO_AEAD select CRYPTO_LIB_AES select CRYPTO_LIB_GF128MUL select CRYPTO_SKCIPHER help AEAD cipher: AES with GCM - Length-preserving ciphers: AES with CBC, CTS, CTR, XCTR, XTS + Length-preserving ciphers: AES with CTR, XCTR, XTS =20 Architecture: x86 (32-bit and 64-bit) using: - AES-NI (AES new instructions) diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-inte= l_asm.S index db7f259ee8ab..16c406781b24 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -601,261 +601,6 @@ SYM_FUNC_START_LOCAL(_aesni_dec4) RET SYM_FUNC_END(_aesni_dec4) =20 -/* - * void aesni_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) - */ -SYM_FUNC_START(aesni_cbc_enc) - FRAME_BEGIN -#ifndef __x86_64__ - pushl IVP - pushl LEN - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+20)(%esp), KEYP # ctx - movl (FRAME_OFFSET+24)(%esp), OUTP # dst - movl (FRAME_OFFSET+28)(%esp), INP # src - movl (FRAME_OFFSET+32)(%esp), LEN # len - movl (FRAME_OFFSET+36)(%esp), IVP # iv -#endif - cmp $16, LEN - jb .Lcbc_enc_ret - mov 480(KEYP), KLEN - movups (IVP), STATE # load iv as initial state -.align 4 -.Lcbc_enc_loop: - movups (INP), IN # load input - pxor IN, STATE - call _aesni_enc1 - movups STATE, (OUTP) # store output - sub $16, LEN - add $16, INP - add $16, OUTP - cmp $16, LEN - jge .Lcbc_enc_loop - movups STATE, (IVP) -.Lcbc_enc_ret: -#ifndef __x86_64__ - popl KLEN - popl KEYP - popl LEN - popl IVP -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_cbc_enc) - -/* - * void aesni_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) - */ -SYM_FUNC_START(aesni_cbc_dec) - FRAME_BEGIN -#ifndef __x86_64__ - pushl IVP - pushl LEN - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+20)(%esp), KEYP # ctx - movl (FRAME_OFFSET+24)(%esp), OUTP # dst - movl (FRAME_OFFSET+28)(%esp), INP # src - movl (FRAME_OFFSET+32)(%esp), LEN # len - movl (FRAME_OFFSET+36)(%esp), IVP # iv -#endif - cmp $16, LEN - jb .Lcbc_dec_just_ret - mov 480(KEYP), KLEN - add $240, KEYP - movups (IVP), IV - cmp $64, LEN - jb .Lcbc_dec_loop1 -.align 4 -.Lcbc_dec_loop4: - movups (INP), IN1 - movaps IN1, STATE1 - movups 0x10(INP), IN2 - movaps IN2, STATE2 -#ifdef __x86_64__ - movups 0x20(INP), IN3 - movaps IN3, STATE3 - movups 0x30(INP), IN4 - movaps IN4, STATE4 -#else - movups 0x20(INP), IN1 - movaps IN1, STATE3 - movups 0x30(INP), IN2 - movaps IN2, STATE4 -#endif - call _aesni_dec4 - pxor IV, STATE1 -#ifdef __x86_64__ - pxor IN1, STATE2 - pxor IN2, STATE3 - pxor IN3, STATE4 - movaps IN4, IV -#else - pxor IN1, STATE4 - movaps IN2, IV - movups (INP), IN1 - pxor IN1, STATE2 - movups 0x10(INP), IN2 - pxor IN2, STATE3 -#endif - movups STATE1, (OUTP) - movups STATE2, 0x10(OUTP) - movups STATE3, 0x20(OUTP) - movups STATE4, 0x30(OUTP) - sub $64, LEN - add $64, INP - add $64, OUTP - cmp $64, LEN - jge .Lcbc_dec_loop4 - cmp $16, LEN - jb .Lcbc_dec_ret -.align 4 -.Lcbc_dec_loop1: - movups (INP), IN - movaps IN, STATE - call _aesni_dec1 - pxor IV, STATE - movups STATE, (OUTP) - movaps IN, IV - sub $16, LEN - add $16, INP - add $16, OUTP - cmp $16, LEN - jge .Lcbc_dec_loop1 -.Lcbc_dec_ret: - movups IV, (IVP) -.Lcbc_dec_just_ret: -#ifndef __x86_64__ - popl KLEN - popl KEYP - popl LEN - popl IVP -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_cbc_dec) - -/* - * void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *s= rc, - * size_t len, u8 *iv) - */ -SYM_FUNC_START(aesni_cts_cbc_enc) - FRAME_BEGIN -#ifndef __x86_64__ - pushl IVP - pushl LEN - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+20)(%esp), KEYP # ctx - movl (FRAME_OFFSET+24)(%esp), OUTP # dst - movl (FRAME_OFFSET+28)(%esp), INP # src - movl (FRAME_OFFSET+32)(%esp), LEN # len - movl (FRAME_OFFSET+36)(%esp), IVP # iv - lea .Lcts_permute_table, T1 -#else - lea .Lcts_permute_table(%rip), T1 -#endif - mov 480(KEYP), KLEN - movups (IVP), STATE - sub $16, LEN - mov T1, IVP - add $32, IVP - add LEN, T1 - sub LEN, IVP - movups (T1), %xmm4 - movups (IVP), %xmm5 - - movups (INP), IN1 - add LEN, INP - movups (INP), IN2 - - pxor IN1, STATE - call _aesni_enc1 - - pshufb %xmm5, IN2 - pxor STATE, IN2 - pshufb %xmm4, STATE - add OUTP, LEN - movups STATE, (LEN) - - movaps IN2, STATE - call _aesni_enc1 - movups STATE, (OUTP) - -#ifndef __x86_64__ - popl KLEN - popl KEYP - popl LEN - popl IVP -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_cts_cbc_enc) - -/* - * void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *s= rc, - * size_t len, u8 *iv) - */ -SYM_FUNC_START(aesni_cts_cbc_dec) - FRAME_BEGIN -#ifndef __x86_64__ - pushl IVP - pushl LEN - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+20)(%esp), KEYP # ctx - movl (FRAME_OFFSET+24)(%esp), OUTP # dst - movl (FRAME_OFFSET+28)(%esp), INP # src - movl (FRAME_OFFSET+32)(%esp), LEN # len - movl (FRAME_OFFSET+36)(%esp), IVP # iv - lea .Lcts_permute_table, T1 -#else - lea .Lcts_permute_table(%rip), T1 -#endif - mov 480(KEYP), KLEN - add $240, KEYP - movups (IVP), IV - sub $16, LEN - mov T1, IVP - add $32, IVP - add LEN, T1 - sub LEN, IVP - movups (T1), %xmm4 - - movups (INP), STATE - add LEN, INP - movups (INP), IN1 - - call _aesni_dec1 - movaps STATE, IN2 - pshufb %xmm4, STATE - pxor IN1, STATE - - add OUTP, LEN - movups STATE, (LEN) - - movups (IVP), %xmm0 - pshufb %xmm0, IN1 - pblendvb IN2, IN1 - movaps IN1, STATE - call _aesni_dec1 - - pxor IV, STATE - movups STATE, (OUTP) - -#ifndef __x86_64__ - popl KLEN - popl KEYP - popl LEN - popl IVP -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_cts_cbc_dec) - .pushsection .rodata .align 16 .Lcts_permute_table: diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index f3b9cfb0b813..00b74acd01bd 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -61,14 +61,6 @@ static inline void *aes_align_addr(void *addr) asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in); -asmlinkage void aesni_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); -asmlinkage void aesni_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); -asmlinkage void aesni_cts_cbc_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); -asmlinkage void aesni_cts_cbc_dec(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); =20 asmlinkage void aesni_xts_enc(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); @@ -115,162 +107,6 @@ static int aesni_skcipher_setkey(struct crypto_skciph= er *tfm, const u8 *key, return aes_set_key_common(aes_ctx(crypto_skcipher_ctx(tfm)), key, len); } =20 -static int cbc_encrypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx =3D aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - - while ((nbytes =3D walk.nbytes)) { - kernel_fpu_begin(); - aesni_cbc_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, walk.iv); - kernel_fpu_end(); - nbytes &=3D AES_BLOCK_SIZE - 1; - err =3D skcipher_walk_done(&walk, nbytes); - } - - return err; -} - -static int cbc_decrypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx =3D aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - - while ((nbytes =3D walk.nbytes)) { - kernel_fpu_begin(); - aesni_cbc_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, walk.iv); - kernel_fpu_end(); - nbytes &=3D AES_BLOCK_SIZE - 1; - err =3D skcipher_walk_done(&walk, nbytes); - } - - return err; -} - -static int cts_cbc_encrypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx =3D aes_ctx(crypto_skcipher_ctx(tfm)); - int cbc_blocks =3D DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; - struct scatterlist *src =3D req->src, *dst =3D req->dst; - struct scatterlist sg_src[2], sg_dst[2]; - struct skcipher_request subreq; - struct skcipher_walk walk; - int err; - - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, skcipher_request_flags(req), - NULL, NULL); - - if (req->cryptlen <=3D AES_BLOCK_SIZE) { - if (req->cryptlen < AES_BLOCK_SIZE) - return -EINVAL; - cbc_blocks =3D 1; - } - - if (cbc_blocks > 0) { - skcipher_request_set_crypt(&subreq, req->src, req->dst, - cbc_blocks * AES_BLOCK_SIZE, - req->iv); - - err =3D cbc_encrypt(&subreq); - if (err) - return err; - - if (req->cryptlen =3D=3D AES_BLOCK_SIZE) - return 0; - - dst =3D src =3D scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen); - if (req->dst !=3D req->src) - dst =3D scatterwalk_ffwd(sg_dst, req->dst, - subreq.cryptlen); - } - - /* handle ciphertext stealing */ - skcipher_request_set_crypt(&subreq, src, dst, - req->cryptlen - cbc_blocks * AES_BLOCK_SIZE, - req->iv); - - err =3D skcipher_walk_virt(&walk, &subreq, false); - if (err) - return err; - - kernel_fpu_begin(); - aesni_cts_cbc_enc(ctx, walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - kernel_fpu_end(); - - return skcipher_walk_done(&walk, 0); -} - -static int cts_cbc_decrypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx =3D aes_ctx(crypto_skcipher_ctx(tfm)); - int cbc_blocks =3D DIV_ROUND_UP(req->cryptlen, AES_BLOCK_SIZE) - 2; - struct scatterlist *src =3D req->src, *dst =3D req->dst; - struct scatterlist sg_src[2], sg_dst[2]; - struct skcipher_request subreq; - struct skcipher_walk walk; - int err; - - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, skcipher_request_flags(req), - NULL, NULL); - - if (req->cryptlen <=3D AES_BLOCK_SIZE) { - if (req->cryptlen < AES_BLOCK_SIZE) - return -EINVAL; - cbc_blocks =3D 1; - } - - if (cbc_blocks > 0) { - skcipher_request_set_crypt(&subreq, req->src, req->dst, - cbc_blocks * AES_BLOCK_SIZE, - req->iv); - - err =3D cbc_decrypt(&subreq); - if (err) - return err; - - if (req->cryptlen =3D=3D AES_BLOCK_SIZE) - return 0; - - dst =3D src =3D scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen); - if (req->dst !=3D req->src) - dst =3D scatterwalk_ffwd(sg_dst, req->dst, - subreq.cryptlen); - } - - /* handle ciphertext stealing */ - skcipher_request_set_crypt(&subreq, src, dst, - req->cryptlen - cbc_blocks * AES_BLOCK_SIZE, - req->iv); - - err =3D skcipher_walk_virt(&walk, &subreq, false); - if (err) - return err; - - kernel_fpu_begin(); - aesni_cts_cbc_dec(ctx, walk.dst.virt.addr, walk.src.virt.addr, - walk.nbytes, walk.iv); - kernel_fpu_end(); - - return skcipher_walk_done(&walk, 0); -} - #ifdef CONFIG_X86_64 /* This is the non-AVX version. */ static int ctr_crypt_aesni(struct skcipher_request *req) @@ -464,39 +300,8 @@ static int xts_decrypt_aesni(struct skcipher_request *= req) } =20 static struct skcipher_alg aesni_skciphers[] =3D { - { - .base =3D { - .cra_name =3D "cbc(aes)", - .cra_driver_name =3D "cbc-aes-aesni", - .cra_priority =3D 400, - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D CRYPTO_AES_CTX_SIZE, - .cra_module =3D THIS_MODULE, - }, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .setkey =3D aesni_skcipher_setkey, - .encrypt =3D cbc_encrypt, - .decrypt =3D cbc_decrypt, - }, { - .base =3D { - .cra_name =3D "cts(cbc(aes))", - .cra_driver_name =3D "cts-cbc-aes-aesni", - .cra_priority =3D 400, - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D CRYPTO_AES_CTX_SIZE, - .cra_module =3D THIS_MODULE, - }, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .walksize =3D 2 * AES_BLOCK_SIZE, - .setkey =3D aesni_skcipher_setkey, - .encrypt =3D cts_cbc_encrypt, - .decrypt =3D cts_cbc_decrypt, #ifdef CONFIG_X86_64 - }, { + { .base =3D { .cra_name =3D "ctr(aes)", .cra_driver_name =3D "ctr-aes-aesni", @@ -512,8 +317,9 @@ static struct skcipher_alg aesni_skciphers[] =3D { .setkey =3D aesni_skcipher_setkey, .encrypt =3D ctr_crypt_aesni, .decrypt =3D ctr_crypt_aesni, + }, #endif - }, { + { .base =3D { .cra_name =3D "xts(aes)", .cra_driver_name =3D "xts-aes-aesni", --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBCB535E1AD; Mon, 21 Sep 2026 05:16:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967770; cv=none; b=oaXL4eXSQm3MFDJJ+0Pb9gCa0gvUhXGAAPok5XChwH4tS4zsV0kkRyy+fL0WklcZfx9dECC5IXVa/Vcko/2n9hp70laSUcKsTE3fC0E9pop1xn0cTUOP1zwxKYDToVre1tLQXI26t1p63a7lUZru/B6ppoH48w9a743Ji1aeFzQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967770; c=relaxed/simple; bh=Abgb3bfjc8JiCh+4D+HkRFdBmtZGT2b9NyaqT/809wM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nLAmyrOID66mbfo9gjTgP/aNqiYnrZCbV0JRJQldqhmsVpXDGwG8vv+p/ZzYYBvdDkoQ0nHTcqyyvURhmA6f2Dcjk4pEeMqowgZvwtPvE8kSMRJNTO48xGrGnIPp21rM8Db/4SclK42Ligt3f4VvSPmLeTYOueGjRHzOt90tTg4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UGjJTpob; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UGjJTpob" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CEE4E1F0089D; Mon, 21 Sep 2026 05:16:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967767; bh=W+V1/Q6lb2YF3sLGA0HiySZ/5F17viwdUxWS8D02K14=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=UGjJTpobLoggxG2oZd7jBpjnwA32p7ppsSycTiiALJIm9qJAyrXKdzNCcmt029LyP kuXX+mexKTH/8OE6sBuq5k8fPOBOmGZ5mRZKVgDwq921FvGoXj9CDwqdEIZOhoyI9U 6AfIN0BTfoGwCgzKFrw+dGI2w3kFJj3vgsAVfS82BR1fuAcmCxH6GvFRWrYb4U94Hq WyjdRQT9R/WWGk6pDYlkNauijcZnYWdsnH0VuDvgc/l9M84L54APLfDidrUtarDO7t Vwdz8H/rO9S4k/kSVv7dofezGfT3NJ0DkTIeu4mQEIZguYqLG8rz1XHzTTlQ/8NNmb V5eyrpn17sxIw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 10/20] crypto: x86/aes-ctr - Remove superseded CTR skcipher Date: Sun, 20 Sep 2026 22:08:56 -0700 Message-ID: <20260921050910.296144-11-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the crypto library's AES-CTR support is optimized with AES-NI (without AVX), the similar code in aesni-intel is redundant. Remove it. This only affects the non-AVX implementation ("ctr-aes-aesni"), not aes-ctr-avx-x86_64.S which is handled later. Signed-off-by: Eric Biggers --- arch/x86/crypto/aesni-intel_asm.S | 125 ----------------------------- arch/x86/crypto/aesni-intel_glue.c | 60 -------------- 2 files changed, 185 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-inte= l_asm.S index 16c406781b24..c4d54c4a2c23 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -33,10 +33,6 @@ #define KEY %xmm2 #define IV %xmm3 =20 -#define BSWAP_MASK %xmm10 -#define CTR %xmm11 -#define INC %xmm12 - #define GF128MUL_MASK %xmm7 =20 #ifdef __x86_64__ @@ -51,7 +47,6 @@ #define T1 %r10 #define TKEYP T1 #define T2 %r11 -#define TCTR_LOW T2 #else #define AREG %eax #define KEYP %edi @@ -610,128 +605,8 @@ SYM_FUNC_END(_aesni_dec4) .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 -#ifdef __x86_64__ -.Lbswap_mask: - .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 -#endif .popsection =20 -#ifdef __x86_64__ -/* - * _aesni_inc_init: internal ABI - * setup registers used by _aesni_inc - * input: - * IV - * output: - * CTR: =3D=3D IV, in little endian - * TCTR_LOW: =3D=3D lower qword of CTR - * INC: =3D=3D 1, in little endian - * BSWAP_MASK =3D=3D endian swapping mask - */ -SYM_FUNC_START_LOCAL(_aesni_inc_init) - movaps .Lbswap_mask(%rip), BSWAP_MASK - movaps IV, CTR - pshufb BSWAP_MASK, CTR - mov $1, TCTR_LOW - movq TCTR_LOW, INC - movq CTR, TCTR_LOW - RET -SYM_FUNC_END(_aesni_inc_init) - -/* - * _aesni_inc: internal ABI - * Increase IV by 1, IV is in big endian - * input: - * IV - * CTR: =3D=3D IV, in little endian - * TCTR_LOW: =3D=3D lower qword of CTR - * INC: =3D=3D 1, in little endian - * BSWAP_MASK =3D=3D endian swapping mask - * output: - * IV: Increase by 1 - * changed: - * CTR: =3D=3D output IV, in little endian - * TCTR_LOW: =3D=3D lower qword of CTR - */ -SYM_FUNC_START_LOCAL(_aesni_inc) - paddq INC, CTR - add $1, TCTR_LOW - jnc .Linc_low - pslldq $8, INC - paddq INC, CTR - psrldq $8, INC -.Linc_low: - movaps CTR, IV - pshufb BSWAP_MASK, IV - RET -SYM_FUNC_END(_aesni_inc) - -/* - * void aesni_ctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src, - * size_t len, u8 *iv) - */ -SYM_FUNC_START(aesni_ctr_enc) - ANNOTATE_NOENDBR - FRAME_BEGIN - cmp $16, LEN - jb .Lctr_enc_just_ret - mov 480(KEYP), KLEN - movups (IVP), IV - call _aesni_inc_init - cmp $64, LEN - jb .Lctr_enc_loop1 -.align 4 -.Lctr_enc_loop4: - movaps IV, STATE1 - call _aesni_inc - movups (INP), IN1 - movaps IV, STATE2 - call _aesni_inc - movups 0x10(INP), IN2 - movaps IV, STATE3 - call _aesni_inc - movups 0x20(INP), IN3 - movaps IV, STATE4 - call _aesni_inc - movups 0x30(INP), IN4 - call _aesni_enc4 - pxor IN1, STATE1 - movups STATE1, (OUTP) - pxor IN2, STATE2 - movups STATE2, 0x10(OUTP) - pxor IN3, STATE3 - movups STATE3, 0x20(OUTP) - pxor IN4, STATE4 - movups STATE4, 0x30(OUTP) - sub $64, LEN - add $64, INP - add $64, OUTP - cmp $64, LEN - jge .Lctr_enc_loop4 - cmp $16, LEN - jb .Lctr_enc_ret -.align 4 -.Lctr_enc_loop1: - movaps IV, STATE - call _aesni_inc - movups (INP), IN - call _aesni_enc1 - pxor IN, STATE - movups STATE, (OUTP) - sub $16, LEN - add $16, INP - add $16, OUTP - cmp $16, LEN - jge .Lctr_enc_loop1 -.Lctr_enc_ret: - movups IV, (IVP) -.Lctr_enc_just_ret: - FRAME_END - RET -SYM_FUNC_END(aesni_ctr_enc) - -#endif - .section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 .align 16 .Lgf128mul_x_ble_mask: diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 00b74acd01bd..7d248f2719c2 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -68,11 +68,6 @@ asmlinkage void aesni_xts_enc(const struct crypto_aes_ct= x *ctx, u8 *out, asmlinkage void aesni_xts_dec(const struct crypto_aes_ctx *ctx, u8 *out, const u8 *in, unsigned int len, u8 *iv); =20 -#ifdef CONFIG_X86_64 -asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); -#endif - static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) { return aes_align_addr(raw_ctx); @@ -107,42 +102,6 @@ static int aesni_skcipher_setkey(struct crypto_skciphe= r *tfm, const u8 *key, return aes_set_key_common(aes_ctx(crypto_skcipher_ctx(tfm)), key, len); } =20 -#ifdef CONFIG_X86_64 -/* This is the non-AVX version. */ -static int ctr_crypt_aesni(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct crypto_aes_ctx *ctx =3D aes_ctx(crypto_skcipher_ctx(tfm)); - u8 keystream[AES_BLOCK_SIZE]; - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - - while ((nbytes =3D walk.nbytes) > 0) { - kernel_fpu_begin(); - if (nbytes & AES_BLOCK_MASK) - aesni_ctr_enc(ctx, walk.dst.virt.addr, - walk.src.virt.addr, - nbytes & AES_BLOCK_MASK, walk.iv); - nbytes &=3D ~AES_BLOCK_MASK; - - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - aesni_enc(ctx, keystream, walk.iv); - crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes - nbytes, - walk.src.virt.addr + walk.nbytes - nbytes, - keystream, nbytes); - crypto_inc(walk.iv, AES_BLOCK_SIZE); - nbytes =3D 0; - } - kernel_fpu_end(); - err =3D skcipher_walk_done(&walk, nbytes); - } - return err; -} -#endif - static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen) { @@ -300,25 +259,6 @@ static int xts_decrypt_aesni(struct skcipher_request *= req) } =20 static struct skcipher_alg aesni_skciphers[] =3D { -#ifdef CONFIG_X86_64 - { - .base =3D { - .cra_name =3D "ctr(aes)", - .cra_driver_name =3D "ctr-aes-aesni", - .cra_priority =3D 400, - .cra_blocksize =3D 1, - .cra_ctxsize =3D CRYPTO_AES_CTX_SIZE, - .cra_module =3D THIS_MODULE, - }, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .chunksize =3D AES_BLOCK_SIZE, - .setkey =3D aesni_skcipher_setkey, - .encrypt =3D ctr_crypt_aesni, - .decrypt =3D ctr_crypt_aesni, - }, -#endif { .base =3D { .cra_name =3D "xts(aes)", --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FF2B352037; Mon, 21 Sep 2026 05:16:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967771; cv=none; b=QUJvSArtpMKSyn/odU+bkZp1y2nkKgflZvj8OQ4eDTTuqEFzFgpixNMvbU3kzXY5XAcZzByXlb0bdnYjsAeliY451yH/ULHE/A0KTW2G949FduinH68u+GfeB9YqQQJ98CxHjKk0LIjsA4sQxTl077oq4WJ4fr0FzxxRM/t7C7U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967771; c=relaxed/simple; bh=E0J9wHC77t4UpencBzutAp+L2Z6ppNGrwmHZay+nn4I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CfEmZX92YsAwVEoPVszkbroi6qwrMtRu40Ay1EN5DaIZ4Yrdxp111peKCHPkDvj6U1fyyhDwrNPZVqXGLf3DDmfoU2SO5SVX4pGOX0EJOu7wRsnlla38z+QGn1CvIvmDfWP9Zd1X2wRR6ZDJ1TfztQXf2QWCBdoyXK6mJx35FmA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BINuyM1A; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BINuyM1A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 34CF01F0089C; Mon, 21 Sep 2026 05:16:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967767; bh=KCaFQVHPwizlKMhoDceW3K55+bEOnSGiOzgLrGQHS1o=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=BINuyM1A2ZumFTiaDfJewtdw72fRb2Zst6tJ+kn7GgKU6Ys6qAOLta7IZRn0RaL9N T4OGouHjJx1bDcBISGVTPZHpaYv75siMneHZRWRFIsP8ZNHvjG36v/vf394rK2qDgK GpIGp0J6AFn7eUtDThqqXz2xGn2nvyGx7qZ72pByQ6dRa2F5qKfky/yM46sVzyd5mx O8UHJjJVWxvtjk+0LqIyNxfti7IhWvyJoK3Dtly6XA5YZCjHlwsaBJh0dvdtn/Ezxi DQTdD5IuRgf6xdrqXCkXxPgOvSqWeaoYpbb8NV/IfJi8PiqcFW4Mc3nrfm5TZ+eG9G A0wFGbIz1+8Bw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 11/20] crypto: x86/aes-xts - Remove superseded XTS skcipher Date: Sun, 20 Sep 2026 22:08:57 -0700 Message-ID: <20260921050910.296144-12-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the crypto library's AES-XTS support is optimized with AES-NI (without AVX), the similar code in aesni-intel is redundant. Remove it. This only affects the non-AVX implementation ("xts-aes-aesni"), not aes-xts-avx-x86_64.S which is handled later. Signed-off-by: Eric Biggers --- arch/x86/crypto/aesni-intel_asm.S | 617 ----------------------------- arch/x86/crypto/aesni-intel_glue.c | 69 +--- 2 files changed, 1 insertion(+), 685 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-inte= l_asm.S index c4d54c4a2c23..b12a0f2bf006 100644 --- a/arch/x86/crypto/aesni-intel_asm.S +++ b/arch/x86/crypto/aesni-intel_asm.S @@ -20,30 +20,11 @@ #include #include =20 -#define STATE1 %xmm0 -#define STATE2 %xmm4 -#define STATE3 %xmm5 -#define STATE4 %xmm6 -#define STATE STATE1 -#define IN1 %xmm1 -#define IN2 %xmm7 -#define IN3 %xmm8 -#define IN4 %xmm9 -#define IN IN1 -#define KEY %xmm2 -#define IV %xmm3 - -#define GF128MUL_MASK %xmm7 - #ifdef __x86_64__ #define AREG %rax #define KEYP %rdi #define OUTP %rsi #define UKEYP OUTP -#define INP %rdx -#define LEN %rcx -#define IVP %r8 -#define KLEN %r9d #define T1 %r10 #define TKEYP T1 #define T2 %r11 @@ -52,10 +33,6 @@ #define KEYP %edi #define OUTP AREG #define UKEYP OUTP -#define INP %edx -#define LEN %esi -#define IVP %ebp -#define KLEN %ebx #define T1 %ecx #define TKEYP T1 #endif @@ -241,597 +218,3 @@ SYM_FUNC_START(aesni_set_key) FRAME_END RET SYM_FUNC_END(aesni_set_key) - -/* - * void aesni_enc(const void *ctx, u8 *dst, const u8 *src) - */ -SYM_FUNC_START(aesni_enc) - FRAME_BEGIN -#ifndef __x86_64__ - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+12)(%esp), KEYP # ctx - movl (FRAME_OFFSET+16)(%esp), OUTP # dst - movl (FRAME_OFFSET+20)(%esp), INP # src -#endif - movl 480(KEYP), KLEN # key length - movups (INP), STATE # input - call _aesni_enc1 - movups STATE, (OUTP) # output -#ifndef __x86_64__ - popl KLEN - popl KEYP -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_enc) - -/* - * _aesni_enc1: internal ABI - * input: - * KEYP: key struct pointer - * KLEN: round count - * STATE: initial state (input) - * output: - * STATE: finial state (output) - * changed: - * KEY - * TKEYP (T1) - */ -SYM_FUNC_START_LOCAL(_aesni_enc1) - movaps (KEYP), KEY # key - mov KEYP, TKEYP - pxor KEY, STATE # round 0 - add $0x30, TKEYP - cmp $24, KLEN - jb .Lenc128 - lea 0x20(TKEYP), TKEYP - je .Lenc192 - add $0x20, TKEYP - movaps -0x60(TKEYP), KEY - aesenc KEY, STATE - movaps -0x50(TKEYP), KEY - aesenc KEY, STATE -.align 4 -.Lenc192: - movaps -0x40(TKEYP), KEY - aesenc KEY, STATE - movaps -0x30(TKEYP), KEY - aesenc KEY, STATE -.align 4 -.Lenc128: - movaps -0x20(TKEYP), KEY - aesenc KEY, STATE - movaps -0x10(TKEYP), KEY - aesenc KEY, STATE - movaps (TKEYP), KEY - aesenc KEY, STATE - movaps 0x10(TKEYP), KEY - aesenc KEY, STATE - movaps 0x20(TKEYP), KEY - aesenc KEY, STATE - movaps 0x30(TKEYP), KEY - aesenc KEY, STATE - movaps 0x40(TKEYP), KEY - aesenc KEY, STATE - movaps 0x50(TKEYP), KEY - aesenc KEY, STATE - movaps 0x60(TKEYP), KEY - aesenc KEY, STATE - movaps 0x70(TKEYP), KEY - aesenclast KEY, STATE - RET -SYM_FUNC_END(_aesni_enc1) - -/* - * _aesni_enc4: internal ABI - * input: - * KEYP: key struct pointer - * KLEN: round count - * STATE1: initial state (input) - * STATE2 - * STATE3 - * STATE4 - * output: - * STATE1: finial state (output) - * STATE2 - * STATE3 - * STATE4 - * changed: - * KEY - * TKEYP (T1) - */ -SYM_FUNC_START_LOCAL(_aesni_enc4) - movaps (KEYP), KEY # key - mov KEYP, TKEYP - pxor KEY, STATE1 # round 0 - pxor KEY, STATE2 - pxor KEY, STATE3 - pxor KEY, STATE4 - add $0x30, TKEYP - cmp $24, KLEN - jb .L4enc128 - lea 0x20(TKEYP), TKEYP - je .L4enc192 - add $0x20, TKEYP - movaps -0x60(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps -0x50(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 -#.align 4 -.L4enc192: - movaps -0x40(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps -0x30(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 -#.align 4 -.L4enc128: - movaps -0x20(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps -0x10(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps (TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps 0x10(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps 0x20(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps 0x30(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps 0x40(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps 0x50(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps 0x60(TKEYP), KEY - aesenc KEY, STATE1 - aesenc KEY, STATE2 - aesenc KEY, STATE3 - aesenc KEY, STATE4 - movaps 0x70(TKEYP), KEY - aesenclast KEY, STATE1 # last round - aesenclast KEY, STATE2 - aesenclast KEY, STATE3 - aesenclast KEY, STATE4 - RET -SYM_FUNC_END(_aesni_enc4) - -/* - * _aesni_dec1: internal ABI - * input: - * KEYP: key struct pointer - * KLEN: key length - * STATE: initial state (input) - * output: - * STATE: finial state (output) - * changed: - * KEY - * TKEYP (T1) - */ -SYM_FUNC_START_LOCAL(_aesni_dec1) - movaps (KEYP), KEY # key - mov KEYP, TKEYP - pxor KEY, STATE # round 0 - add $0x30, TKEYP - cmp $24, KLEN - jb .Ldec128 - lea 0x20(TKEYP), TKEYP - je .Ldec192 - add $0x20, TKEYP - movaps -0x60(TKEYP), KEY - aesdec KEY, STATE - movaps -0x50(TKEYP), KEY - aesdec KEY, STATE -.align 4 -.Ldec192: - movaps -0x40(TKEYP), KEY - aesdec KEY, STATE - movaps -0x30(TKEYP), KEY - aesdec KEY, STATE -.align 4 -.Ldec128: - movaps -0x20(TKEYP), KEY - aesdec KEY, STATE - movaps -0x10(TKEYP), KEY - aesdec KEY, STATE - movaps (TKEYP), KEY - aesdec KEY, STATE - movaps 0x10(TKEYP), KEY - aesdec KEY, STATE - movaps 0x20(TKEYP), KEY - aesdec KEY, STATE - movaps 0x30(TKEYP), KEY - aesdec KEY, STATE - movaps 0x40(TKEYP), KEY - aesdec KEY, STATE - movaps 0x50(TKEYP), KEY - aesdec KEY, STATE - movaps 0x60(TKEYP), KEY - aesdec KEY, STATE - movaps 0x70(TKEYP), KEY - aesdeclast KEY, STATE - RET -SYM_FUNC_END(_aesni_dec1) - -/* - * _aesni_dec4: internal ABI - * input: - * KEYP: key struct pointer - * KLEN: key length - * STATE1: initial state (input) - * STATE2 - * STATE3 - * STATE4 - * output: - * STATE1: finial state (output) - * STATE2 - * STATE3 - * STATE4 - * changed: - * KEY - * TKEYP (T1) - */ -SYM_FUNC_START_LOCAL(_aesni_dec4) - movaps (KEYP), KEY # key - mov KEYP, TKEYP - pxor KEY, STATE1 # round 0 - pxor KEY, STATE2 - pxor KEY, STATE3 - pxor KEY, STATE4 - add $0x30, TKEYP - cmp $24, KLEN - jb .L4dec128 - lea 0x20(TKEYP), TKEYP - je .L4dec192 - add $0x20, TKEYP - movaps -0x60(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps -0x50(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 -.align 4 -.L4dec192: - movaps -0x40(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps -0x30(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 -.align 4 -.L4dec128: - movaps -0x20(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps -0x10(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps (TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps 0x10(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps 0x20(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps 0x30(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps 0x40(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps 0x50(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps 0x60(TKEYP), KEY - aesdec KEY, STATE1 - aesdec KEY, STATE2 - aesdec KEY, STATE3 - aesdec KEY, STATE4 - movaps 0x70(TKEYP), KEY - aesdeclast KEY, STATE1 # last round - aesdeclast KEY, STATE2 - aesdeclast KEY, STATE3 - aesdeclast KEY, STATE4 - RET -SYM_FUNC_END(_aesni_dec4) - -.pushsection .rodata -.align 16 -.Lcts_permute_table: - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 - .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 -.popsection - -.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16 -.align 16 -.Lgf128mul_x_ble_mask: - .octa 0x00000000000000010000000000000087 -.previous - -/* - * _aesni_gf128mul_x_ble: Multiply in GF(2^128) for XTS IVs - * input: - * IV: current IV - * GF128MUL_MASK =3D=3D mask with 0x87 and 0x01 - * output: - * IV: next IV - * changed: - * KEY: =3D=3D temporary value - */ -.macro _aesni_gf128mul_x_ble - pshufd $0x13, IV, KEY - paddq IV, IV - psrad $31, KEY - pand GF128MUL_MASK, KEY - pxor KEY, IV -.endm - -.macro _aesni_xts_crypt enc - FRAME_BEGIN -#ifndef __x86_64__ - pushl IVP - pushl LEN - pushl KEYP - pushl KLEN - movl (FRAME_OFFSET+20)(%esp), KEYP # ctx - movl (FRAME_OFFSET+24)(%esp), OUTP # dst - movl (FRAME_OFFSET+28)(%esp), INP # src - movl (FRAME_OFFSET+32)(%esp), LEN # len - movl (FRAME_OFFSET+36)(%esp), IVP # iv - movdqa .Lgf128mul_x_ble_mask, GF128MUL_MASK -#else - movdqa .Lgf128mul_x_ble_mask(%rip), GF128MUL_MASK -#endif - movups (IVP), IV - - mov 480(KEYP), KLEN -.if !\enc - add $240, KEYP - - test $15, LEN - jz .Lxts_loop4\@ - sub $16, LEN -.endif - -.Lxts_loop4\@: - sub $64, LEN - jl .Lxts_1x\@ - - movdqa IV, STATE1 - movdqu 0x00(INP), IN - pxor IN, STATE1 - movdqu IV, 0x00(OUTP) - - _aesni_gf128mul_x_ble - movdqa IV, STATE2 - movdqu 0x10(INP), IN - pxor IN, STATE2 - movdqu IV, 0x10(OUTP) - - _aesni_gf128mul_x_ble - movdqa IV, STATE3 - movdqu 0x20(INP), IN - pxor IN, STATE3 - movdqu IV, 0x20(OUTP) - - _aesni_gf128mul_x_ble - movdqa IV, STATE4 - movdqu 0x30(INP), IN - pxor IN, STATE4 - movdqu IV, 0x30(OUTP) - -.if \enc - call _aesni_enc4 -.else - call _aesni_dec4 -.endif - - movdqu 0x00(OUTP), IN - pxor IN, STATE1 - movdqu STATE1, 0x00(OUTP) - - movdqu 0x10(OUTP), IN - pxor IN, STATE2 - movdqu STATE2, 0x10(OUTP) - - movdqu 0x20(OUTP), IN - pxor IN, STATE3 - movdqu STATE3, 0x20(OUTP) - - movdqu 0x30(OUTP), IN - pxor IN, STATE4 - movdqu STATE4, 0x30(OUTP) - - _aesni_gf128mul_x_ble - - add $64, INP - add $64, OUTP - test LEN, LEN - jnz .Lxts_loop4\@ - -.Lxts_ret_iv\@: - movups IV, (IVP) - -.Lxts_ret\@: -#ifndef __x86_64__ - popl KLEN - popl KEYP - popl LEN - popl IVP -#endif - FRAME_END - RET - -.Lxts_1x\@: - add $64, LEN - jz .Lxts_ret_iv\@ -.if \enc - sub $16, LEN - jl .Lxts_cts4\@ -.endif - -.Lxts_loop1\@: - movdqu (INP), STATE -.if \enc - pxor IV, STATE - call _aesni_enc1 -.else - add $16, INP - sub $16, LEN - jl .Lxts_cts1\@ - pxor IV, STATE - call _aesni_dec1 -.endif - pxor IV, STATE - _aesni_gf128mul_x_ble - - test LEN, LEN - jz .Lxts_out\@ - -.if \enc - add $16, INP - sub $16, LEN - jl .Lxts_cts1\@ -.endif - - movdqu STATE, (OUTP) - add $16, OUTP - jmp .Lxts_loop1\@ - -.Lxts_out\@: - movdqu STATE, (OUTP) - jmp .Lxts_ret_iv\@ - -.if \enc -.Lxts_cts4\@: - movdqa STATE4, STATE - sub $16, OUTP -.Lxts_cts1\@: -.else -.Lxts_cts1\@: - movdqa IV, STATE4 - _aesni_gf128mul_x_ble - - pxor IV, STATE - call _aesni_dec1 - pxor IV, STATE -.endif -#ifndef __x86_64__ - lea .Lcts_permute_table, T1 -#else - lea .Lcts_permute_table(%rip), T1 -#endif - add LEN, INP /* rewind input pointer */ - add $16, LEN /* # bytes in final block */ - movups (INP), IN1 - - mov T1, IVP - add $32, IVP - add LEN, T1 - sub LEN, IVP - add OUTP, LEN - - movups (T1), %xmm4 - movaps STATE, IN2 - pshufb %xmm4, STATE - movups STATE, (LEN) - - movups (IVP), %xmm0 - pshufb %xmm0, IN1 - pblendvb IN2, IN1 - movaps IN1, STATE - -.if \enc - pxor IV, STATE - call _aesni_enc1 - pxor IV, STATE -.else - pxor STATE4, STATE - call _aesni_dec1 - pxor STATE4, STATE -.endif - - movups STATE, (OUTP) - jmp .Lxts_ret\@ -.endm - -/* - * void aesni_xts_enc(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) - */ -SYM_FUNC_START(aesni_xts_enc) - _aesni_xts_crypt 1 -SYM_FUNC_END(aesni_xts_enc) - -/* - * void aesni_xts_dec(const struct crypto_aes_ctx *ctx, u8 *dst, - * const u8 *src, unsigned int len, le128 *iv) - */ -SYM_FUNC_START(aesni_xts_dec) - _aesni_xts_crypt 0 -SYM_FUNC_END(aesni_xts_dec) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 7d248f2719c2..6acb1fa32c6e 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -60,13 +60,6 @@ static inline void *aes_align_addr(void *addr) =20 asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); -asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in); - -asmlinkage void aesni_xts_enc(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); - -asmlinkage void aesni_xts_dec(const struct crypto_aes_ctx *ctx, u8 *out, - const u8 *in, unsigned int len, u8 *iv); =20 static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) { @@ -228,56 +221,6 @@ xts_crypt(struct skcipher_request *req, xts_encrypt_iv= _func encrypt_iv, return xts_crypt_slowpath(req, crypt_func); } =20 -static void aesni_xts_encrypt_iv(const struct crypto_aes_ctx *tweak_key, - u8 iv[AES_BLOCK_SIZE]) -{ - aesni_enc(tweak_key, iv, iv); -} - -static void aesni_xts_encrypt(const struct crypto_aes_ctx *key, - const u8 *src, u8 *dst, int len, - u8 tweak[AES_BLOCK_SIZE]) -{ - aesni_xts_enc(key, dst, src, len, tweak); -} - -static void aesni_xts_decrypt(const struct crypto_aes_ctx *key, - const u8 *src, u8 *dst, int len, - u8 tweak[AES_BLOCK_SIZE]) -{ - aesni_xts_dec(key, dst, src, len, tweak); -} - -static int xts_encrypt_aesni(struct skcipher_request *req) -{ - return xts_crypt(req, aesni_xts_encrypt_iv, aesni_xts_encrypt); -} - -static int xts_decrypt_aesni(struct skcipher_request *req) -{ - return xts_crypt(req, aesni_xts_encrypt_iv, aesni_xts_decrypt); -} - -static struct skcipher_alg aesni_skciphers[] =3D { - { - .base =3D { - .cra_name =3D "xts(aes)", - .cra_driver_name =3D "xts-aes-aesni", - .cra_priority =3D 401, - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D XTS_AES_CTX_SIZE, - .cra_module =3D THIS_MODULE, - }, - .min_keysize =3D 2 * AES_MIN_KEY_SIZE, - .max_keysize =3D 2 * AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .walksize =3D 2 * AES_BLOCK_SIZE, - .setkey =3D xts_setkey_aesni, - .encrypt =3D xts_encrypt_aesni, - .decrypt =3D xts_decrypt_aesni, - } -}; - #ifdef CONFIG_X86_64 asmlinkage void aes_xts_encrypt_iv(const struct crypto_aes_ctx *tweak_key, u8 iv[AES_BLOCK_SIZE]); @@ -1314,15 +1257,10 @@ static int __init aesni_init(void) if (!x86_match_cpu(aesni_cpu_id)) return -ENODEV; =20 - err =3D crypto_register_skciphers(aesni_skciphers, - ARRAY_SIZE(aesni_skciphers)); - if (err) - return err; - err =3D crypto_register_aeads(aes_gcm_algs_aesni, ARRAY_SIZE(aes_gcm_algs_aesni)); if (err) - goto unregister_skciphers; + return err; =20 err =3D register_avx_algs(); if (err) @@ -1334,9 +1272,6 @@ static int __init aesni_init(void) unregister_avx_algs(); crypto_unregister_aeads(aes_gcm_algs_aesni, ARRAY_SIZE(aes_gcm_algs_aesni)); -unregister_skciphers: - crypto_unregister_skciphers(aesni_skciphers, - ARRAY_SIZE(aesni_skciphers)); return err; } =20 @@ -1344,8 +1279,6 @@ static void __exit aesni_exit(void) { crypto_unregister_aeads(aes_gcm_algs_aesni, ARRAY_SIZE(aes_gcm_algs_aesni)); - crypto_unregister_skciphers(aesni_skciphers, - ARRAY_SIZE(aesni_skciphers)); unregister_avx_algs(); } =20 --=20 2.55.0 From nobody Thu Sep 24 20:31:09 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0867F353A98; Mon, 21 Sep 2026 05:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967771; cv=none; b=vDIZi28rC39/nc/X8Jnj+LY88v1U2+/zsn1q5oZTihaKj69gll9Dlvb1Ib5eiZhu5wa+ln7FeKBupLb53Di5faZusG2gF0dTNKGymad5tp8pMwmD4ahEnMjCgiRtuu31ISpexpfBpzPZLBvZVHnaJHfVSSBzMHmI7l5m4lnjJvI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967771; c=relaxed/simple; bh=tSS2mfvgCGt6ZXDQLKxMbb4zRQYLENPQH/XfVX4QetU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e0m46tQUZyjgl8QkKyXvmfjZYxlrWuXbR8htUT/7Bhj7XPYDXTgVuqQTa7MsqQSLs8lGQ6LCL0tRbzcZSVm5V9qXnkmTaLAGlyKD0w+7hfc2CdDphZZYLwHtnvmMLLqvM6SSR8xS4YWpGSjoj3XBxF5DG13xwXJutPKe9kzCSMc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MWBmfkRw; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MWBmfkRw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E84C1F00898; Mon, 21 Sep 2026 05:16:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967767; bh=2aZNbWYnMAIewNbbTA4KNPAmAvZGl6Lz+s1HvM1Jdew=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=MWBmfkRwWKzwWI9M2lA1FbOlUUwG9rxl0lHhiufFTv4MquQxlcFS+ppvCyBCdVr4g CRpD/++FvU3myBNYA2TglTT52PeVUXGA+nWRoEuoAB65TI2vhn/Ag42HeXXunjMxxk 2yzpT4qklm3xFU5OIHuj82M0FzzVpSoaryJ2U9FezVuAoPZLhOvaqa4c9KB7n4QCGJ ZkUdy4feQma0tpwNipqj01pC/bMj3ZARvRmkdwQWuVKtOa0+2/h7IV+zDa76VUBY8d zF6c8juUNmivoiGgbNq18Kaqt0Xd0xR92oiIvp2MTLaNSUY5VzG8T/5VNaA8vpjYE4 tez7MFGk5ka4w== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 12/20] lib/crypto: x86/aes-ctr: Migrate AVX-optimized code into library Date: Sun, 20 Sep 2026 22:08:58 -0700 Message-ID: <20260921050910.296144-13-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Migrate aes-ctr-avx-x86_64.S into lib/crypto/, wiring it up to the CTR and XCTR library functions instead of the crypto_skcipher API. It still remains available through crypto_skcipher via crypto/aes.c. Some slight adjustments to the assembly code were needed: - Take 'struct aes_enckey' instead of 'struct crypto_aes_ctx'. - Upgrade the length argument from 32-bit to 64-bit so that it's compatible with the library's use of size_t (at least assuming no lengths over S64_MAX, which seems quite safe to assume...) - Remove the CFI stubs, as the functions are now called directly. To reduce the diff, the argument order of the assembly functions is kept as-is for now rather than changed to match their callers. Signed-off-by: Eric Biggers --- arch/x86/crypto/Kconfig | 4 +- arch/x86/crypto/Makefile | 3 +- arch/x86/crypto/aesni-intel_glue.c | 154 ------------------ crypto/aes.c | 7 +- lib/crypto/Makefile | 4 + .../crypto/x86}/aes-ctr-avx-x86_64.S | 75 +++++---- lib/crypto/x86/aes.h | 77 ++++++++- 7 files changed, 123 insertions(+), 201 deletions(-) rename {arch/x86/crypto =3D> lib/crypto/x86}/aes-ctr-avx-x86_64.S (92%) diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index 6dbf5e083966..60d9a144d63a 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -3,14 +3,14 @@ menu "Accelerated Cryptographic Algorithms for CPU (x86)" =20 config CRYPTO_AES_NI_INTEL - tristate "Ciphers: AES, modes: CTR, XCTR, XTS, GCM (AES-NI/VAES)" + tristate "Ciphers: AES, modes: XTS, GCM (AES-NI/VAES)" select CRYPTO_AEAD select CRYPTO_LIB_AES select CRYPTO_LIB_GF128MUL select CRYPTO_SKCIPHER help AEAD cipher: AES with GCM - Length-preserving ciphers: AES with CTR, XCTR, XTS + Length-preserving ciphers: AES with XTS =20 Architecture: x86 (32-bit and 64-bit) using: - AES-NI (AES new instructions) diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index e04ff8718d6b..370a9cc7eab2 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -41,8 +41,7 @@ aegis128-aesni-y :=3D aegis128-aesni-asm.o aegis128-aesni= -glue.o =20 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) +=3D aesni-intel.o aesni-intel-y :=3D aesni-intel_asm.o aesni-intel_glue.o -aesni-intel-$(CONFIG_64BIT) +=3D aes-ctr-avx-x86_64.o \ - aes-gcm-aesni-x86_64.o \ +aesni-intel-$(CONFIG_64BIT) +=3D aes-gcm-aesni-x86_64.o \ aes-gcm-vaes-avx2.o \ aes-gcm-vaes-avx512.o \ aes-xts-avx-x86_64.o diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 6acb1fa32c6e..0bda9abae368 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -41,9 +41,7 @@ =20 #define AESNI_ALIGN 16 #define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN))) -#define AES_BLOCK_MASK (~(AES_BLOCK_SIZE - 1)) #define AESNI_ALIGN_EXTRA ((AESNI_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) -#define CRYPTO_AES_CTX_SIZE (sizeof(struct crypto_aes_ctx) + AESNI_ALIGN_E= XTRA) #define XTS_AES_CTX_SIZE (sizeof(struct aesni_xts_ctx) + AESNI_ALIGN_EXTRA) =20 struct aesni_xts_ctx { @@ -61,11 +59,6 @@ static inline void *aes_align_addr(void *addr) asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); =20 -static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) -{ - return aes_align_addr(raw_ctx); -} - static inline struct aesni_xts_ctx *aes_xts_ctx(struct crypto_skcipher *tf= m) { return aes_align_addr(crypto_skcipher_ctx(tfm)); @@ -89,12 +82,6 @@ static int aes_set_key_common(struct crypto_aes_ctx *ctx, return 0; } =20 -static int aesni_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *ke= y, - unsigned int len) -{ - return aes_set_key_common(aes_ctx(crypto_skcipher_ctx(tfm)), key, len); -} - static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen) { @@ -225,100 +212,6 @@ xts_crypt(struct skcipher_request *req, xts_encrypt_i= v_func encrypt_iv, asmlinkage void aes_xts_encrypt_iv(const struct crypto_aes_ctx *tweak_key, u8 iv[AES_BLOCK_SIZE]); =20 -/* __always_inline to avoid indirect call */ -static __always_inline int -ctr_crypt(struct skcipher_request *req, - void (*ctr64_func)(const struct crypto_aes_ctx *key, - const u8 *src, u8 *dst, int len, - const u64 le_ctr[2])) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct crypto_aes_ctx *key =3D aes_ctx(crypto_skcipher_ctx(tfm)); - unsigned int nbytes, p1_nbytes, nblocks; - struct skcipher_walk walk; - u64 le_ctr[2]; - u64 ctr64; - int err; - - ctr64 =3D le_ctr[0] =3D get_unaligned_be64(&req->iv[8]); - le_ctr[1] =3D get_unaligned_be64(&req->iv[0]); - - err =3D skcipher_walk_virt(&walk, req, false); - - while ((nbytes =3D walk.nbytes) !=3D 0) { - if (nbytes < walk.total) { - /* Not the end yet, so keep the length block-aligned. */ - nbytes =3D round_down(nbytes, AES_BLOCK_SIZE); - nblocks =3D nbytes / AES_BLOCK_SIZE; - } else { - /* It's the end, so include any final partial block. */ - nblocks =3D DIV_ROUND_UP(nbytes, AES_BLOCK_SIZE); - } - ctr64 +=3D nblocks; - - kernel_fpu_begin(); - if (likely(ctr64 >=3D nblocks)) { - /* The low 64 bits of the counter won't overflow. */ - (*ctr64_func)(key, walk.src.virt.addr, - walk.dst.virt.addr, nbytes, le_ctr); - } else { - /* - * The low 64 bits of the counter will overflow. The - * assembly doesn't handle this case, so split the - * operation into two at the point where the overflow - * will occur. After the first part, add the carry bit. - */ - p1_nbytes =3D min(nbytes, (nblocks - ctr64) * AES_BLOCK_SIZE); - (*ctr64_func)(key, walk.src.virt.addr, - walk.dst.virt.addr, p1_nbytes, le_ctr); - le_ctr[0] =3D 0; - le_ctr[1]++; - (*ctr64_func)(key, walk.src.virt.addr + p1_nbytes, - walk.dst.virt.addr + p1_nbytes, - nbytes - p1_nbytes, le_ctr); - } - kernel_fpu_end(); - le_ctr[0] =3D ctr64; - - err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); - } - - put_unaligned_be64(ctr64, &req->iv[8]); - put_unaligned_be64(le_ctr[1], &req->iv[0]); - - return err; -} - -/* __always_inline to avoid indirect call */ -static __always_inline int -xctr_crypt(struct skcipher_request *req, - void (*xctr_func)(const struct crypto_aes_ctx *key, - const u8 *src, u8 *dst, int len, - const u8 iv[AES_BLOCK_SIZE], u64 ctr)) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct crypto_aes_ctx *key =3D aes_ctx(crypto_skcipher_ctx(tfm)); - struct skcipher_walk walk; - unsigned int nbytes; - u64 ctr =3D 1; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - while ((nbytes =3D walk.nbytes) !=3D 0) { - if (nbytes < walk.total) - nbytes =3D round_down(nbytes, AES_BLOCK_SIZE); - - kernel_fpu_begin(); - (*xctr_func)(key, walk.src.virt.addr, walk.dst.virt.addr, - nbytes, req->iv, ctr); - kernel_fpu_end(); - - ctr +=3D DIV_ROUND_UP(nbytes, AES_BLOCK_SIZE); - err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); - } - return err; -} - #define DEFINE_AVX_SKCIPHER_ALGS(suffix, driver_name_suffix, priority) = \ \ asmlinkage void \ @@ -338,25 +231,6 @@ static int xts_decrypt_##suffix(struct skcipher_reques= t *req) \ return xts_crypt(req, aes_xts_encrypt_iv, aes_xts_decrypt_##suffix); \ } \ \ -asmlinkage void \ -aes_ctr64_crypt_##suffix(const struct crypto_aes_ctx *key, \ - const u8 *src, u8 *dst, int len, const u64 le_ctr[2]);\ - \ -static int ctr_crypt_##suffix(struct skcipher_request *req) \ -{ \ - return ctr_crypt(req, aes_ctr64_crypt_##suffix); \ -} \ - \ -asmlinkage void \ -aes_xctr_crypt_##suffix(const struct crypto_aes_ctx *key, \ - const u8 *src, u8 *dst, int len, \ - const u8 iv[AES_BLOCK_SIZE], u64 ctr); \ - \ -static int xctr_crypt_##suffix(struct skcipher_request *req) \ -{ \ - return xctr_crypt(req, aes_xctr_crypt_##suffix); \ -} \ - \ static struct skcipher_alg skcipher_algs_##suffix[] =3D {{ \ .base.cra_name =3D "xts(aes)", \ .base.cra_driver_name =3D "xts-aes-" driver_name_suffix, \ @@ -371,34 +245,6 @@ static struct skcipher_alg skcipher_algs_##suffix[] = =3D {{ \ .setkey =3D xts_setkey_aesni, \ .encrypt =3D xts_encrypt_##suffix, \ .decrypt =3D xts_decrypt_##suffix, \ -}, { \ - .base.cra_name =3D "ctr(aes)", \ - .base.cra_driver_name =3D "ctr-aes-" driver_name_suffix, \ - .base.cra_priority =3D priority, \ - .base.cra_blocksize =3D 1, \ - .base.cra_ctxsize =3D CRYPTO_AES_CTX_SIZE, \ - .base.cra_module =3D THIS_MODULE, \ - .min_keysize =3D AES_MIN_KEY_SIZE, \ - .max_keysize =3D AES_MAX_KEY_SIZE, \ - .ivsize =3D AES_BLOCK_SIZE, \ - .chunksize =3D AES_BLOCK_SIZE, \ - .setkey =3D aesni_skcipher_setkey, \ - .encrypt =3D ctr_crypt_##suffix, \ - .decrypt =3D ctr_crypt_##suffix, \ -}, { \ - .base.cra_name =3D "xctr(aes)", \ - .base.cra_driver_name =3D "xctr-aes-" driver_name_suffix, \ - .base.cra_priority =3D priority, \ - .base.cra_blocksize =3D 1, \ - .base.cra_ctxsize =3D CRYPTO_AES_CTX_SIZE, \ - .base.cra_module =3D THIS_MODULE, \ - .min_keysize =3D AES_MIN_KEY_SIZE, \ - .max_keysize =3D AES_MAX_KEY_SIZE, \ - .ivsize =3D AES_BLOCK_SIZE, \ - .chunksize =3D AES_BLOCK_SIZE, \ - .setkey =3D aesni_skcipher_setkey, \ - .encrypt =3D xctr_crypt_##suffix, \ - .decrypt =3D xctr_crypt_##suffix, \ }} =20 DEFINE_AVX_SKCIPHER_ALGS(aesni_avx, "aesni-avx", 500); diff --git a/crypto/aes.c b/crypto/aes.c index cc2cd6b08eee..ac484a28b30e 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -667,7 +667,7 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "ctr(aes)", .base.cra_driver_name =3D "ctr-aes-lib", - .base.cra_priority =3D 110, + .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, .base.cra_blocksize =3D 1, .base.cra_ctxsize =3D sizeof(struct aes_enckey), .base.cra_module =3D THIS_MODULE, @@ -684,7 +684,7 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "xctr(aes)", .base.cra_driver_name =3D "xctr-aes-lib", - .base.cra_priority =3D 110, + .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, .base.cra_blocksize =3D 1, .base.cra_ctxsize =3D sizeof(struct aes_enckey), .base.cra_module =3D THIS_MODULE, @@ -1044,8 +1044,7 @@ static struct aead_alg aead_algs[] =3D { IS_ENABLED(CONFIG_POWERPC) || \ IS_ENABLED(CONFIG_RISCV) || \ IS_ENABLED(CONFIG_S390) || \ - IS_ENABLED(CONFIG_SPARC) || \ - IS_ENABLED(CONFIG_X86)) + IS_ENABLED(CONFIG_SPARC)) { .base.cra_name =3D "ccm(aes)", .base.cra_driver_name =3D "ccm-aes-lib", diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile index ca068df1f71f..5d5484fc78ea 100644 --- a/lib/crypto/Makefile +++ b/lib/crypto/Makefile @@ -52,7 +52,11 @@ endif # CONFIG_PPC =20 libaes-$(CONFIG_RISCV) +=3D riscv/aes-riscv64-zvkned.o libaes-$(CONFIG_SPARC) +=3D sparc/aes_asm.o + libaes-$(CONFIG_X86) +=3D x86/aes-aesni.o +ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),) +libaes-$(CONFIG_X86_64) +=3D x86/aes-ctr-avx-x86_64.o +endif endif # CONFIG_CRYPTO_LIB_AES_ARCH =20 # clean-files must be defined unconditionally diff --git a/arch/x86/crypto/aes-ctr-avx-x86_64.S b/lib/crypto/x86/aes-ctr-= avx-x86_64.S similarity index 92% rename from arch/x86/crypto/aes-ctr-avx-x86_64.S rename to lib/crypto/x86/aes-ctr-avx-x86_64.S index 2745918f68ee..654a7cd88027 100644 --- a/arch/x86/crypto/aes-ctr-avx-x86_64.S +++ b/lib/crypto/x86/aes-ctr-avx-x86_64.S @@ -53,7 +53,10 @@ // See the function definitions at the bottom of the file for more informa= tion. =20 #include -#include + +// Offsets in struct aes_enckey +#define OFFSETOF_KEYLEN 0 +#define OFFSETOF_RNDKEYS 16 =20 .section .rodata .p2align 4 @@ -279,16 +282,16 @@ =20 // Function arguments .set KEY, %rdi // Initially points to the start of the - // crypto_aes_ctx, then is advanced to + // aes_enckey, then is advanced to // point to the index 1 round key .set KEY32, %edi // Available as temp register after all // keystream blocks have been generated .set SRC, %rsi // Pointer to next source data .set DST, %rdx // Pointer to next destination data - .set LEN, %ecx // Remaining length in bytes. + .set LEN, %rcx // Remaining length in bytes. // Note: _load_partial_block relies on - // this being in %ecx. - .set LEN64, %rcx // Zero-extend LEN before using! + // this being in %rcx. + .set LEN32, %ecx .set LEN8, %cl .if \is_xctr .set XCTR_IV_PTR, %r8 // const u8 iv[AES_BLOCK_SIZE]; @@ -355,17 +358,17 @@ vpsllq $1, LE_CTR_INC1, LE_CTR_INC2 =20 // Load the AES key length: 16 (AES-128), 24 (AES-192), or 32 (AES-256). - movl 480(KEY), %eax + movl OFFSETOF_KEYLEN(KEY), %eax =20 // Compute the pointer to the last round key. - lea 6*16(KEY, %rax, 4), RNDKEYLAST_PTR + lea OFFSETOF_RNDKEYS+6*16(KEY, %rax, 4), RNDKEYLAST_PTR =20 // Load the zero-th and last round keys. - _vbroadcast128 (KEY), RNDKEY0 + _vbroadcast128 OFFSETOF_RNDKEYS(KEY), RNDKEY0 _vbroadcast128 (RNDKEYLAST_PTR), RNDKEYLAST =20 // Make KEY point to the first round key. - add $16, KEY + add $OFFSETOF_RNDKEYS+16, KEY =20 // This is the main loop, which encrypts 8 vectors of data at a time. add $-8*VL, LEN @@ -390,7 +393,7 @@ =20 _prepare_2_ctr_vecs \is_xctr, 0, 1 _prepare_2_ctr_vecs \is_xctr, 2, 3 - cmp $4*VL, LEN + cmp $4*VL, LEN32 jle .Lenc_tail_atmost4vecs\@ =20 // 4*VL < LEN < 8*VL. Generate 8 vectors of keystream blocks. Use the @@ -405,23 +408,23 @@ vaesenclast RNDKEYLAST, AESDATA7, AESDATA3 sub $-4*VL, SRC sub $-4*VL, DST - add $-4*VL, LEN - cmp $1*VL-1, LEN + add $-4*VL, LEN32 + cmp $1*VL-1, LEN32 jle .Lxor_tail_partial_vec_0\@ _xor_data 0 - cmp $2*VL-1, LEN + cmp $2*VL-1, LEN32 jle .Lxor_tail_partial_vec_1\@ _xor_data 1 - cmp $3*VL-1, LEN + cmp $3*VL-1, LEN32 jle .Lxor_tail_partial_vec_2\@ _xor_data 2 - cmp $4*VL-1, LEN + cmp $4*VL-1, LEN32 jle .Lxor_tail_partial_vec_3\@ _xor_data 3 jmp .Ldone\@ =20 .Lenc_tail_atmost4vecs\@: - cmp $2*VL, LEN + cmp $2*VL, LEN32 jle .Lenc_tail_atmost2vecs\@ =20 // 2*VL < LEN <=3D 4*VL. Generate 4 vectors of keystream blocks. Use the @@ -432,7 +435,7 @@ vaesenclast RNDKEYLAST, AESDATA3, AESDATA1 sub $-2*VL, SRC sub $-2*VL, DST - add $-2*VL, LEN + add $-2*VL, LEN32 jmp .Lxor_tail_upto2vecs\@ =20 .Lenc_tail_atmost2vecs\@: @@ -443,16 +446,16 @@ vaesenclast RNDKEYLAST, AESDATA1, AESDATA1 =20 .Lxor_tail_upto2vecs\@: - cmp $1*VL-1, LEN + cmp $1*VL-1, LEN32 jle .Lxor_tail_partial_vec_0\@ _xor_data 0 - cmp $2*VL-1, LEN + cmp $2*VL-1, LEN32 jle .Lxor_tail_partial_vec_1\@ _xor_data 1 jmp .Ldone\@ =20 .Lxor_tail_partial_vec_1\@: - add $-1*VL, LEN + add $-1*VL, LEN32 jz .Ldone\@ sub $-1*VL, SRC sub $-1*VL, DST @@ -460,7 +463,7 @@ jmp .Lxor_tail_partial_vec_0\@ =20 .Lxor_tail_partial_vec_2\@: - add $-2*VL, LEN + add $-2*VL, LEN32 jz .Ldone\@ sub $-2*VL, SRC sub $-2*VL, DST @@ -468,7 +471,7 @@ jmp .Lxor_tail_partial_vec_0\@ =20 .Lxor_tail_partial_vec_3\@: - add $-3*VL, LEN + add $-3*VL, LEN32 jz .Ldone\@ sub $-3*VL, SRC sub $-3*VL, DST @@ -479,25 +482,25 @@ // loads/stores are available; otherwise it's a bit harder... .if USE_AVX512 mov $-1, %rax - bzhi LEN64, %rax, %rax + bzhi LEN, %rax, %rax kmovq %rax, %k1 vmovdqu8 (SRC), AESDATA1{%k1}{z} vpxord AESDATA1, AESDATA0, AESDATA0 vmovdqu8 AESDATA0, (DST){%k1} .else .if VL =3D=3D 32 - cmp $16, LEN + cmp $16, LEN32 jl 1f vpxor (SRC), AESDATA0_XMM, AESDATA1_XMM vmovdqu AESDATA1_XMM, (DST) add $16, SRC add $16, DST - sub $16, LEN + sub $16, LEN32 jz .Ldone\@ vextracti128 $1, AESDATA0, AESDATA0_XMM 1: .endif - mov LEN, %r10d + mov LEN32, %r10d _load_partial_block SRC, AESDATA1_XMM, KEY, KEY32 vpxor AESDATA1_XMM, AESDATA0_XMM, AESDATA0_XMM mov %r10d, %ecx @@ -515,12 +518,12 @@ // They have the following prototypes: // // -// void aes_ctr64_crypt_##suffix(const struct crypto_aes_ctx *key, -// const u8 *src, u8 *dst, int len, +// void aes_ctr64_crypt_##suffix(const struct aes_enckey *key, +// const u8 *src, u8 *dst, s64 len, // const u64 le_ctr[2]); // -// void aes_xctr_crypt_##suffix(const struct crypto_aes_ctx *key, -// const u8 *src, u8 *dst, int len, +// void aes_xctr_crypt_##suffix(const struct aes_enckey *key, +// const u8 *src, u8 *dst, s64 len, // const u8 iv[AES_BLOCK_SIZE], u64 ctr); // // Both functions generate |len| bytes of keystream, XOR it with the data = from @@ -545,27 +548,27 @@ =20 .set VL, 16 .set USE_AVX512, 0 -SYM_TYPED_FUNC_START(aes_ctr64_crypt_aesni_avx) +SYM_FUNC_START(aes_ctr64_crypt_aesni_avx) _aes_ctr_crypt 0 SYM_FUNC_END(aes_ctr64_crypt_aesni_avx) -SYM_TYPED_FUNC_START(aes_xctr_crypt_aesni_avx) +SYM_FUNC_START(aes_xctr_crypt_aesni_avx) _aes_ctr_crypt 1 SYM_FUNC_END(aes_xctr_crypt_aesni_avx) =20 .set VL, 32 .set USE_AVX512, 0 -SYM_TYPED_FUNC_START(aes_ctr64_crypt_vaes_avx2) +SYM_FUNC_START(aes_ctr64_crypt_vaes_avx2) _aes_ctr_crypt 0 SYM_FUNC_END(aes_ctr64_crypt_vaes_avx2) -SYM_TYPED_FUNC_START(aes_xctr_crypt_vaes_avx2) +SYM_FUNC_START(aes_xctr_crypt_vaes_avx2) _aes_ctr_crypt 1 SYM_FUNC_END(aes_xctr_crypt_vaes_avx2) =20 .set VL, 64 .set USE_AVX512, 1 -SYM_TYPED_FUNC_START(aes_ctr64_crypt_vaes_avx512) +SYM_FUNC_START(aes_ctr64_crypt_vaes_avx512) _aes_ctr_crypt 0 SYM_FUNC_END(aes_ctr64_crypt_vaes_avx512) -SYM_TYPED_FUNC_START(aes_xctr_crypt_vaes_avx512) +SYM_FUNC_START(aes_xctr_crypt_vaes_avx512) _aes_ctr_crypt 1 SYM_FUNC_END(aes_xctr_crypt_vaes_avx512) diff --git a/lib/crypto/x86/aes.h b/lib/crypto/x86/aes.h index def9799302c1..5b4205870b9f 100644 --- a/lib/crypto/x86/aes.h +++ b/lib/crypto/x86/aes.h @@ -8,8 +8,12 @@ #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aesni); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aesni_avx); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vaes_avx2); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vaes_avx512); =20 /* The assembly code assumes the following offsets. */ +static_assert(offsetof(struct aes_enckey, len) =3D=3D 0); static_assert(offsetof(struct aes_enckey, nrounds) =3D=3D 4); static_assert(offsetof(struct aes_enckey, k.rndkeys) =3D=3D 16); static_assert(offsetof(struct aes_key, inv_k.inv_rndkeys) =3D=3D 256); @@ -206,11 +210,33 @@ static bool aes_cbc_cts_decrypt_arch(u8 *dst, const u= 8 *src, size_t len, #if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CTR) && IS_ENABLED(CONFIG_X86_64) void aes_ctr64_crypt_aesni(u8 *dst, const u8 *src, s64 len, const u64 le_c= tr[2], const struct aes_enckey *key); +void aes_ctr64_crypt_aesni_avx(const struct aes_enckey *key, const u8 *src, + u8 *dst, s64 len, const u64 le_ctr[2]); +void aes_ctr64_crypt_vaes_avx2(const struct aes_enckey *key, const u8 *src, + u8 *dst, s64 len, const u64 le_ctr[2]); +void aes_ctr64_crypt_vaes_avx512(const struct aes_enckey *key, const u8 *s= rc, + u8 *dst, s64 len, const u64 le_ctr[2]); +void aes_xctr_crypt_aesni_avx(const struct aes_enckey *key, const u8 *src, + u8 *dst, s64 len, const u8 iv[AES_BLOCK_SIZE], + u64 ctr); +void aes_xctr_crypt_vaes_avx2(const struct aes_enckey *key, const u8 *src, + u8 *dst, s64 len, const u8 iv[AES_BLOCK_SIZE], + u64 ctr); +void aes_xctr_crypt_vaes_avx512(const struct aes_enckey *key, const u8 *sr= c, + u8 *dst, s64 len, const u8 iv[AES_BLOCK_SIZE], + u64 ctr); =20 static void aes_ctr64_x86(u8 *dst, const u8 *src, size_t len, const u64 le_ctr[2], const struct aes_enckey *key) { - aes_ctr64_crypt_aesni(dst, src, len, le_ctr, key); + if (static_branch_likely(&have_vaes_avx512)) + aes_ctr64_crypt_vaes_avx512(key, src, dst, len, le_ctr); + else if (static_branch_likely(&have_vaes_avx2)) + aes_ctr64_crypt_vaes_avx2(key, src, dst, len, le_ctr); + else if (static_branch_likely(&have_aesni_avx)) + aes_ctr64_crypt_aesni_avx(key, src, dst, len, le_ctr); + else + aes_ctr64_crypt_aesni(dst, src, len, le_ctr, key); } =20 #define aes_ctr_arch aes_ctr_arch @@ -255,6 +281,25 @@ static bool aes_ctr_arch(u8 *dst, const u8 *src, size_= t len, put_unaligned_be64(le_ctr[1], &ctr[0]); return true; } + +#define aes_xctr_arch aes_xctr_arch +static bool aes_xctr_arch(u8 *dst, const u8 *src, size_t len, u64 ctr, + const u8 iv[AES_BLOCK_SIZE], + const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_aesni_avx) || + unlikely(!irq_fpu_usable())) + return false; + kernel_fpu_begin(); + if (static_branch_likely(&have_vaes_avx512)) + aes_xctr_crypt_vaes_avx512(key, src, dst, len, iv, ctr); + else if (static_branch_likely(&have_vaes_avx2)) + aes_xctr_crypt_vaes_avx2(key, src, dst, len, iv, ctr); + else + aes_xctr_crypt_aesni_avx(key, src, dst, len, iv, ctr); + kernel_fpu_end(); + return true; +} #endif /* CONFIG_CRYPTO_LIB_AES_CTR && CONFIG_X86_64 */ =20 #if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_XTS) @@ -304,6 +349,32 @@ static bool aes_xts_decrypt_arch(u8 *dst, const u8 *sr= c, size_t len, #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { - if (boot_cpu_has(X86_FEATURE_AES)) - static_branch_enable(&have_aesni); + /* Everything below requires AES-NI. */ + if (!boot_cpu_has(X86_FEATURE_AES)) + return; + static_branch_enable(&have_aesni); + + /* Everything below requires AVX and is also 64-bit only. */ + if (!boot_cpu_has(X86_FEATURE_AVX) || !IS_ENABLED(CONFIG_X86_64)) + return; + static_branch_enable(&have_aesni_avx); + + /* + * Everything below requires VAES, and also sometimes AVX2, VPCLMULQDQ, + * and PCLMULQDQ. Use a single static key for all of them, since in + * practice every CPU with VAES also has the others. + */ + if (!boot_cpu_has(X86_FEATURE_AVX2) || + !boot_cpu_has(X86_FEATURE_VAES) || + !boot_cpu_has(X86_FEATURE_VPCLMULQDQ) || + !boot_cpu_has(X86_FEATURE_PCLMULQDQ)) + return; + static_branch_enable(&have_vaes_avx2); + + if (!boot_cpu_has(X86_FEATURE_AVX512BW) || + !boot_cpu_has(X86_FEATURE_AVX512VL) || + !boot_cpu_has(X86_FEATURE_BMI2) || + boot_cpu_has(X86_FEATURE_PREFER_YMM)) + return; + static_branch_enable(&have_vaes_avx512); } --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2443353A65; Mon, 21 Sep 2026 05:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967773; cv=none; b=CH6dGk4QH0/20NOTuPkwD0M1OTReKBMhlVQNBnRY/B+XQ2NuWvTbj1Xax7+/NRM/wcduNRLSqNH8Tr6CdnQWtaOt3b8bCnUtJt3se6D5RyuzWu700DO6WMungV2jRrG+W3vYstOlJDD22bRG8fuziuDstM6//siinv4sn/O/l5g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967773; c=relaxed/simple; bh=LsE2vmlgbkaxmbjgsiAWLEGy6zXp+UESQ+dwDznW6AY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VONw6TlVvLNzXyMvKZA+1CVXig+t/cEjDjFS0ND+xRcBexOCijI1J9aTB0N6F6sg1oT+lxD2nmCnwE84Xv/buw9kRrVHMiXgGJf+KYqVXwjuhjJE8W2hqSbTA/48jeqRkMmQtIrfEOtChkC5b45LTMzLUvhpm/Kuuzv+3faEVck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=W0YAIjI+; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="W0YAIjI+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 04D291F00893; Mon, 21 Sep 2026 05:16:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967768; bh=nQAxI9vIMtDk2I+S4IW8A5jdoMKPiNLsvoe4yfa0L3k=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=W0YAIjI+t30bZvQYizfK0VUjZFStlmvnnZnjGaKMQJIiGreK/9pdQiphPvNwv3lj2 OC9CAE1ND1Ky6O4gMIuUPLkJQ8NjdMBjUIjKchQ6deRo36eRWxWRh8Bd5Ve4FSyZYB porB0DO+Y0rhwwxB3mtURvdEVvarjR3XqzAKvqjjKjiCKM57/NEvkBzEVU2jcqIbOb Y5FZPi3OoNc8UifPxYOUez2zqrl98B4Vajs37POxYiwlU++7/o5KlHlhzkQLxyHP0O RfPfs1/iUayaChbb7kKMdiKny9ZJ3jvE9PaD41YeHMP6dchCpDZUboymu6m9AAS8Ci Wm31OtX4d7qQQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 13/20] lib/crypto: x86/aes-xts: Migrate AVX-optimized code into library Date: Sun, 20 Sep 2026 22:08:59 -0700 Message-ID: <20260921050910.296144-14-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Migrate aes-xts-avx-x86_64.S into lib/crypto/, wiring it up to the XTS library functions instead of the crypto_skcipher API. It still remains available through crypto_skcipher via crypto/aes.c. Some adjustments to the assembly code were needed: - Take 'struct aes_key' instead of 'struct crypto_aes_ctx'. - Remove the ciphertext stealing support from the assembly code, as the library implements it in a generic way instead. (This does slightly reduce performance when the length isn't a multiple of 16 bytes; however, that case seems to never be reached in practice in the kernel. So it makes sense to not extensively optimize for it yet.) - Change 'int len' to 'long nblocks' for compatibility with the library's use of size_t lengths. - Remove the CFI stubs, as the functions are now called directly. To reduce the diff, the argument order of the assembly functions is kept as-is for now rather than changed to match their callers. This makes the remaining code in aesni-intel_asm.S (which just handled key expansion) unused, so remove that too. Signed-off-by: Eric Biggers --- arch/x86/crypto/Kconfig | 4 +- arch/x86/crypto/Makefile | 5 +- arch/x86/crypto/aesni-intel_asm.S | 220 ---------------- arch/x86/crypto/aesni-intel_glue.c | 236 +----------------- crypto/aes.c | 2 +- lib/crypto/Makefile | 3 + .../crypto/x86}/aes-xts-avx-x86_64.S | 172 ++++--------- lib/crypto/x86/aes.h | 62 ++++- 8 files changed, 107 insertions(+), 597 deletions(-) delete mode 100644 arch/x86/crypto/aesni-intel_asm.S rename {arch/x86/crypto =3D> lib/crypto/x86}/aes-xts-avx-x86_64.S (81%) diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index 60d9a144d63a..d68b31fad508 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -3,14 +3,12 @@ menu "Accelerated Cryptographic Algorithms for CPU (x86)" =20 config CRYPTO_AES_NI_INTEL - tristate "Ciphers: AES, modes: XTS, GCM (AES-NI/VAES)" + tristate "Ciphers: AES, modes: GCM (AES-NI/VAES)" select CRYPTO_AEAD select CRYPTO_LIB_AES select CRYPTO_LIB_GF128MUL - select CRYPTO_SKCIPHER help AEAD cipher: AES with GCM - Length-preserving ciphers: AES with XTS =20 Architecture: x86 (32-bit and 64-bit) using: - AES-NI (AES new instructions) diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index 370a9cc7eab2..e05d6e2257d4 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -40,11 +40,10 @@ obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) +=3D aegis128-= aesni.o aegis128-aesni-y :=3D aegis128-aesni-asm.o aegis128-aesni-glue.o =20 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) +=3D aesni-intel.o -aesni-intel-y :=3D aesni-intel_asm.o aesni-intel_glue.o +aesni-intel-y :=3D aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) +=3D aes-gcm-aesni-x86_64.o \ aes-gcm-vaes-avx2.o \ - aes-gcm-vaes-avx512.o \ - aes-xts-avx-x86_64.o + aes-gcm-vaes-avx512.o =20 obj-$(CONFIG_CRYPTO_SM4_AESNI_AVX_X86_64) +=3D sm4-aesni-avx-x86_64.o sm4-aesni-avx-x86_64-y :=3D sm4-aesni-avx-asm_64.o sm4_aesni_avx_glue.o diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-inte= l_asm.S deleted file mode 100644 index b12a0f2bf006..000000000000 --- a/arch/x86/crypto/aesni-intel_asm.S +++ /dev/null @@ -1,220 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Implement AES algorithm in Intel AES-NI instructions. - * - * The white paper of AES-NI instructions can be downloaded from: - * http://softwarecommunity.intel.com/isn/downloads/intelavx/AES-Instruc= tions-Set_WP.pdf - * - * Copyright (C) 2008, Intel Corp. - * Author: Huang Ying - * Vinodh Gopal - * Kahraman Akdemir - * - * Copyright (c) 2010, Intel Corporation. - * - * Ported x86_64 version to x86: - * Author: Mathias Krause - */ - -#include -#include -#include - -#ifdef __x86_64__ -#define AREG %rax -#define KEYP %rdi -#define OUTP %rsi -#define UKEYP OUTP -#define T1 %r10 -#define TKEYP T1 -#define T2 %r11 -#else -#define AREG %eax -#define KEYP %edi -#define OUTP AREG -#define UKEYP OUTP -#define T1 %ecx -#define TKEYP T1 -#endif - -SYM_FUNC_START_LOCAL(_key_expansion_256a) - pshufd $0b11111111, %xmm1, %xmm1 - shufps $0b00010000, %xmm0, %xmm4 - pxor %xmm4, %xmm0 - shufps $0b10001100, %xmm0, %xmm4 - pxor %xmm4, %xmm0 - pxor %xmm1, %xmm0 - movaps %xmm0, (TKEYP) - add $0x10, TKEYP - RET -SYM_FUNC_END(_key_expansion_256a) -SYM_FUNC_ALIAS_LOCAL(_key_expansion_128, _key_expansion_256a) - -SYM_FUNC_START_LOCAL(_key_expansion_192a) - pshufd $0b01010101, %xmm1, %xmm1 - shufps $0b00010000, %xmm0, %xmm4 - pxor %xmm4, %xmm0 - shufps $0b10001100, %xmm0, %xmm4 - pxor %xmm4, %xmm0 - pxor %xmm1, %xmm0 - - movaps %xmm2, %xmm5 - movaps %xmm2, %xmm6 - pslldq $4, %xmm5 - pshufd $0b11111111, %xmm0, %xmm3 - pxor %xmm3, %xmm2 - pxor %xmm5, %xmm2 - - movaps %xmm0, %xmm1 - shufps $0b01000100, %xmm0, %xmm6 - movaps %xmm6, (TKEYP) - shufps $0b01001110, %xmm2, %xmm1 - movaps %xmm1, 0x10(TKEYP) - add $0x20, TKEYP - RET -SYM_FUNC_END(_key_expansion_192a) - -SYM_FUNC_START_LOCAL(_key_expansion_192b) - pshufd $0b01010101, %xmm1, %xmm1 - shufps $0b00010000, %xmm0, %xmm4 - pxor %xmm4, %xmm0 - shufps $0b10001100, %xmm0, %xmm4 - pxor %xmm4, %xmm0 - pxor %xmm1, %xmm0 - - movaps %xmm2, %xmm5 - pslldq $4, %xmm5 - pshufd $0b11111111, %xmm0, %xmm3 - pxor %xmm3, %xmm2 - pxor %xmm5, %xmm2 - - movaps %xmm0, (TKEYP) - add $0x10, TKEYP - RET -SYM_FUNC_END(_key_expansion_192b) - -SYM_FUNC_START_LOCAL(_key_expansion_256b) - pshufd $0b10101010, %xmm1, %xmm1 - shufps $0b00010000, %xmm2, %xmm4 - pxor %xmm4, %xmm2 - shufps $0b10001100, %xmm2, %xmm4 - pxor %xmm4, %xmm2 - pxor %xmm1, %xmm2 - movaps %xmm2, (TKEYP) - add $0x10, TKEYP - RET -SYM_FUNC_END(_key_expansion_256b) - -/* - * void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - * unsigned int key_len) - */ -SYM_FUNC_START(aesni_set_key) - FRAME_BEGIN -#ifndef __x86_64__ - pushl KEYP - movl (FRAME_OFFSET+8)(%esp), KEYP # ctx - movl (FRAME_OFFSET+12)(%esp), UKEYP # in_key - movl (FRAME_OFFSET+16)(%esp), %edx # key_len -#endif - movups (UKEYP), %xmm0 # user key (first 16 bytes) - movaps %xmm0, (KEYP) - lea 0x10(KEYP), TKEYP # key addr - movl %edx, 480(KEYP) - pxor %xmm4, %xmm4 # xmm4 is assumed 0 in _key_expansion_x - cmp $24, %dl - jb .Lenc_key128 - je .Lenc_key192 - movups 0x10(UKEYP), %xmm2 # other user key - movaps %xmm2, (TKEYP) - add $0x10, TKEYP - aeskeygenassist $0x1, %xmm2, %xmm1 # round 1 - call _key_expansion_256a - aeskeygenassist $0x1, %xmm0, %xmm1 - call _key_expansion_256b - aeskeygenassist $0x2, %xmm2, %xmm1 # round 2 - call _key_expansion_256a - aeskeygenassist $0x2, %xmm0, %xmm1 - call _key_expansion_256b - aeskeygenassist $0x4, %xmm2, %xmm1 # round 3 - call _key_expansion_256a - aeskeygenassist $0x4, %xmm0, %xmm1 - call _key_expansion_256b - aeskeygenassist $0x8, %xmm2, %xmm1 # round 4 - call _key_expansion_256a - aeskeygenassist $0x8, %xmm0, %xmm1 - call _key_expansion_256b - aeskeygenassist $0x10, %xmm2, %xmm1 # round 5 - call _key_expansion_256a - aeskeygenassist $0x10, %xmm0, %xmm1 - call _key_expansion_256b - aeskeygenassist $0x20, %xmm2, %xmm1 # round 6 - call _key_expansion_256a - aeskeygenassist $0x20, %xmm0, %xmm1 - call _key_expansion_256b - aeskeygenassist $0x40, %xmm2, %xmm1 # round 7 - call _key_expansion_256a - jmp .Ldec_key -.Lenc_key192: - movq 0x10(UKEYP), %xmm2 # other user key - aeskeygenassist $0x1, %xmm2, %xmm1 # round 1 - call _key_expansion_192a - aeskeygenassist $0x2, %xmm2, %xmm1 # round 2 - call _key_expansion_192b - aeskeygenassist $0x4, %xmm2, %xmm1 # round 3 - call _key_expansion_192a - aeskeygenassist $0x8, %xmm2, %xmm1 # round 4 - call _key_expansion_192b - aeskeygenassist $0x10, %xmm2, %xmm1 # round 5 - call _key_expansion_192a - aeskeygenassist $0x20, %xmm2, %xmm1 # round 6 - call _key_expansion_192b - aeskeygenassist $0x40, %xmm2, %xmm1 # round 7 - call _key_expansion_192a - aeskeygenassist $0x80, %xmm2, %xmm1 # round 8 - call _key_expansion_192b - jmp .Ldec_key -.Lenc_key128: - aeskeygenassist $0x1, %xmm0, %xmm1 # round 1 - call _key_expansion_128 - aeskeygenassist $0x2, %xmm0, %xmm1 # round 2 - call _key_expansion_128 - aeskeygenassist $0x4, %xmm0, %xmm1 # round 3 - call _key_expansion_128 - aeskeygenassist $0x8, %xmm0, %xmm1 # round 4 - call _key_expansion_128 - aeskeygenassist $0x10, %xmm0, %xmm1 # round 5 - call _key_expansion_128 - aeskeygenassist $0x20, %xmm0, %xmm1 # round 6 - call _key_expansion_128 - aeskeygenassist $0x40, %xmm0, %xmm1 # round 7 - call _key_expansion_128 - aeskeygenassist $0x80, %xmm0, %xmm1 # round 8 - call _key_expansion_128 - aeskeygenassist $0x1b, %xmm0, %xmm1 # round 9 - call _key_expansion_128 - aeskeygenassist $0x36, %xmm0, %xmm1 # round 10 - call _key_expansion_128 -.Ldec_key: - sub $0x10, TKEYP - movaps (KEYP), %xmm0 - movaps (TKEYP), %xmm1 - movaps %xmm0, 240(TKEYP) - movaps %xmm1, 240(KEYP) - add $0x10, KEYP - lea 240-16(TKEYP), UKEYP -.align 4 -.Ldec_key_loop: - movaps (KEYP), %xmm0 - aesimc %xmm0, %xmm1 - movaps %xmm1, (UKEYP) - add $0x10, KEYP - sub $0x10, UKEYP - cmp TKEYP, KEYP - jb .Ldec_key_loop -#ifndef __x86_64__ - popl KEYP -#endif - FRAME_END - RET -SYM_FUNC_END(aesni_set_key) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 0bda9abae368..3f86c8997d7d 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later /* * Support for AES-NI and VAES instructions. This file contains glue code. - * The real AES implementations are in aesni-intel_asm.S and other .S file= s. + * The real AES implementations are in .S files. * * Copyright (C) 2008, Intel Corp. * Author: Huang Ying @@ -26,7 +26,6 @@ #include #include #include -#include #include #include #include @@ -38,218 +37,7 @@ #include #include =20 - -#define AESNI_ALIGN 16 -#define AESNI_ALIGN_ATTR __attribute__ ((__aligned__(AESNI_ALIGN))) -#define AESNI_ALIGN_EXTRA ((AESNI_ALIGN - 1) & ~(CRYPTO_MINALIGN - 1)) -#define XTS_AES_CTX_SIZE (sizeof(struct aesni_xts_ctx) + AESNI_ALIGN_EXTRA) - -struct aesni_xts_ctx { - struct crypto_aes_ctx tweak_ctx AESNI_ALIGN_ATTR; - struct crypto_aes_ctx crypt_ctx AESNI_ALIGN_ATTR; -}; - -static inline void *aes_align_addr(void *addr) -{ - if (crypto_tfm_ctx_alignment() >=3D AESNI_ALIGN) - return addr; - return PTR_ALIGN(addr, AESNI_ALIGN); -} - -asmlinkage void aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, - unsigned int key_len); - -static inline struct aesni_xts_ctx *aes_xts_ctx(struct crypto_skcipher *tf= m) -{ - return aes_align_addr(crypto_skcipher_ctx(tfm)); -} - -static int aes_set_key_common(struct crypto_aes_ctx *ctx, - const u8 *in_key, unsigned int key_len) -{ - int err; - - if (!crypto_simd_usable()) - return aes_expandkey(ctx, in_key, key_len); - - err =3D aes_check_keylen(key_len); - if (err) - return err; - - kernel_fpu_begin(); - aesni_set_key(ctx, in_key, key_len); - kernel_fpu_end(); - return 0; -} - -static int xts_setkey_aesni(struct crypto_skcipher *tfm, const u8 *key, - unsigned int keylen) -{ - struct aesni_xts_ctx *ctx =3D aes_xts_ctx(tfm); - int err; - - err =3D xts_verify_key(tfm, key, keylen); - if (err) - return err; - - keylen /=3D 2; - - /* first half of xts-key is for crypt */ - err =3D aes_set_key_common(&ctx->crypt_ctx, key, keylen); - if (err) - return err; - - /* second half of xts-key is for tweak */ - return aes_set_key_common(&ctx->tweak_ctx, key + keylen, keylen); -} - -typedef void (*xts_encrypt_iv_func)(const struct crypto_aes_ctx *tweak_key, - u8 iv[AES_BLOCK_SIZE]); -typedef void (*xts_crypt_func)(const struct crypto_aes_ctx *key, - const u8 *src, u8 *dst, int len, - u8 tweak[AES_BLOCK_SIZE]); - -/* This handles cases where the source and/or destination span pages. */ -static noinline int -xts_crypt_slowpath(struct skcipher_request *req, xts_crypt_func crypt_func) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct aesni_xts_ctx *ctx =3D aes_xts_ctx(tfm); - int tail =3D req->cryptlen % AES_BLOCK_SIZE; - struct scatterlist sg_src[2], sg_dst[2]; - struct skcipher_request subreq; - struct skcipher_walk walk; - struct scatterlist *src, *dst; - int err; - - /* - * If the message length isn't divisible by the AES block size, then - * separate off the last full block and the partial block. This ensures - * that they are processed in the same call to the assembly function, - * which is required for ciphertext stealing. - */ - if (tail) { - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, - skcipher_request_flags(req), - NULL, NULL); - skcipher_request_set_crypt(&subreq, req->src, req->dst, - req->cryptlen - tail - AES_BLOCK_SIZE, - req->iv); - req =3D &subreq; - } - - err =3D skcipher_walk_virt(&walk, req, false); - - while (walk.nbytes) { - kernel_fpu_begin(); - (*crypt_func)(&ctx->crypt_ctx, - walk.src.virt.addr, walk.dst.virt.addr, - walk.nbytes & ~(AES_BLOCK_SIZE - 1), req->iv); - kernel_fpu_end(); - err =3D skcipher_walk_done(&walk, - walk.nbytes & (AES_BLOCK_SIZE - 1)); - } - - if (err || !tail) - return err; - - /* Do ciphertext stealing with the last full block and partial block. */ - - dst =3D src =3D scatterwalk_ffwd(sg_src, req->src, req->cryptlen); - if (req->dst !=3D req->src) - dst =3D scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); - - skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, - req->iv); - - err =3D skcipher_walk_virt(&walk, req, false); - if (err) - return err; - - kernel_fpu_begin(); - (*crypt_func)(&ctx->crypt_ctx, walk.src.virt.addr, walk.dst.virt.addr, - walk.nbytes, req->iv); - kernel_fpu_end(); - - return skcipher_walk_done(&walk, 0); -} - -/* __always_inline to avoid indirect call in fastpath */ -static __always_inline int -xts_crypt(struct skcipher_request *req, xts_encrypt_iv_func encrypt_iv, - xts_crypt_func crypt_func) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct aesni_xts_ctx *ctx =3D aes_xts_ctx(tfm); - - if (unlikely(req->cryptlen < AES_BLOCK_SIZE)) - return -EINVAL; - - kernel_fpu_begin(); - (*encrypt_iv)(&ctx->tweak_ctx, req->iv); - - /* - * In practice, virtually all XTS plaintexts and ciphertexts are either - * 512 or 4096 bytes and do not use multiple scatterlist elements. To - * optimize the performance of these cases, the below fast-path handles - * single-scatterlist-element messages as efficiently as possible. The - * code is 64-bit specific, as it assumes no page mapping is needed. - */ - if (IS_ENABLED(CONFIG_X86_64) && - likely(req->src->length >=3D req->cryptlen && - req->dst->length >=3D req->cryptlen)) { - (*crypt_func)(&ctx->crypt_ctx, sg_virt(req->src), - sg_virt(req->dst), req->cryptlen, req->iv); - kernel_fpu_end(); - return 0; - } - kernel_fpu_end(); - return xts_crypt_slowpath(req, crypt_func); -} - #ifdef CONFIG_X86_64 -asmlinkage void aes_xts_encrypt_iv(const struct crypto_aes_ctx *tweak_key, - u8 iv[AES_BLOCK_SIZE]); - -#define DEFINE_AVX_SKCIPHER_ALGS(suffix, driver_name_suffix, priority) = \ - \ -asmlinkage void \ -aes_xts_encrypt_##suffix(const struct crypto_aes_ctx *key, const u8 *src, = \ - u8 *dst, int len, u8 tweak[AES_BLOCK_SIZE]); \ -asmlinkage void \ -aes_xts_decrypt_##suffix(const struct crypto_aes_ctx *key, const u8 *src, = \ - u8 *dst, int len, u8 tweak[AES_BLOCK_SIZE]); \ - \ -static int xts_encrypt_##suffix(struct skcipher_request *req) \ -{ \ - return xts_crypt(req, aes_xts_encrypt_iv, aes_xts_encrypt_##suffix); \ -} \ - \ -static int xts_decrypt_##suffix(struct skcipher_request *req) \ -{ \ - return xts_crypt(req, aes_xts_encrypt_iv, aes_xts_decrypt_##suffix); \ -} \ - \ -static struct skcipher_alg skcipher_algs_##suffix[] =3D {{ \ - .base.cra_name =3D "xts(aes)", \ - .base.cra_driver_name =3D "xts-aes-" driver_name_suffix, \ - .base.cra_priority =3D priority, \ - .base.cra_blocksize =3D AES_BLOCK_SIZE, \ - .base.cra_ctxsize =3D XTS_AES_CTX_SIZE, \ - .base.cra_module =3D THIS_MODULE, \ - .min_keysize =3D 2 * AES_MIN_KEY_SIZE, \ - .max_keysize =3D 2 * AES_MAX_KEY_SIZE, \ - .ivsize =3D AES_BLOCK_SIZE, \ - .walksize =3D 2 * AES_BLOCK_SIZE, \ - .setkey =3D xts_setkey_aesni, \ - .encrypt =3D xts_encrypt_##suffix, \ - .decrypt =3D xts_decrypt_##suffix, \ -}} - -DEFINE_AVX_SKCIPHER_ALGS(aesni_avx, "aesni-avx", 500); -DEFINE_AVX_SKCIPHER_ALGS(vaes_avx2, "vaes-avx2", 600); -DEFINE_AVX_SKCIPHER_ALGS(vaes_avx512, "vaes-avx512", 800); =20 /* The common part of the x86_64 AES-GCM key struct */ struct aes_gcm_key { @@ -1004,10 +792,6 @@ static int __init register_avx_algs(void) =20 if (!boot_cpu_has(X86_FEATURE_AVX)) return 0; - err =3D crypto_register_skciphers(skcipher_algs_aesni_avx, - ARRAY_SIZE(skcipher_algs_aesni_avx)); - if (err) - return err; err =3D crypto_register_aeads(aes_gcm_algs_aesni_avx, ARRAY_SIZE(aes_gcm_algs_aesni_avx)); if (err) @@ -1024,10 +808,6 @@ static int __init register_avx_algs(void) !boot_cpu_has(X86_FEATURE_PCLMULQDQ) || !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) return 0; - err =3D crypto_register_skciphers(skcipher_algs_vaes_avx2, - ARRAY_SIZE(skcipher_algs_vaes_avx2)); - if (err) - return err; err =3D crypto_register_aeads(aes_gcm_algs_vaes_avx2, ARRAY_SIZE(aes_gcm_algs_vaes_avx2)); if (err) @@ -1043,16 +823,10 @@ static int __init register_avx_algs(void) if (boot_cpu_has(X86_FEATURE_PREFER_YMM)) { int i; =20 - for (i =3D 0; i < ARRAY_SIZE(skcipher_algs_vaes_avx512); i++) - skcipher_algs_vaes_avx512[i].base.cra_priority =3D 1; for (i =3D 0; i < ARRAY_SIZE(aes_gcm_algs_vaes_avx512); i++) aes_gcm_algs_vaes_avx512[i].base.cra_priority =3D 1; } =20 - err =3D crypto_register_skciphers(skcipher_algs_vaes_avx512, - ARRAY_SIZE(skcipher_algs_vaes_avx512)); - if (err) - return err; err =3D crypto_register_aeads(aes_gcm_algs_vaes_avx512, ARRAY_SIZE(aes_gcm_algs_vaes_avx512)); if (err) @@ -1061,19 +835,13 @@ static int __init register_avx_algs(void) return 0; } =20 -#define unregister_skciphers(A) \ - if (refcount_read(&(A)[0].base.cra_refcnt) !=3D 0) \ - crypto_unregister_skciphers((A), ARRAY_SIZE(A)) #define unregister_aeads(A) \ if (refcount_read(&(A)[0].base.cra_refcnt) !=3D 0) \ crypto_unregister_aeads((A), ARRAY_SIZE(A)) =20 static void unregister_avx_algs(void) { - unregister_skciphers(skcipher_algs_aesni_avx); unregister_aeads(aes_gcm_algs_aesni_avx); - unregister_skciphers(skcipher_algs_vaes_avx2); - unregister_skciphers(skcipher_algs_vaes_avx512); unregister_aeads(aes_gcm_algs_vaes_avx2); unregister_aeads(aes_gcm_algs_vaes_avx512); } @@ -1131,6 +899,6 @@ static void __exit aesni_exit(void) module_init(aesni_init); module_exit(aesni_exit); =20 -MODULE_DESCRIPTION("AES cipher and modes, optimized with AES-NI or VAES in= structions"); +MODULE_DESCRIPTION("AES-GCM, optimized with AES-NI or VAES instructions"); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("aes"); diff --git a/crypto/aes.c b/crypto/aes.c index ac484a28b30e..c19234f8a31c 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -701,7 +701,7 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "xts(aes)", .base.cra_driver_name =3D "xts-aes-lib", - .base.cra_priority =3D 110, + .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_xts_key), .base.cra_module =3D THIS_MODULE, diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile index 5d5484fc78ea..02d89a226377 100644 --- a/lib/crypto/Makefile +++ b/lib/crypto/Makefile @@ -57,6 +57,9 @@ libaes-$(CONFIG_X86) +=3D x86/aes-aesni.o ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),) libaes-$(CONFIG_X86_64) +=3D x86/aes-ctr-avx-x86_64.o endif +ifneq ($(CONFIG_CRYPTO_LIB_AES_XTS),) +libaes-$(CONFIG_X86_64) +=3D x86/aes-xts-avx-x86_64.o +endif endif # CONFIG_CRYPTO_LIB_AES_ARCH =20 # clean-files must be defined unconditionally diff --git a/arch/x86/crypto/aes-xts-avx-x86_64.S b/lib/crypto/x86/aes-xts-= avx-x86_64.S similarity index 81% rename from arch/x86/crypto/aes-xts-avx-x86_64.S rename to lib/crypto/x86/aes-xts-avx-x86_64.S index a30753a3e207..76788b275ab4 100644 --- a/arch/x86/crypto/aes-xts-avx-x86_64.S +++ b/lib/crypto/x86/aes-xts-avx-x86_64.S @@ -80,14 +80,17 @@ * any CPUs that support VAES but not VPCLMULQDQ. If that changes, we mig= ht * need to start also providing an implementation using VAES alone. * - * The AES-XTS implementations in this file support everything required by= the - * crypto API, including support for arbitrary input lengths and multi-part - * processing. However, they are most heavily optimized for the common ca= se of - * power-of-2 length inputs that are processed in a single part (disk sect= ors). + * These assembly functions don't handle ciphertext stealing, i.e, lengths= that + * aren't a multiple of 16 bytes. That case is not actually reached in the + * current use cases of AES-XTS in the kernel and is just handled by the C= code. */ =20 #include -#include + +// Offsets in struct aes_key +#define OFFSETOF_KEYLEN 0 +#define OFFSETOF_RNDKEYS 16 +#define OFFSETOF_INVRNDKEYS 256 =20 .section .rodata .p2align 4 @@ -111,16 +114,6 @@ .Llshift_amounts: .byte 0, 0, 1, 1, 2, 2, 3, 3 =20 - // This table contains constants for vpshufb and vpblendvb, used to - // handle variable byte shifts and blending during ciphertext stealing - // on CPUs that don't support AVX512-style masking. -.Lcts_permute_table: - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 - .byte 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 - .byte 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80 .text =20 .macro _define_Vi i @@ -149,13 +142,12 @@ .endif =20 // Function parameters - .set KEY, %rdi // Initially points to crypto_aes_ctx, then is + .set KEY, %rdi // Initially points to struct aes_key, then is // advanced to point to 7th-from-last round key .set SRC, %rsi // Pointer to next source data .set DST, %rdx // Pointer to next destination data - .set LEN, %ecx // Remaining length in bytes - .set LEN8, %cl - .set LEN64, %rcx + .set NBLOCKS, %rcx // Number of blocks remaining + .set NBLOCKS32, %ecx .set TWEAK, %r8 // Pointer to next tweak =20 // %rax holds the AES key length in bytes. @@ -468,9 +460,9 @@ =20 // Select either the encryption round keys or the decryption round keys. .if \enc - .set OFFS, 0 + .set OFFS, OFFSETOF_RNDKEYS .else - .set OFFS, 240 + .set OFFS, OFFSETOF_INVRNDKEYS .endif =20 // Load the round key for "round 0". @@ -615,19 +607,8 @@ .macro _aes_xts_crypt enc _define_aliases =20 -.if !\enc - // When decrypting a message whose length isn't a multiple of the AES - // block length, exclude the last full block from the main loop by - // subtracting 16 from LEN. This is needed because ciphertext stealing - // decryption uses the last two tweaks in reverse order. We'll handle - // the last full block and the partial block specially at the end. - lea -16(LEN), %eax - test $15, LEN8 - cmovnz %eax, LEN -.endif - // Load the AES key length: 16 (AES-128), 24 (AES-192), or 32 (AES-256). - movl 480(KEY), KEYLEN + movl OFFSETOF_KEYLEN(KEY), KEYLEN =20 // Setup the pointer to the round keys and cache as many as possible. _setup_round_keys \enc @@ -635,7 +616,7 @@ // Compute the first set of tweaks TWEAK[0-3]. _compute_first_set_of_tweaks =20 - add $-4*VL, LEN // shorter than 'sub 4*VL' when VL=3D32 + sub $4*VL/16, NBLOCKS jl .Lhandle_remainder\@ =20 .Lmain_loop\@: @@ -715,13 +696,13 @@ =20 sub $-4*VL, SRC // shorter than 'add 4*VL' when VL=3D32 sub $-4*VL, DST - add $-4*VL, LEN + sub $4*VL/16, NBLOCKS jge .Lmain_loop\@ =20 // Check for the uncommon case where the data length isn't a multiple of // 4*VL. Handle it out-of-line in order to optimize for the common // case. In the common case, just fall through to the ret. - test $4*VL-1, LEN8 + test $(4*VL/16)-1, NBLOCKS32 jnz .Lhandle_remainder\@ .Ldone\@: // Store the next tweak back to *TWEAK to support continuation calls. @@ -733,9 +714,9 @@ =20 .Lhandle_remainder\@: =20 - // En/decrypt any remaining full blocks, one vector at a time. + // En/decrypt any remaining blocks, one vector at a time. .if VL > 16 - add $3*VL, LEN // Undo extra sub of 4*VL, then sub VL. + add $3*VL/16, NBLOCKS32 jl .Lvec_at_a_time_done\@ .Lvec_at_a_time\@: _vmovdqu (SRC), V0 @@ -744,16 +725,16 @@ _next_tweakvec TWEAK0, V0, V1, TWEAK0 add $VL, SRC add $VL, DST - sub $VL, LEN + sub $VL/16, NBLOCKS32 jge .Lvec_at_a_time\@ .Lvec_at_a_time_done\@: - add $VL-16, LEN // Undo extra sub of VL, then sub 16. + add $VL/16, NBLOCKS32 .else - add $4*VL-16, LEN // Undo extra sub of 4*VL, then sub 16. + add $4*VL/16, NBLOCKS32 .endif =20 - // En/decrypt any remaining full blocks, one at a time. - jl .Lblock_at_a_time_done\@ + // En/decrypt any remaining blocks, one at a time. + jz .Ldone\@ .Lblock_at_a_time\@: vmovdqu (SRC), %xmm0 _aes_crypt \enc, _XMM, TWEAK0_XMM, %xmm0, tmp=3D%xmm1 @@ -761,92 +742,26 @@ _next_tweak TWEAK0_XMM, %xmm0, TWEAK0_XMM add $16, SRC add $16, DST - sub $16, LEN - jge .Lblock_at_a_time\@ -.Lblock_at_a_time_done\@: - add $16, LEN // Undo the extra sub of 16. - // Now 0 <=3D LEN <=3D 15. If LEN is zero, we're done. - jz .Ldone\@ - - // Otherwise 1 <=3D LEN <=3D 15, but the real remaining length is 16 + LE= N. - // Do ciphertext stealing to process the last 16 + LEN bytes. - -.if \enc - // If encrypting, the main loop already encrypted the last full block to - // create the CTS intermediate ciphertext. Prepare for the rest of CTS - // by rewinding the pointers and loading the intermediate ciphertext. - sub $16, SRC - sub $16, DST - vmovdqu (DST), %xmm0 -.else - // If decrypting, the main loop didn't decrypt the last full block - // because CTS decryption uses the last two tweaks in reverse order. - // Do it now by advancing the tweak and decrypting the last full block. - _next_tweak TWEAK0_XMM, %xmm0, TWEAK1_XMM - vmovdqu (SRC), %xmm0 - _aes_crypt \enc, _XMM, TWEAK1_XMM, %xmm0, tmp=3D%xmm1 -.endif - -.if USE_AVX512 - // Create a mask that has the first LEN bits set. - mov $-1, %r9d - bzhi LEN, %r9d, %r9d - kmovd %r9d, %k1 - - // Swap the first LEN bytes of the en/decryption of the last full block - // with the partial block. Note that to support in-place en/decryption, - // the load from the src partial block must happen before the store to - // the dst partial block. - vmovdqa %xmm0, %xmm1 - vmovdqu8 16(SRC), %xmm0{%k1} - vmovdqu8 %xmm1, 16(DST){%k1} -.else - lea .Lcts_permute_table(%rip), %r9 - - // Load the src partial block, left-aligned. Note that to support - // in-place en/decryption, this must happen before the store to the dst - // partial block. - vmovdqu (SRC, LEN64, 1), %xmm1 - - // Shift the first LEN bytes of the en/decryption of the last full block - // to the end of a register, then store it to DST+LEN. This stores the - // dst partial block. It also writes to the second part of the dst last - // full block, but that part is overwritten later. - vpshufb (%r9, LEN64, 1), %xmm0, %xmm2 - vmovdqu %xmm2, (DST, LEN64, 1) - - // Make xmm3 contain [16-LEN,16-LEN+1,...,14,15,0x80,0x80,...]. - sub LEN64, %r9 - vmovdqu 32(%r9), %xmm3 - - // Shift the src partial block to the beginning of its register. - vpshufb %xmm3, %xmm1, %xmm1 - - // Do a blend to generate the src partial block followed by the second - // part of the en/decryption of the last full block. - vpblendvb %xmm3, %xmm0, %xmm1, %xmm0 -.endif - // En/decrypt again and store the last full block. - _aes_crypt \enc, _XMM, TWEAK0_XMM, %xmm0, tmp=3D%xmm1 - vmovdqu %xmm0, (DST) + dec NBLOCKS32 + jnz .Lblock_at_a_time\@ jmp .Ldone\@ .endm =20 -// void aes_xts_encrypt_iv(const struct crypto_aes_ctx *tweak_key, +// void aes_xts_encrypt_iv(const struct aes_enckey *tweak_key, // u8 iv[AES_BLOCK_SIZE]); // // Encrypt |iv| using the AES key |tweak_key| to get the first tweak. Ass= umes // that the CPU supports AES-NI and AVX, but not necessarily VAES or AVX51= 2. -SYM_TYPED_FUNC_START(aes_xts_encrypt_iv) +SYM_FUNC_START(aes_xts_encrypt_iv) .set TWEAK_KEY, %rdi .set IV, %rsi .set KEYLEN, %eax .set KEYLEN64, %rax =20 vmovdqu (IV), %xmm0 - vpxor (TWEAK_KEY), %xmm0, %xmm0 - movl 480(TWEAK_KEY), KEYLEN - lea -16(TWEAK_KEY, KEYLEN64, 4), TWEAK_KEY + vpxor OFFSETOF_RNDKEYS(TWEAK_KEY), %xmm0, %xmm0 + movl OFFSETOF_KEYLEN(TWEAK_KEY), KEYLEN + lea OFFSETOF_RNDKEYS-16(TWEAK_KEY, KEYLEN64, 4), TWEAK_KEY cmp $24, KEYLEN jl .Lencrypt_iv_aes128 je .Lencrypt_iv_aes192 @@ -867,39 +782,36 @@ SYM_FUNC_END(aes_xts_encrypt_iv) // Below are the actual AES-XTS encryption and decryption functions, // instantiated from the above macro. They all have the following prototy= pe: // -// void (*xts_crypt_func)(const struct crypto_aes_ctx *key, -// const u8 *src, u8 *dst, int len, -// u8 tweak[AES_BLOCK_SIZE]); +// void (*xts_crypt_func)(const struct aes_key *key, const u8 *src, u8 *ds= t, +// long nblocks, u8 tweak[AES_BLOCK_SIZE]); // -// |key| is the data key. |tweak| contains the next tweak; the encryption= of -// the original IV with the tweak key was already done. This function sup= ports -// incremental computation, but |len| must always be >=3D 16 (AES_BLOCK_SI= ZE), and -// |len| must be a multiple of 16 except on the last call. If |len| is a -// multiple of 16, then this function updates |tweak| to contain the next = tweak. +// `tweak` must have already been encrypted by the tweak key; `key` is jus= t the +// main key. To allow incremental computation, `tweak` is updated to cont= ain +// the next tweak. =20 .set VL, 16 .set USE_AVX512, 0 -SYM_TYPED_FUNC_START(aes_xts_encrypt_aesni_avx) +SYM_FUNC_START(aes_xts_encrypt_aesni_avx) _aes_xts_crypt 1 SYM_FUNC_END(aes_xts_encrypt_aesni_avx) -SYM_TYPED_FUNC_START(aes_xts_decrypt_aesni_avx) +SYM_FUNC_START(aes_xts_decrypt_aesni_avx) _aes_xts_crypt 0 SYM_FUNC_END(aes_xts_decrypt_aesni_avx) =20 .set VL, 32 .set USE_AVX512, 0 -SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx2) +SYM_FUNC_START(aes_xts_encrypt_vaes_avx2) _aes_xts_crypt 1 SYM_FUNC_END(aes_xts_encrypt_vaes_avx2) -SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx2) +SYM_FUNC_START(aes_xts_decrypt_vaes_avx2) _aes_xts_crypt 0 SYM_FUNC_END(aes_xts_decrypt_vaes_avx2) =20 .set VL, 64 .set USE_AVX512, 1 -SYM_TYPED_FUNC_START(aes_xts_encrypt_vaes_avx512) +SYM_FUNC_START(aes_xts_encrypt_vaes_avx512) _aes_xts_crypt 1 SYM_FUNC_END(aes_xts_encrypt_vaes_avx512) -SYM_TYPED_FUNC_START(aes_xts_decrypt_vaes_avx512) +SYM_FUNC_START(aes_xts_decrypt_vaes_avx512) _aes_xts_crypt 0 SYM_FUNC_END(aes_xts_decrypt_vaes_avx512) diff --git a/lib/crypto/x86/aes.h b/lib/crypto/x86/aes.h index 5b4205870b9f..8b800f109263 100644 --- a/lib/crypto/x86/aes.h +++ b/lib/crypto/x86/aes.h @@ -307,6 +307,22 @@ void aes_xts_encrypt_aesni(u8 *dst, const u8 *src, lon= g nblocks, u8 tweak[AES_BLOCK_SIZE], const struct aes_key *key); void aes_xts_decrypt_aesni(u8 *dst, const u8 *src, long nblocks, u8 tweak[AES_BLOCK_SIZE], const struct aes_key *key); +void aes_xts_encrypt_iv(const struct aes_enckey *tweak_key, + u8 iv[AES_BLOCK_SIZE]); +void aes_xts_encrypt_aesni_avx(const struct aes_key *key, const u8 *src, + u8 *dst, long nblocks, u8 tweak[AES_BLOCK_SIZE]); +void aes_xts_decrypt_aesni_avx(const struct aes_key *key, const u8 *src, + u8 *dst, long nblocks, u8 tweak[AES_BLOCK_SIZE]); +void aes_xts_encrypt_vaes_avx2(const struct aes_key *key, const u8 *src, + u8 *dst, long nblocks, u8 tweak[AES_BLOCK_SIZE]); +void aes_xts_decrypt_vaes_avx2(const struct aes_key *key, const u8 *src, + u8 *dst, long nblocks, u8 tweak[AES_BLOCK_SIZE]); +void aes_xts_encrypt_vaes_avx512(const struct aes_key *key, const u8 *src, + u8 *dst, long nblocks, + u8 tweak[AES_BLOCK_SIZE]); +void aes_xts_decrypt_vaes_avx512(const struct aes_key *key, const u8 *src, + u8 *dst, long nblocks, + u8 tweak[AES_BLOCK_SIZE]); =20 /* len is always a positive multiple of AES_BLOCK_SIZE here. */ static __always_inline bool @@ -319,12 +335,46 @@ aes_xts_crypt_x86(u8 *dst, const u8 *src, size_t len,= u8 tweak[AES_BLOCK_SIZE], return false; =20 kernel_fpu_begin(); - if (!cont) - aes_encrypt_aesni(tweak, tweak, &key->tweak_key); - if (enc) - aes_xts_encrypt_aesni(dst, src, nblocks, tweak, &key->main_key); - else - aes_xts_decrypt_aesni(dst, src, nblocks, tweak, &key->main_key); + if (IS_ENABLED(CONFIG_X86_64) && + static_branch_likely(&have_vaes_avx512)) { + if (!cont) + aes_xts_encrypt_iv(&key->tweak_key, tweak); + if (enc) + aes_xts_encrypt_vaes_avx512(&key->main_key, src, dst, + nblocks, tweak); + else + aes_xts_decrypt_vaes_avx512(&key->main_key, src, dst, + nblocks, tweak); + } else if (IS_ENABLED(CONFIG_X86_64) && + static_branch_likely(&have_vaes_avx2)) { + if (!cont) + aes_xts_encrypt_iv(&key->tweak_key, tweak); + if (enc) + aes_xts_encrypt_vaes_avx2(&key->main_key, src, dst, + nblocks, tweak); + else + aes_xts_decrypt_vaes_avx2(&key->main_key, src, dst, + nblocks, tweak); + } else if (IS_ENABLED(CONFIG_X86_64) && + static_branch_likely(&have_aesni_avx)) { + if (!cont) + aes_xts_encrypt_iv(&key->tweak_key, tweak); + if (enc) + aes_xts_encrypt_aesni_avx(&key->main_key, src, dst, + nblocks, tweak); + else + aes_xts_decrypt_aesni_avx(&key->main_key, src, dst, + nblocks, tweak); + } else { + if (!cont) + aes_encrypt_aesni(tweak, tweak, &key->tweak_key); + if (enc) + aes_xts_encrypt_aesni(dst, src, nblocks, tweak, + &key->main_key); + else + aes_xts_decrypt_aesni(dst, src, nblocks, tweak, + &key->main_key); + } kernel_fpu_end(); return true; } --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C83A361974; Mon, 21 Sep 2026 05:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967771; cv=none; b=eMbrUSip1oC+dD5y0ZayXVKAYn45LTSjLUr0hCFKpiHelE3wpvdwR+nNpQGb5guLDJ/fW0Sk6f5/w7XAxIYrJjQZjT9j2Bx1kKwJqZZLava339cmEEcUATU2J/XvgHLpyKjGozuiw5fRdvTHBMVakglEV2BHnWXflJbmkm3DLio= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967771; c=relaxed/simple; bh=WdbOpTBnvX7IHZKY62+X8zA8hprUTgiUDovCozc88kc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=A+hyTrBlxN+emXyiCB7rAyKwEk7bMO7hzOq7ygvecw+Bjnt3sTf54zGmzVrj2JkTYTYpd44OXUwNU7YsO750ED0VRPsXLZgS01ErCaC48tH2j4uxbWg+nV/xHcqkeHoarMIP+NWfibk2pHGNHQCQAl6harXsWL3kqw2TUPAVrV8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fjHkWEA4; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fjHkWEA4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5F3E41F0089A; Mon, 21 Sep 2026 05:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967768; bh=ShajkHyOS9HMwDyE76SdTvTXI8joKtZkz+CVThGSWQU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=fjHkWEA4BZBbPqGzXi1TJZSClKpl6a65mrBbJTmF+VOnXn5aEGgQKQ/HW2uJHhpmG BtiIA96RqU4///rnVGYJafW6DLvTLH7zPcLA4BDb+zYe0Mwveu6rsMlXkhDdQfCPXN FoVZYdChexVqEBAZKHiW6W9BCO7OmFKknPcrEc6akWD79BDPR8YLkWtKGG9PGSnJ0r Q8ZpsgXs0sXi9sG+o+odO1i1p1oUA6/QVVTy//PbnBg+V7h+m5MnzrbTfmBArgpEgk 0slDqZ6sXwddif5zidaU2gRLhX2SYDH1uliZgVdREfZFSz8rVRZyOajTECs2hps348 1RYpFF85sWxPg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 14/20] crypto: x86/aes - Drop superseded 32-bit build support Date: Sun, 20 Sep 2026 22:09:00 -0700 Message-ID: <20260921050910.296144-15-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that the AES-NI and/or VAES accelerated implementations of AES-ECB, AES-CBC, AES-CBC-CTS, AES-CTR, AES-XCTR, and AES-XTS have been migrated into the "libaes" module in lib/crypto/ and enabled by default when the corresponding non-arch-specific options (e.g. CRYPTO_XTS) are enabled, the traditional "aesni-intel" module only has AES-GCM left. That functionality is 64-bit only. Therefore, aesni-intel no longer has any functionality on 32-bit. Stop building it on 32-bit. Also update the help text to mention the generic options. To be clear: AES-NI accelerated AES-ECB, AES-CBC, AES-CBC-CTS, and AES-XTS remain fully supported in 32-bit x86 kernels via libaes. Signed-off-by: Eric Biggers --- arch/x86/crypto/Kconfig | 7 ++++--- arch/x86/crypto/Makefile | 6 ++---- arch/x86/crypto/aesni-intel_glue.c | 14 -------------- 3 files changed, 6 insertions(+), 21 deletions(-) diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index d68b31fad508..9cb5176931aa 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -4,18 +4,19 @@ menu "Accelerated Cryptographic Algorithms for CPU (x86)" =20 config CRYPTO_AES_NI_INTEL tristate "Ciphers: AES, modes: GCM (AES-NI/VAES)" + depends on 64BIT select CRYPTO_AEAD select CRYPTO_LIB_AES select CRYPTO_LIB_GF128MUL help AEAD cipher: AES with GCM =20 - Architecture: x86 (32-bit and 64-bit) using: + Architecture: x86_64 using: - AES-NI (AES new instructions) - VAES (Vector AES) =20 - Some algorithm implementations are supported only in 64-bit builds, - and some have additional prerequisites such as AVX2 or AVX512. + Note: this option no longer provides the accelerated XTS, CBC, CTR, + and ECB code. For those just use CRYPTO_XTS, CRYPTO_CBC, etc. =20 config CRYPTO_BLOWFISH_X86_64 tristate "Ciphers: Blowfish, modes: ECB, CBC" diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile index e05d6e2257d4..aba817fe64f5 100644 --- a/arch/x86/crypto/Makefile +++ b/arch/x86/crypto/Makefile @@ -40,10 +40,8 @@ obj-$(CONFIG_CRYPTO_AEGIS128_AESNI_SSE2) +=3D aegis128-a= esni.o aegis128-aesni-y :=3D aegis128-aesni-asm.o aegis128-aesni-glue.o =20 obj-$(CONFIG_CRYPTO_AES_NI_INTEL) +=3D aesni-intel.o -aesni-intel-y :=3D aesni-intel_glue.o -aesni-intel-$(CONFIG_64BIT) +=3D aes-gcm-aesni-x86_64.o \ - aes-gcm-vaes-avx2.o \ - aes-gcm-vaes-avx512.o +aesni-intel-y :=3D aesni-intel_glue.o aes-gcm-aesni-x86_64.o \ + aes-gcm-vaes-avx2.o aes-gcm-vaes-avx512.o =20 obj-$(CONFIG_CRYPTO_SM4_AESNI_AVX_X86_64) +=3D sm4-aesni-avx-x86_64.o sm4-aesni-avx-x86_64-y :=3D sm4-aesni-avx-asm_64.o sm4_aesni_avx_glue.o diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 3f86c8997d7d..af52c442361f 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -37,8 +37,6 @@ #include #include =20 -#ifdef CONFIG_X86_64 - /* The common part of the x86_64 AES-GCM key struct */ struct aes_gcm_key { /* Expanded AES key and the AES key length in bytes */ @@ -845,18 +843,6 @@ static void unregister_avx_algs(void) unregister_aeads(aes_gcm_algs_vaes_avx2); unregister_aeads(aes_gcm_algs_vaes_avx512); } -#else /* CONFIG_X86_64 */ -static struct aead_alg aes_gcm_algs_aesni[0]; - -static int __init register_avx_algs(void) -{ - return 0; -} - -static void unregister_avx_algs(void) -{ -} -#endif /* !CONFIG_X86_64 */ =20 static const struct x86_cpu_id aesni_cpu_id[] =3D { X86_MATCH_FEATURE(X86_FEATURE_AES, NULL), --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24D1436195C; Mon, 21 Sep 2026 05:16:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967772; cv=none; b=TFdl7fv+iWyeE1fTKlgVLdJ+1+VsWhF5VYJwjyeDu/WSb1ubVPA9dslJ9vERu5d5cboGQ5X5Bj1Wp7dfa8ANwpdVgHJ7Mn4jACd87Tk/bQwBydCHu7vooGFLz7ZNoPO8/gUKZXzNAhbOH2o6I2SXi4JlXAtMBYRISFkNJaW9Jtg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967772; c=relaxed/simple; bh=L9TXqDmje3l7vbXeBbsMs2/svsGmBQAvC/uNFmhLLmw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mVGxefTCZ1V5p/Fqv2Lbb1la38caHTddOJc9oiBB7uLwNNEnRlQl9twSmwLjHe9ljZ3yDNGOOauEeUTQhaSxEW14OPmOrrbpem1q1AH22FV2pfSyWgkyr625xWykwzzK3x4mKY9bY2dr0+hjtvHB36PbH7IaLCfIhFhfIzRuHFc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Rw2a6kH4; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Rw2a6kH4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BAE431F0089B; Mon, 21 Sep 2026 05:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967769; bh=JtXdB4tOMY8yQZAnGFn5daFwsAwdwklSo1JeZmkW/k8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Rw2a6kH4ITMGqzwwlWBH9DjIOk8Nemf8R0QqFZUafMVf/lrVBFiIiSUCeU5njYOeE FhZPgIlXE2uAUSXDx4RK7H4143qgM0dwasVO6sStF3azimpR9Xbc/FB3wQsvVzTeA1 lG+Z3zG/EKGvqfI9KoFy4zoBFwXTTM0FNW4ykSLwr8Jo+4U1LSiNI41f5PXWJJXDrp 5M5UmqdGYCKjw0EK0CM7/Zz45UL7YY4R9WFMs9TWiKmr4uScADPUpLlMJ945bIbmZy eWcr1v1iEJWVmwUvrnnRz+8/X6chmPSWBLjrcac85zVOKGmdHZvFqqgbnkb4vhpjjo Gef2XAlWBgKGg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 15/20] lib/crypto: riscv/aes: Copy aes-macros.S to library Date: Sun, 20 Sep 2026 22:09:01 -0700 Message-ID: <20260921050910.296144-16-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Copy arch/riscv/crypto/aes-macros.S to lib/crypto/riscv/aes-macros.S and make lib/crypto/riscv/aes-riscv64-zvkned.S include the latter copy. This makes it possible to change these macros without interfering with the remaining code in arch/riscv/crypto/. Of course, the copy in arch/riscv/crypto/ will be removed once the rest of the AES code there is migrated to the library. Signed-off-by: Eric Biggers --- lib/crypto/riscv/aes-macros.S | 166 ++++++++++++++++++++++++++ lib/crypto/riscv/aes-riscv64-zvkned.S | 2 +- 2 files changed, 167 insertions(+), 1 deletion(-) create mode 100644 lib/crypto/riscv/aes-macros.S diff --git a/lib/crypto/riscv/aes-macros.S b/lib/crypto/riscv/aes-macros.S new file mode 100644 index 000000000000..1384164621a5 --- /dev/null +++ b/lib/crypto/riscv/aes-macros.S @@ -0,0 +1,166 @@ +/* SPDX-License-Identifier: Apache-2.0 OR BSD-2-Clause */ +// +// This file is dual-licensed, meaning that you can use it under your +// choice of either of the following two licenses: +// +// Copyright 2023 The OpenSSL Project Authors. All Rights Reserved. +// +// Licensed under the Apache License 2.0 (the "License"). You can obtain +// a copy in the file LICENSE in the source distribution or at +// https://www.openssl.org/source/license.html +// +// or +// +// Copyright (c) 2023, Christoph M=C3=BCllner +// Copyright (c) 2023, Phoebe Chen +// Copyright (c) 2023, Jerry Shih +// Copyright 2024 Google LLC +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions +// are met: +// 1. Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// 2. Redistributions in binary form must reproduce the above copyright +// notice, this list of conditions and the following disclaimer in the +// documentation and/or other materials provided with the distribution. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +// This file contains macros that are shared by the other aes-*.S files. = The +// generated code of these macros depends on the following RISC-V extensio= ns: +// - RV64I +// - RISC-V Vector ('V') with VLEN >=3D 128 +// - RISC-V Vector AES block cipher extension ('Zvkned') + +// Loads the AES round keys from \keyp into vector registers and jumps to = code +// specific to the length of the key. Specifically: +// - If AES-128, loads round keys into v1-v11 and jumps to \label128. +// - If AES-192, loads round keys into v1-v13 and jumps to \label192. +// - If AES-256, loads round keys into v1-v15 and continues onwards. +// +// Also sets vl=3D4 and vtype=3De32,m1,ta,ma. Clobbers t0 and t1. +.macro aes_begin keyp, label128, label192, key_len +.ifb \key_len + lwu t0, 480(\keyp) // t0 =3D key length in bytes +.endif + li t1, 24 // t1 =3D key length for AES-192 + vsetivli zero, 4, e32, m1, ta, ma + vle32.v v1, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v2, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v3, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v4, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v5, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v6, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v7, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v8, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v9, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v10, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v11, (\keyp) +.ifb \key_len + blt t0, t1, \label128 // If AES-128, goto label128. +.else + blt \key_len, t1, \label128 // If AES-128, goto label128. +.endif + addi \keyp, \keyp, 16 + vle32.v v12, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v13, (\keyp) +.ifb \key_len + beq t0, t1, \label192 // If AES-192, goto label192. +.else + beq \key_len, t1, \label192 // If AES-192, goto label192. +.endif + // Else, it's AES-256. + addi \keyp, \keyp, 16 + vle32.v v14, (\keyp) + addi \keyp, \keyp, 16 + vle32.v v15, (\keyp) +.endm + +// Encrypts \data using zvkned instructions, using the round keys loaded i= nto +// v1-v11 (for AES-128), v1-v13 (for AES-192), or v1-v15 (for AES-256). \= keylen +// is the AES key length in bits. vl and vtype must already be set +// appropriately. Note that if vl > 4, multiple blocks are encrypted. +.macro aes_encrypt data, keylen + vaesz.vs \data, v1 + vaesem.vs \data, v2 + vaesem.vs \data, v3 + vaesem.vs \data, v4 + vaesem.vs \data, v5 + vaesem.vs \data, v6 + vaesem.vs \data, v7 + vaesem.vs \data, v8 + vaesem.vs \data, v9 + vaesem.vs \data, v10 +.if \keylen =3D=3D 128 + vaesef.vs \data, v11 +.elseif \keylen =3D=3D 192 + vaesem.vs \data, v11 + vaesem.vs \data, v12 + vaesef.vs \data, v13 +.else + vaesem.vs \data, v11 + vaesem.vs \data, v12 + vaesem.vs \data, v13 + vaesem.vs \data, v14 + vaesef.vs \data, v15 +.endif +.endm + +// Same as aes_encrypt, but decrypts instead of encrypts. +.macro aes_decrypt data, keylen +.if \keylen =3D=3D 128 + vaesz.vs \data, v11 +.elseif \keylen =3D=3D 192 + vaesz.vs \data, v13 + vaesdm.vs \data, v12 + vaesdm.vs \data, v11 +.else + vaesz.vs \data, v15 + vaesdm.vs \data, v14 + vaesdm.vs \data, v13 + vaesdm.vs \data, v12 + vaesdm.vs \data, v11 +.endif + vaesdm.vs \data, v10 + vaesdm.vs \data, v9 + vaesdm.vs \data, v8 + vaesdm.vs \data, v7 + vaesdm.vs \data, v6 + vaesdm.vs \data, v5 + vaesdm.vs \data, v4 + vaesdm.vs \data, v3 + vaesdm.vs \data, v2 + vaesdf.vs \data, v1 +.endm + +// Expands to aes_encrypt or aes_decrypt according to \enc, which is 1 or = 0. +.macro aes_crypt data, enc, keylen +.if \enc + aes_encrypt \data, \keylen +.else + aes_decrypt \data, \keylen +.endif +.endm diff --git a/lib/crypto/riscv/aes-riscv64-zvkned.S b/lib/crypto/riscv/aes-r= iscv64-zvkned.S index 0d988bc3d37b..7a52ea6c669d 100644 --- a/lib/crypto/riscv/aes-riscv64-zvkned.S +++ b/lib/crypto/riscv/aes-riscv64-zvkned.S @@ -48,7 +48,7 @@ .text .option arch, +zvkned =20 -#include "../../arch/riscv/crypto/aes-macros.S" +#include "aes-macros.S" =20 #define RNDKEYS a0 #define KEY_LEN a1 --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28A7F36308F; Mon, 21 Sep 2026 05:16:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967772; cv=none; b=MB9pytQYv07TTyDlOCkBaB2mWu5HN7pXRFAybDywux8/nyJzTIr2YFO8XzAzT+A9eypsAyHLE26tceTJGwG+E2XHKHsCTRn1rlMTO/EfAK/h2OQMD+0/3mYNBrLU+/asyz3J16eXZFEOlnT9De0HDTeoEOcFrdKafiWe4sENQA0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967772; c=relaxed/simple; bh=bPmW6wqCL/cuG9YCAT2htcCSbOqD3uiz4WQ7EBZgVXA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aPolry2oynwlbWEPTeLvzdSG/19K1T8+seEuetUxOypaAHViRmxrHH9RyAfbF/4N3UwjRWivnjz9shlqSXQiJECXH7NsuFJPoKMorjdbLlJtxaylZZ8cgpCBsxOJ+x7kSVs/kxKzJdo/y77voVPGiZCuZcXCb07A657K0ou/9W0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mMNlvo4H; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mMNlvo4H" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 21E141F008A0; Mon, 21 Sep 2026 05:16:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967769; bh=A0Q4ywSNwm+bmXk82fgsubJuFKPHWy+87PLk05b7EkA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=mMNlvo4HsVDSxO6tsOLlvbCX8oCQA4G1VYajpXq6dhScvBcMOJRl/d6MD1GgBMYez Id+wqhS0GV2CUCYazuUWGpf4fsMT692B2b1VWKXHplwMwg0Ok8RzzmkAh4BKxxUGDH to3vdTBZxClo/BbIsP/ab/4ZMsdGYPI080kFmgqiK6Geb7NepVtbq1I40sHahLEdwd 8lrjHLLYitnCqXgOLJ1AcGN//Jx2sglV8ekGrERbTCoX8KEsE4UuNjg3dN7glIBNuN lJkOVEaJwqjiPYD4/5QKN83dzEoEQkJNiflyfc4KZCVhDTCwzQOXMWQDteg6oq38JF UsmaxlrWWJ9jA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 16/20] lib/crypto: riscv/aes: Pass key struct to assembly code Date: Sun, 20 Sep 2026 22:09:02 -0700 Message-ID: <20260921050910.296144-17-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make the assembly code take the AES key struct directly, rather than the round keys pointer and key length separately. Make the aes_begin macro assume this convention, and remove support for the legacy 'struct crypto_aes_ctx' from it since that isn't used here. This aligns with the convention that is being used (and will continue to be used) for the AES modes, it makes the C glue code slightly simpler, and it avoids the unnecessary shuffling around of arguments. Signed-off-by: Eric Biggers --- lib/crypto/riscv/aes-macros.S | 25 ++++++++++--------------- lib/crypto/riscv/aes-riscv64-zvkned.S | 13 ++++++------- lib/crypto/riscv/aes.h | 12 ++++++++---- 3 files changed, 24 insertions(+), 26 deletions(-) diff --git a/lib/crypto/riscv/aes-macros.S b/lib/crypto/riscv/aes-macros.S index 1384164621a5..720ad69a41ac 100644 --- a/lib/crypto/riscv/aes-macros.S +++ b/lib/crypto/riscv/aes-macros.S @@ -44,17 +44,20 @@ // - RISC-V Vector ('V') with VLEN >=3D 128 // - RISC-V Vector AES block cipher extension ('Zvkned') =20 -// Loads the AES round keys from \keyp into vector registers and jumps to = code -// specific to the length of the key. Specifically: +// Offsets in struct aes_enckey +#define OFFSETOF_KEYLEN 0 +#define OFFSETOF_RNDKEYS 16 + +// Loads the AES round keys from the struct aes_enckey \keyp into vector +// registers and jumps to code specific to the length of the key. Specifi= cally: // - If AES-128, loads round keys into v1-v11 and jumps to \label128. // - If AES-192, loads round keys into v1-v13 and jumps to \label192. // - If AES-256, loads round keys into v1-v15 and continues onwards. // -// Also sets vl=3D4 and vtype=3De32,m1,ta,ma. Clobbers t0 and t1. -.macro aes_begin keyp, label128, label192, key_len -.ifb \key_len - lwu t0, 480(\keyp) // t0 =3D key length in bytes -.endif +// Also sets vl=3D4 and vtype=3De32,m1,ta,ma. Clobbers keyp, t0, and t1. +.macro aes_begin keyp, label128, label192 + lwu t0, OFFSETOF_KEYLEN(\keyp) // t0 =3D key length in bytes + addi \keyp, \keyp, OFFSETOF_RNDKEYS li t1, 24 // t1 =3D key length for AES-192 vsetivli zero, 4, e32, m1, ta, ma vle32.v v1, (\keyp) @@ -78,20 +81,12 @@ vle32.v v10, (\keyp) addi \keyp, \keyp, 16 vle32.v v11, (\keyp) -.ifb \key_len blt t0, t1, \label128 // If AES-128, goto label128. -.else - blt \key_len, t1, \label128 // If AES-128, goto label128. -.endif addi \keyp, \keyp, 16 vle32.v v12, (\keyp) addi \keyp, \keyp, 16 vle32.v v13, (\keyp) -.ifb \key_len beq t0, t1, \label192 // If AES-192, goto label192. -.else - beq \key_len, t1, \label192 // If AES-192, goto label192. -.endif // Else, it's AES-256. addi \keyp, \keyp, 16 vle32.v v14, (\keyp) diff --git a/lib/crypto/riscv/aes-riscv64-zvkned.S b/lib/crypto/riscv/aes-r= iscv64-zvkned.S index 7a52ea6c669d..374fc4dba11b 100644 --- a/lib/crypto/riscv/aes-riscv64-zvkned.S +++ b/lib/crypto/riscv/aes-riscv64-zvkned.S @@ -50,10 +50,9 @@ =20 #include "aes-macros.S" =20 -#define RNDKEYS a0 -#define KEY_LEN a1 -#define OUTP a2 -#define INP a3 +#define KEYP a0 +#define OUTP a1 +#define INP a2 =20 .macro __aes_crypt_zvkned enc, keybits vle32.v v16, (INP) @@ -63,7 +62,7 @@ .endm =20 .macro aes_crypt_zvkned enc - aes_begin RNDKEYS, 128f, 192f, KEY_LEN + aes_begin KEYP, 128f, 192f __aes_crypt_zvkned \enc, 256 128: __aes_crypt_zvkned \enc, 128 @@ -71,13 +70,13 @@ __aes_crypt_zvkned \enc, 192 .endm =20 -// void aes_encrypt_zvkned(const u32 rndkeys[], int key_len, +// void aes_encrypt_zvkned(const struct aes_enckey *key, // u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); SYM_FUNC_START(aes_encrypt_zvkned) aes_crypt_zvkned 1 SYM_FUNC_END(aes_encrypt_zvkned) =20 -// void aes_decrypt_zvkned(const u32 rndkeys[], int key_len, +// void aes_decrypt_zvkned(const struct aes_key *key, // u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); SYM_FUNC_START(aes_decrypt_zvkned) aes_crypt_zvkned 0 diff --git a/lib/crypto/riscv/aes.h b/lib/crypto/riscv/aes.h index 0b26f58faf2b..a288b4c5b493 100644 --- a/lib/crypto/riscv/aes.h +++ b/lib/crypto/riscv/aes.h @@ -10,9 +10,13 @@ =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_zvkned); =20 -void aes_encrypt_zvkned(const u32 rndkeys[], int key_len, +/* The assembly code assumes the following offsets. */ +static_assert(offsetof(struct aes_enckey, len) =3D=3D 0); +static_assert(offsetof(struct aes_enckey, k.rndkeys) =3D=3D 16); + +void aes_encrypt_zvkned(const struct aes_enckey *key, u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); -void aes_decrypt_zvkned(const u32 rndkeys[], int key_len, +void aes_decrypt_zvkned(const struct aes_key *key, u8 out[AES_BLOCK_SIZE], const u8 in[AES_BLOCK_SIZE]); =20 static void aes_preparekey_arch(union aes_enckey_arch *k, @@ -29,7 +33,7 @@ static void aes_encrypt_arch(const struct aes_enckey *key, { if (static_branch_likely(&have_zvkned) && likely(may_use_simd())) { kernel_vector_begin(); - aes_encrypt_zvkned(key->k.rndkeys, key->len, out, in); + aes_encrypt_zvkned(key, out, in); kernel_vector_end(); } else { aes_encrypt_generic(key->k.rndkeys, key->nrounds, out, in); @@ -46,7 +50,7 @@ static void aes_decrypt_arch(const struct aes_key *key, */ if (static_branch_likely(&have_zvkned) && likely(may_use_simd())) { kernel_vector_begin(); - aes_decrypt_zvkned(key->k.rndkeys, key->len, out, in); + aes_decrypt_zvkned(key, out, in); kernel_vector_end(); } else { aes_decrypt_generic(key->inv_k.inv_rndkeys, key->nrounds, --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8F82364E98; Mon, 21 Sep 2026 05:16:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967773; cv=none; b=h9WCuPzewQvtyDOoz2dFOeXkGBmGFfu7uCDT+bMaMAtZwkFzSkwNbKxtYbnn5qgFmY4novMIso0Scq+RLGbVK0OEwlv+emYDNvcTH2Bu5N453PkWasQJVyWXUCyINcb9yUlolXyJIvxIRUbEOOFXhbGLhpl88d4EyzuYY9/AJ+s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967773; c=relaxed/simple; bh=3Oix9rxWfCPGMJLQY0b6ysQ93dwLwQKQNObL47lLbsw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Zhly90EK7swxgiiyUV7Jlg/OFaaSjpnGWqDSYXWv/A8q6dv0/nK5GL1GGqiGXF+ar8HrkNhoBbWGjrIFb3rVTszq1rcmDGWJJVRbYCYbsUbuz8GisjaisBOhuCw2mif42sDi1kUWVfKnzeJscc/CAOdR90QVTnWCW+hPQDj4KsU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iLMSvSP+; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iLMSvSP+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7C56B1F008A1; Mon, 21 Sep 2026 05:16:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967769; bh=1rXrCsJ2jBFqqSsvzm3pQPgVrApubNgiwE9EAaJKpl4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=iLMSvSP+hsDozBuQYXP0wdq5wiw3kG4EQ5otrQB8aIyQlyO6zoT1k6SafjV+cw3m6 6N86uX9mSqp40btGQ7PaFBulg+sdHQdvRqxi/qzE83Gn4kUCSHwTuPxa4TsyTaohgi hL6eeBOY4NfwFtxg7ZVW9LBRaQTdfSAdlL9UG+mEzbaH09meRm4OElxLde0UBfCX6f 0vwbcGo3QgvkMm+3htI4JgebrJWK54USFIGEMgqh5VW0TB3GHQZfrqmqez2/NI38nT d6p2h4G0QaVDJDGBbtZzMx6dSE5jYg7dSbmWVrRA2C6s+HNmYSgKUMEPbbwu7gVkeC blPlCqf+waU0g== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 17/20] lib/crypto: riscv/aes-ecb: Migrate optimized code into library Date: Sun, 20 Sep 2026 22:09:03 -0700 Message-ID: <20260921050910.296144-18-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of exposing the riscv-optimized AES-ECB code via a riscv-specific crypto_skcipher algorithm, just implement the AES-ECB library functions. This is simpler, it makes the AES-ECB library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-ECB code was disabled by default. AES-ECB support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it. To match what the library expects, update the assembly functions to operate on struct aes_enckey or struct aes_key rather than struct crypto_aes_ctx, and adjust the argument order. Bump up the priority of the corresponding library-based algorithm on riscv now that it no longer has to be lower than arch/riscv/crypto/. Signed-off-by: Eric Biggers --- arch/riscv/crypto/Kconfig | 4 +- arch/riscv/crypto/aes-riscv64-glue.c | 61 +------------------------- arch/riscv/crypto/aes-riscv64-zvkned.S | 39 ---------------- crypto/aes.c | 3 +- lib/crypto/riscv/aes-riscv64-zvkned.S | 51 +++++++++++++++++++++ lib/crypto/riscv/aes.h | 33 ++++++++++++++ 6 files changed, 89 insertions(+), 102 deletions(-) diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig index 6905232ddb03..84c41824b433 100644 --- a/arch/riscv/crypto/Kconfig +++ b/arch/riscv/crypto/Kconfig @@ -3,13 +3,13 @@ menu "Accelerated Cryptographic Algorithms for CPU (riscv)" =20 config CRYPTO_AES_RISCV64 - tristate "Ciphers: AES, modes: ECB, CBC, CTS, CTR, XTS" + tristate "Ciphers: AES, modes: CBC, CTS, CTR, XTS" depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \ RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS select CRYPTO_LIB_AES select CRYPTO_SKCIPHER help - Length-preserving ciphers: AES with ECB, CBC, CTS, CTR, XTS + Length-preserving ciphers: AES with CBC, CTS, CTR, XTS =20 Architecture: riscv64 using: - Zvkned vector crypto extension diff --git a/arch/riscv/crypto/aes-riscv64-glue.c b/arch/riscv/crypto/aes-r= iscv64-glue.c index bbd920c9e29d..f7c492dcfd57 100644 --- a/arch/riscv/crypto/aes-riscv64-glue.c +++ b/arch/riscv/crypto/aes-riscv64-glue.c @@ -22,11 +22,6 @@ #include #include =20 -asmlinkage void aes_ecb_encrypt_zvkned(const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len); -asmlinkage void aes_ecb_decrypt_zvkned(const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len); - asmlinkage void aes_cbc_encrypt_zvkned(const struct crypto_aes_ctx *key, const u8 *in, u8 *out, size_t len, u8 iv[AES_BLOCK_SIZE]); @@ -86,44 +81,6 @@ static int riscv64_aes_setkey_skcipher(struct crypto_skc= ipher *tfm, return riscv64_aes_setkey(ctx, key, keylen); } =20 -/* AES-ECB */ - -static inline int riscv64_aes_ecb_crypt(struct skcipher_request *req, bool= enc) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct crypto_aes_ctx *ctx =3D crypto_skcipher_ctx(tfm); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - while ((nbytes =3D walk.nbytes) !=3D 0) { - kernel_vector_begin(); - if (enc) - aes_ecb_encrypt_zvkned(ctx, walk.src.virt.addr, - walk.dst.virt.addr, - nbytes & ~(AES_BLOCK_SIZE - 1)); - else - aes_ecb_decrypt_zvkned(ctx, walk.src.virt.addr, - walk.dst.virt.addr, - nbytes & ~(AES_BLOCK_SIZE - 1)); - kernel_vector_end(); - err =3D skcipher_walk_done(&walk, nbytes & (AES_BLOCK_SIZE - 1)); - } - - return err; -} - -static int riscv64_aes_ecb_encrypt(struct skcipher_request *req) -{ - return riscv64_aes_ecb_crypt(req, true); -} - -static int riscv64_aes_ecb_decrypt(struct skcipher_request *req) -{ - return riscv64_aes_ecb_crypt(req, false); -} - /* AES-CBC */ =20 static int riscv64_aes_cbc_crypt(struct skcipher_request *req, bool enc) @@ -411,21 +368,6 @@ static int riscv64_aes_xts_decrypt(struct skcipher_req= uest *req) =20 static struct skcipher_alg riscv64_zvkned_aes_skcipher_algs[] =3D { { - .setkey =3D riscv64_aes_setkey_skcipher, - .encrypt =3D riscv64_aes_ecb_encrypt, - .decrypt =3D riscv64_aes_ecb_decrypt, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .walksize =3D 8 * AES_BLOCK_SIZE, /* matches LMUL=3D8 */ - .base =3D { - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D sizeof(struct crypto_aes_ctx), - .cra_priority =3D 300, - .cra_name =3D "ecb(aes)", - .cra_driver_name =3D "ecb-aes-riscv64-zvkned", - .cra_module =3D THIS_MODULE, - }, - }, { .setkey =3D riscv64_aes_setkey_skcipher, .encrypt =3D riscv64_aes_cbc_encrypt, .decrypt =3D riscv64_aes_cbc_decrypt, @@ -555,11 +497,10 @@ static void __exit riscv64_aes_mod_exit(void) module_init(riscv64_aes_mod_init); module_exit(riscv64_aes_mod_exit); =20 -MODULE_DESCRIPTION("AES-ECB/CBC/CTS/CTR/XTS (RISC-V accelerated)"); +MODULE_DESCRIPTION("AES-CBC/CTS/CTR/XTS (RISC-V accelerated)"); MODULE_AUTHOR("Jerry Shih "); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("aes"); -MODULE_ALIAS_CRYPTO("ecb(aes)"); MODULE_ALIAS_CRYPTO("cbc(aes)"); MODULE_ALIAS_CRYPTO("cts(cbc(aes))"); MODULE_ALIAS_CRYPTO("ctr(aes)"); diff --git a/arch/riscv/crypto/aes-riscv64-zvkned.S b/arch/riscv/crypto/aes= -riscv64-zvkned.S index d0fc4581a380..00f8a06596d3 100644 --- a/arch/riscv/crypto/aes-riscv64-zvkned.S +++ b/arch/riscv/crypto/aes-riscv64-zvkned.S @@ -56,45 +56,6 @@ #define LEN a3 #define IVP a4 =20 -.macro __aes_ecb_crypt enc, keylen - srli t0, LEN, 2 - // t0 is the remaining length in 32-bit words. It's a multiple of 4. -1: - vsetvli t1, t0, e32, m8, ta, ma - sub t0, t0, t1 // Subtract number of words processed - slli t1, t1, 2 // Words to bytes - vle32.v v16, (INP) - aes_crypt v16, \enc, \keylen - vse32.v v16, (OUTP) - add INP, INP, t1 - add OUTP, OUTP, t1 - bnez t0, 1b - - ret -.endm - -.macro aes_ecb_crypt enc - aes_begin KEYP, 128f, 192f - __aes_ecb_crypt \enc, 256 -128: - __aes_ecb_crypt \enc, 128 -192: - __aes_ecb_crypt \enc, 192 -.endm - -// void aes_ecb_encrypt_zvkned(const struct crypto_aes_ctx *key, -// const u8 *in, u8 *out, size_t len); -// -// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). -SYM_FUNC_START(aes_ecb_encrypt_zvkned) - aes_ecb_crypt 1 -SYM_FUNC_END(aes_ecb_encrypt_zvkned) - -// Same prototype and calling convention as the encryption function -SYM_FUNC_START(aes_ecb_decrypt_zvkned) - aes_ecb_crypt 0 -SYM_FUNC_END(aes_ecb_decrypt_zvkned) - .macro aes_cbc_encrypt keylen vle32.v v16, (IVP) // Load IV 1: diff --git a/crypto/aes.c b/crypto/aes.c index c19234f8a31c..0e72351d7f71 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -610,7 +610,8 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "ecb(aes)", .base.cra_driver_name =3D "ecb-aes-lib", - .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, + .base.cra_priority =3D (IS_ENABLED(CONFIG_RISCV) || + IS_ENABLED(CONFIG_X86)) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_key), .base.cra_module =3D THIS_MODULE, diff --git a/lib/crypto/riscv/aes-riscv64-zvkned.S b/lib/crypto/riscv/aes-r= iscv64-zvkned.S index 374fc4dba11b..b722bc90fd30 100644 --- a/lib/crypto/riscv/aes-riscv64-zvkned.S +++ b/lib/crypto/riscv/aes-riscv64-zvkned.S @@ -81,3 +81,54 @@ SYM_FUNC_END(aes_encrypt_zvkned) SYM_FUNC_START(aes_decrypt_zvkned) aes_crypt_zvkned 0 SYM_FUNC_END(aes_decrypt_zvkned) + +#undef KEYP +#undef OUTP +#undef INP + +#define DST a0 +#define SRC a1 +#define LEN a2 +#define KEYP a3 + +.macro __aes_ecb_crypt enc, keylen + srli t0, LEN, 2 + // t0 is the remaining length in 32-bit words. It's a multiple of 4. +1: + vsetvli t1, t0, e32, m8, ta, ma + sub t0, t0, t1 // Subtract number of words processed + slli t1, t1, 2 // Words to bytes + vle32.v v16, (SRC) + aes_crypt v16, \enc, \keylen + vse32.v v16, (DST) + add SRC, SRC, t1 + add DST, DST, t1 + bnez t0, 1b + + ret +.endm + +.macro aes_ecb_crypt enc + aes_begin KEYP, 128f, 192f + __aes_ecb_crypt \enc, 256 +128: + __aes_ecb_crypt \enc, 128 +192: + __aes_ecb_crypt \enc, 192 +.endm + +// void aes_ecb_encrypt_zvkned(u8 *dst, const u8 *src, size_t len, +// const struct aes_enckey *key); +// +// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). +SYM_FUNC_START(aes_ecb_encrypt_zvkned) + aes_ecb_crypt 1 +SYM_FUNC_END(aes_ecb_encrypt_zvkned) + +// void aes_ecb_decrypt_zvkned(u8 *dst, const u8 *src, size_t len, +// const struct aes_key *key); +// +// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). +SYM_FUNC_START(aes_ecb_decrypt_zvkned) + aes_ecb_crypt 0 +SYM_FUNC_END(aes_ecb_decrypt_zvkned) diff --git a/lib/crypto/riscv/aes.h b/lib/crypto/riscv/aes.h index a288b4c5b493..f97d27fa5985 100644 --- a/lib/crypto/riscv/aes.h +++ b/lib/crypto/riscv/aes.h @@ -58,6 +58,39 @@ static void aes_decrypt_arch(const struct aes_key *key, } } =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_ECB) +void aes_ecb_encrypt_zvkned(u8 *dst, const u8 *src, size_t len, + const struct aes_enckey *key); +void aes_ecb_decrypt_zvkned(u8 *dst, const u8 *src, size_t len, + const struct aes_key *key); + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_ecb_encrypt_arch aes_ecb_encrypt_arch +static bool aes_ecb_encrypt_arch(u8 *dst, const u8 *src, size_t len, + const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + aes_ecb_encrypt_zvkned(dst, src, len, key); + kernel_vector_end(); + return true; +} + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_ecb_decrypt_arch aes_ecb_decrypt_arch +static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src, size_t len, + const struct aes_key *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + aes_ecb_decrypt_zvkned(dst, src, len, key); + kernel_vector_end(); + return true; +} +#endif /* CONFIG_CRYPTO_LIB_AES_ECB */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6B9E36494C; Mon, 21 Sep 2026 05:16:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967777; cv=none; b=PG0K6FM5woASROhE1buCckCICctNiy3+ap9e9+s3zpNJj0i2Plpff117vUqZce+VW+FgQujlH5JRDSsgsMesmR0WCaDJ+OQtaUVpaWnVFY+MQkp5VySsgXSMKV3pCLc3KEleaVNfCI3LbB34vn1deJIYUuidYPRG1/8zdN2+JLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967777; c=relaxed/simple; bh=LQU0ekH1JkFOOm+844S0LD7VHQdw8tHjvTAO5tJ1RpE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TRTQ9fcmfUIw2Atxa+SPBE/pc1jln7pM4y0GxYUXi2I6kHOTHseNas7mMgPGZueBRLEcNyJfd8kR7EMgvLvmraAFqZ24p0xH4LouOY7Ii/BMVkpCu9DWnY9t/1vpOwUH/KmkyjmeDSOXSxqBRgRgJoHn6wVma0XspVR1OEwomgk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DY4nYMrc; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DY4nYMrc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D6ACD1F008A2; Mon, 21 Sep 2026 05:16:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967770; bh=H6wzP/STP9FTtgb0vk4MFaPsGNw4kiz4AaKKDaII6dM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=DY4nYMrc8KvtFnrdXTPDwitb/azt8GF5ztu1Y6wdldZjwLG7O3+GSJBIS3mR3NgS4 JiZOYQrhZKj/Z7JJFgGoRPoF4TmnK7jj3+15ilZi3MQzqnSdpfOVU8AndELuHlHi1J thggLVKdX7HhuWxFAmI0tT+NZjELPmwuGI7GS7PG05AtbUOp9f4YzGS4XiIFta/RBx xm+QtDj6qK9F9VgWNVIgYS1rPO6OzhPnv5NtbplYujaQU/SNbIIX5eYe+ApzsSc6Hw oeFR3O+S93F4PdEkdMAyZG0C6alFSg0HHiZa7LZ6BJ48bQnlTVHq0A1/8BZknlAGfK VSDiv/NeBFkvQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 18/20] lib/crypto: riscv/aes-cbc: Migrate optimized code into library Date: Sun, 20 Sep 2026 22:09:04 -0700 Message-ID: <20260921050910.296144-19-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Instead of exposing the riscv-optimized AES-CBC and AES-CBC-CTS code via a riscv-specific crypto_skcipher algorithm, just implement the AES-CBC and AES-CBC-CTS library functions. This is simpler, it makes the AES-CBC and AES-CBC-CTS library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-CBC and AES-CBC-CTS code was disabled by default. AES-CBC and AES-CBC-CTS support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it. To match what the library expects, update the assembly functions to operate on struct aes_enckey or struct aes_key rather than struct crypto_aes_ctx, and adjust the argument order. Bump up the priority of the corresponding library-based algorithms on riscv now that they no longer have to be lower than arch/riscv/crypto/. Signed-off-by: Eric Biggers --- arch/riscv/crypto/Kconfig | 4 +- arch/riscv/crypto/Makefile | 2 +- arch/riscv/crypto/aes-riscv64-glue.c | 167 +-------------- arch/riscv/crypto/aes-riscv64-zvkned.S | 273 ------------------------- crypto/aes.c | 6 +- lib/crypto/riscv/aes-riscv64-zvkned.S | 245 ++++++++++++++++++++++ lib/crypto/riscv/aes.h | 68 ++++++ 7 files changed, 322 insertions(+), 443 deletions(-) delete mode 100644 arch/riscv/crypto/aes-riscv64-zvkned.S diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig index 84c41824b433..0a3f87ad384e 100644 --- a/arch/riscv/crypto/Kconfig +++ b/arch/riscv/crypto/Kconfig @@ -3,13 +3,13 @@ menu "Accelerated Cryptographic Algorithms for CPU (riscv)" =20 config CRYPTO_AES_RISCV64 - tristate "Ciphers: AES, modes: CBC, CTS, CTR, XTS" + tristate "Ciphers: AES, modes: CTR, XTS" depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \ RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS select CRYPTO_LIB_AES select CRYPTO_SKCIPHER help - Length-preserving ciphers: AES with CBC, CTS, CTR, XTS + Length-preserving ciphers: AES with CTR, XTS =20 Architecture: riscv64 using: - Zvkned vector crypto extension diff --git a/arch/riscv/crypto/Makefile b/arch/riscv/crypto/Makefile index 8cf31db57fc4..d8b85afa6d0b 100644 --- a/arch/riscv/crypto/Makefile +++ b/arch/riscv/crypto/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only =20 obj-$(CONFIG_CRYPTO_AES_RISCV64) +=3D aes-riscv64.o -aes-riscv64-y :=3D aes-riscv64-glue.o aes-riscv64-zvkned.o \ +aes-riscv64-y :=3D aes-riscv64-glue.o \ aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o =20 obj-$(CONFIG_CRYPTO_SM4_RISCV64) +=3D sm4-riscv64.o diff --git a/arch/riscv/crypto/aes-riscv64-glue.c b/arch/riscv/crypto/aes-r= iscv64-glue.c index f7c492dcfd57..97f5369d7e71 100644 --- a/arch/riscv/crypto/aes-riscv64-glue.c +++ b/arch/riscv/crypto/aes-riscv64-glue.c @@ -22,17 +22,6 @@ #include #include =20 -asmlinkage void aes_cbc_encrypt_zvkned(const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len, - u8 iv[AES_BLOCK_SIZE]); -asmlinkage void aes_cbc_decrypt_zvkned(const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len, - u8 iv[AES_BLOCK_SIZE]); - -asmlinkage void aes_cbc_cts_crypt_zvkned(const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len, - const u8 iv[AES_BLOCK_SIZE], bool enc); - asmlinkage void aes_ctr32_crypt_zvkned_zvkb(const struct crypto_aes_ctx *k= ey, const u8 *in, u8 *out, size_t len, u8 iv[AES_BLOCK_SIZE]); @@ -81,110 +70,6 @@ static int riscv64_aes_setkey_skcipher(struct crypto_sk= cipher *tfm, return riscv64_aes_setkey(ctx, key, keylen); } =20 -/* AES-CBC */ - -static int riscv64_aes_cbc_crypt(struct skcipher_request *req, bool enc) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct crypto_aes_ctx *ctx =3D crypto_skcipher_ctx(tfm); - struct skcipher_walk walk; - unsigned int nbytes; - int err; - - err =3D skcipher_walk_virt(&walk, req, false); - while ((nbytes =3D walk.nbytes) !=3D 0) { - kernel_vector_begin(); - if (enc) - aes_cbc_encrypt_zvkned(ctx, walk.src.virt.addr, - walk.dst.virt.addr, - nbytes & ~(AES_BLOCK_SIZE - 1), - walk.iv); - else - aes_cbc_decrypt_zvkned(ctx, walk.src.virt.addr, - walk.dst.virt.addr, - nbytes & ~(AES_BLOCK_SIZE - 1), - walk.iv); - kernel_vector_end(); - err =3D skcipher_walk_done(&walk, nbytes & (AES_BLOCK_SIZE - 1)); - } - - return err; -} - -static int riscv64_aes_cbc_encrypt(struct skcipher_request *req) -{ - return riscv64_aes_cbc_crypt(req, true); -} - -static int riscv64_aes_cbc_decrypt(struct skcipher_request *req) -{ - return riscv64_aes_cbc_crypt(req, false); -} - -/* AES-CBC-CTS */ - -static int riscv64_aes_cbc_cts_crypt(struct skcipher_request *req, bool en= c) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct crypto_aes_ctx *ctx =3D crypto_skcipher_ctx(tfm); - struct scatterlist sg_src[2], sg_dst[2]; - struct skcipher_request subreq; - struct scatterlist *src, *dst; - struct skcipher_walk walk; - unsigned int cbc_len; - int err; - - if (req->cryptlen < AES_BLOCK_SIZE) - return -EINVAL; - - err =3D skcipher_walk_virt(&walk, req, false); - if (err) - return err; - /* - * If the full message is available in one step, decrypt it in one call - * to the CBC-CTS assembly function. This reduces overhead, especially - * on short messages. Otherwise, fall back to doing CBC up to the last - * two blocks, then invoke CTS just for the ciphertext stealing. - */ - if (unlikely(walk.nbytes !=3D req->cryptlen)) { - cbc_len =3D round_down(req->cryptlen - AES_BLOCK_SIZE - 1, - AES_BLOCK_SIZE); - skcipher_walk_abort(&walk); - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, - skcipher_request_flags(req), - NULL, NULL); - skcipher_request_set_crypt(&subreq, req->src, req->dst, - cbc_len, req->iv); - err =3D riscv64_aes_cbc_crypt(&subreq, enc); - if (err) - return err; - dst =3D src =3D scatterwalk_ffwd(sg_src, req->src, cbc_len); - if (req->dst !=3D req->src) - dst =3D scatterwalk_ffwd(sg_dst, req->dst, cbc_len); - skcipher_request_set_crypt(&subreq, src, dst, - req->cryptlen - cbc_len, req->iv); - err =3D skcipher_walk_virt(&walk, &subreq, false); - if (err) - return err; - } - kernel_vector_begin(); - aes_cbc_cts_crypt_zvkned(ctx, walk.src.virt.addr, walk.dst.virt.addr, - walk.nbytes, req->iv, enc); - kernel_vector_end(); - return skcipher_walk_done(&walk, 0); -} - -static int riscv64_aes_cbc_cts_encrypt(struct skcipher_request *req) -{ - return riscv64_aes_cbc_cts_crypt(req, true); -} - -static int riscv64_aes_cbc_cts_decrypt(struct skcipher_request *req) -{ - return riscv64_aes_cbc_cts_crypt(req, false); -} - /* AES-CTR */ =20 static int riscv64_aes_ctr_crypt(struct skcipher_request *req) @@ -366,41 +251,6 @@ static int riscv64_aes_xts_decrypt(struct skcipher_req= uest *req) =20 /* Algorithm definitions */ =20 -static struct skcipher_alg riscv64_zvkned_aes_skcipher_algs[] =3D { - { - .setkey =3D riscv64_aes_setkey_skcipher, - .encrypt =3D riscv64_aes_cbc_encrypt, - .decrypt =3D riscv64_aes_cbc_decrypt, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .base =3D { - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D sizeof(struct crypto_aes_ctx), - .cra_priority =3D 300, - .cra_name =3D "cbc(aes)", - .cra_driver_name =3D "cbc-aes-riscv64-zvkned", - .cra_module =3D THIS_MODULE, - }, - }, { - .setkey =3D riscv64_aes_setkey_skcipher, - .encrypt =3D riscv64_aes_cbc_cts_encrypt, - .decrypt =3D riscv64_aes_cbc_cts_decrypt, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .walksize =3D 4 * AES_BLOCK_SIZE, /* matches LMUL=3D4 */ - .base =3D { - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D sizeof(struct crypto_aes_ctx), - .cra_priority =3D 300, - .cra_name =3D "cts(cbc(aes))", - .cra_driver_name =3D "cts-cbc-aes-riscv64-zvkned", - .cra_module =3D THIS_MODULE, - }, - } -}; - static struct skcipher_alg riscv64_zvkned_zvkb_aes_skcipher_alg =3D { .setkey =3D riscv64_aes_setkey_skcipher, .encrypt =3D riscv64_aes_ctr_crypt, @@ -452,17 +302,11 @@ static int __init riscv64_aes_mod_init(void) =20 if (riscv_isa_extension_available(NULL, ZVKNED) && riscv_vector_vlen() >=3D 128) { - err =3D crypto_register_skciphers( - riscv64_zvkned_aes_skcipher_algs, - ARRAY_SIZE(riscv64_zvkned_aes_skcipher_algs)); - if (err) - return err; - if (riscv_isa_extension_available(NULL, ZVKB)) { err =3D crypto_register_skcipher( &riscv64_zvkned_zvkb_aes_skcipher_alg); if (err) - goto unregister_zvkned_skcipher_algs; + return err; } =20 if (riscv64_aes_xts_supported()) { @@ -478,9 +322,6 @@ static int __init riscv64_aes_mod_init(void) unregister_zvkned_zvkb_skcipher_alg: if (riscv_isa_extension_available(NULL, ZVKB)) crypto_unregister_skcipher(&riscv64_zvkned_zvkb_aes_skcipher_alg); -unregister_zvkned_skcipher_algs: - crypto_unregister_skciphers(riscv64_zvkned_aes_skcipher_algs, - ARRAY_SIZE(riscv64_zvkned_aes_skcipher_algs)); return err; } =20 @@ -490,18 +331,14 @@ static void __exit riscv64_aes_mod_exit(void) crypto_unregister_skcipher(&riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg); if (riscv_isa_extension_available(NULL, ZVKB)) crypto_unregister_skcipher(&riscv64_zvkned_zvkb_aes_skcipher_alg); - crypto_unregister_skciphers(riscv64_zvkned_aes_skcipher_algs, - ARRAY_SIZE(riscv64_zvkned_aes_skcipher_algs)); } =20 module_init(riscv64_aes_mod_init); module_exit(riscv64_aes_mod_exit); =20 -MODULE_DESCRIPTION("AES-CBC/CTS/CTR/XTS (RISC-V accelerated)"); +MODULE_DESCRIPTION("AES-CTR/XTS (RISC-V accelerated)"); MODULE_AUTHOR("Jerry Shih "); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("aes"); -MODULE_ALIAS_CRYPTO("cbc(aes)"); -MODULE_ALIAS_CRYPTO("cts(cbc(aes))"); MODULE_ALIAS_CRYPTO("ctr(aes)"); MODULE_ALIAS_CRYPTO("xts(aes)"); diff --git a/arch/riscv/crypto/aes-riscv64-zvkned.S b/arch/riscv/crypto/aes= -riscv64-zvkned.S deleted file mode 100644 index 00f8a06596d3..000000000000 --- a/arch/riscv/crypto/aes-riscv64-zvkned.S +++ /dev/null @@ -1,273 +0,0 @@ -/* SPDX-License-Identifier: Apache-2.0 OR BSD-2-Clause */ -// -// This file is dual-licensed, meaning that you can use it under your -// choice of either of the following two licenses: -// -// Copyright 2023 The OpenSSL Project Authors. All Rights Reserved. -// -// Licensed under the Apache License 2.0 (the "License"). You can obtain -// a copy in the file LICENSE in the source distribution or at -// https://www.openssl.org/source/license.html -// -// or -// -// Copyright (c) 2023, Christoph M=C3=BCllner -// Copyright (c) 2023, Phoebe Chen -// Copyright (c) 2023, Jerry Shih -// Copyright 2024 Google LLC -// All rights reserved. -// -// Redistribution and use in source and binary forms, with or without -// modification, are permitted provided that the following conditions -// are met: -// 1. Redistributions of source code must retain the above copyright -// notice, this list of conditions and the following disclaimer. -// 2. Redistributions in binary form must reproduce the above copyright -// notice, this list of conditions and the following disclaimer in the -// documentation and/or other materials provided with the distribution. -// -// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -// The generated code of this file depends on the following RISC-V extensi= ons: -// - RV64I -// - RISC-V Vector ('V') with VLEN >=3D 128 -// - RISC-V Vector AES block cipher extension ('Zvkned') - -#include - -.text -.option arch, +zvkned - -#include "aes-macros.S" - -#define KEYP a0 -#define INP a1 -#define OUTP a2 -#define LEN a3 -#define IVP a4 - -.macro aes_cbc_encrypt keylen - vle32.v v16, (IVP) // Load IV -1: - vle32.v v17, (INP) // Load plaintext block - vxor.vv v16, v16, v17 // XOR with IV or prev ciphertext block - aes_encrypt v16, \keylen // Encrypt - vse32.v v16, (OUTP) // Store ciphertext block - addi INP, INP, 16 - addi OUTP, OUTP, 16 - addi LEN, LEN, -16 - bnez LEN, 1b - - vse32.v v16, (IVP) // Store next IV - ret -.endm - -.macro aes_cbc_decrypt keylen - srli LEN, LEN, 2 // Convert LEN from bytes to words - vle32.v v16, (IVP) // Load IV -1: - vsetvli t0, LEN, e32, m4, ta, ma - vle32.v v20, (INP) // Load ciphertext blocks - vslideup.vi v16, v20, 4 // Setup prev ciphertext blocks - addi t1, t0, -4 - vslidedown.vx v24, v20, t1 // Save last ciphertext block - aes_decrypt v20, \keylen // Decrypt the blocks - vxor.vv v20, v20, v16 // XOR with prev ciphertext blocks - vse32.v v20, (OUTP) // Store plaintext blocks - vmv.v.v v16, v24 // Next "IV" is last ciphertext block - slli t1, t0, 2 // Words to bytes - add INP, INP, t1 - add OUTP, OUTP, t1 - sub LEN, LEN, t0 - bnez LEN, 1b - - vsetivli zero, 4, e32, m1, ta, ma - vse32.v v16, (IVP) // Store next IV - ret -.endm - -// void aes_cbc_encrypt_zvkned(const struct crypto_aes_ctx *key, -// const u8 *in, u8 *out, size_t len, u8 iv[16]); -// -// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). -SYM_FUNC_START(aes_cbc_encrypt_zvkned) - aes_begin KEYP, 128f, 192f - aes_cbc_encrypt 256 -128: - aes_cbc_encrypt 128 -192: - aes_cbc_encrypt 192 -SYM_FUNC_END(aes_cbc_encrypt_zvkned) - -// Same prototype and calling convention as the encryption function -SYM_FUNC_START(aes_cbc_decrypt_zvkned) - aes_begin KEYP, 128f, 192f - aes_cbc_decrypt 256 -128: - aes_cbc_decrypt 128 -192: - aes_cbc_decrypt 192 -SYM_FUNC_END(aes_cbc_decrypt_zvkned) - -.macro aes_cbc_cts_encrypt keylen - - // CBC-encrypt all blocks except the last. But don't store the - // second-to-last block to the output buffer yet, since it will be - // handled specially in the ciphertext stealing step. Exception: if the - // message is single-block, still encrypt the last (and only) block. - li t0, 16 - j 2f -1: - vse32.v v16, (OUTP) // Store ciphertext block - addi OUTP, OUTP, 16 -2: - vle32.v v17, (INP) // Load plaintext block - vxor.vv v16, v16, v17 // XOR with IV or prev ciphertext block - aes_encrypt v16, \keylen // Encrypt - addi INP, INP, 16 - addi LEN, LEN, -16 - bgt LEN, t0, 1b // Repeat if more than one block remains - - // Special case: if the message is a single block, just do CBC. - beqz LEN, .Lcts_encrypt_done\@ - - // Encrypt the last two blocks using ciphertext stealing as follows: - // C[n-1] =3D Encrypt(Encrypt(P[n-1] ^ C[n-2]) ^ P[n]) - // C[n] =3D Encrypt(P[n-1] ^ C[n-2])[0..LEN] - // - // C[i] denotes the i'th ciphertext block, and likewise P[i] the i'th - // plaintext block. Block n, the last block, may be partial; its length - // is 1 <=3D LEN <=3D 16. If there are only 2 blocks, C[n-2] means the I= V. - // - // v16 already contains Encrypt(P[n-1] ^ C[n-2]). - // INP points to P[n]. OUTP points to where C[n-1] should go. - // To support in-place encryption, load P[n] before storing C[n]. - addi t0, OUTP, 16 // Get pointer to where C[n] should go - vsetvli zero, LEN, e8, m1, tu, ma - vle8.v v17, (INP) // Load P[n] - vse8.v v16, (t0) // Store C[n] - vxor.vv v16, v16, v17 // v16 =3D Encrypt(P[n-1] ^ C[n-2]) ^ P[n] - vsetivli zero, 4, e32, m1, ta, ma - aes_encrypt v16, \keylen -.Lcts_encrypt_done\@: - vse32.v v16, (OUTP) // Store C[n-1] (or C[n] in single-block case) - ret -.endm - -#define LEN32 t4 // Length of remaining full blocks in 32-bit words -#define LEN_MOD16 t5 // Length of message in bytes mod 16 - -.macro aes_cbc_cts_decrypt keylen - andi LEN32, LEN, ~15 - srli LEN32, LEN32, 2 - andi LEN_MOD16, LEN, 15 - - // Save C[n-2] in v28 so that it's available later during the ciphertext - // stealing step. If there are fewer than three blocks, C[n-2] means - // the IV, otherwise it means the third-to-last ciphertext block. - vmv.v.v v28, v16 // IV - add t0, LEN, -33 - bltz t0, .Lcts_decrypt_loop\@ - andi t0, t0, ~15 - add t0, t0, INP - vle32.v v28, (t0) - - // CBC-decrypt all full blocks. For the last full block, or the last 2 - // full blocks if the message is block-aligned, this doesn't write the - // correct output blocks (unless the message is only a single block), - // because it XORs the wrong values with the raw AES plaintexts. But we - // fix this after this loop without redoing the AES decryptions. This - // approach allows more of the AES decryptions to be parallelized. -.Lcts_decrypt_loop\@: - vsetvli t0, LEN32, e32, m4, ta, ma - addi t1, t0, -4 - vle32.v v20, (INP) // Load next set of ciphertext blocks - vmv.v.v v24, v16 // Get IV or last ciphertext block of prev set - vslideup.vi v24, v20, 4 // Setup prev ciphertext blocks - vslidedown.vx v16, v20, t1 // Save last ciphertext block of this set - aes_decrypt v20, \keylen // Decrypt this set of blocks - vxor.vv v24, v24, v20 // XOR prev ciphertext blocks with decrypted blocks - vse32.v v24, (OUTP) // Store this set of plaintext blocks - sub LEN32, LEN32, t0 - slli t0, t0, 2 // Words to bytes - add INP, INP, t0 - add OUTP, OUTP, t0 - bnez LEN32, .Lcts_decrypt_loop\@ - - vsetivli zero, 4, e32, m4, ta, ma - vslidedown.vx v20, v20, t1 // Extract raw plaintext of last full block - addi t0, OUTP, -16 // Get pointer to last full plaintext block - bnez LEN_MOD16, .Lcts_decrypt_non_block_aligned\@ - - // Special case: if the message is a single block, just do CBC. - li t1, 16 - beq LEN, t1, .Lcts_decrypt_done\@ - - // Block-aligned message. Just fix up the last 2 blocks. We need: - // - // P[n-1] =3D Decrypt(C[n]) ^ C[n-2] - // P[n] =3D Decrypt(C[n-1]) ^ C[n] - // - // We have C[n] in v16, Decrypt(C[n]) in v20, and C[n-2] in v28. - // Together with Decrypt(C[n-1]) ^ C[n-2] from the output buffer, this - // is everything needed to fix the output without re-decrypting blocks. - addi t1, OUTP, -32 // Get pointer to where P[n-1] should go - vxor.vv v20, v20, v28 // Decrypt(C[n]) ^ C[n-2] =3D=3D P[n-1] - vle32.v v24, (t1) // Decrypt(C[n-1]) ^ C[n-2] - vse32.v v20, (t1) // Store P[n-1] - vxor.vv v20, v24, v16 // Decrypt(C[n-1]) ^ C[n-2] ^ C[n] =3D=3D P[n] ^ C= [n-2] - j .Lcts_decrypt_finish\@ - -.Lcts_decrypt_non_block_aligned\@: - // Decrypt the last two blocks using ciphertext stealing as follows: - // - // P[n-1] =3D Decrypt(C[n] || Decrypt(C[n-1])[LEN_MOD16..16]) ^ C[n-2] - // P[n] =3D (Decrypt(C[n-1]) ^ C[n])[0..LEN_MOD16] - // - // We already have Decrypt(C[n-1]) in v20 and C[n-2] in v28. - vmv.v.v v16, v20 // v16 =3D Decrypt(C[n-1]) - vsetvli zero, LEN_MOD16, e8, m1, tu, ma - vle8.v v20, (INP) // v20 =3D C[n] || Decrypt(C[n-1])[LEN_MOD16..16] - vxor.vv v16, v16, v20 // v16 =3D Decrypt(C[n-1]) ^ C[n] - vse8.v v16, (OUTP) // Store P[n] - vsetivli zero, 4, e32, m1, ta, ma - aes_decrypt v20, \keylen // v20 =3D Decrypt(C[n] || Decrypt(C[n-1])[LEN_M= OD16..16]) -.Lcts_decrypt_finish\@: - vxor.vv v20, v20, v28 // XOR with C[n-2] - vse32.v v20, (t0) // Store last full plaintext block -.Lcts_decrypt_done\@: - ret -.endm - -.macro aes_cbc_cts_crypt keylen - vle32.v v16, (IVP) // Load IV - beqz a5, .Lcts_decrypt\@ - aes_cbc_cts_encrypt \keylen -.Lcts_decrypt\@: - aes_cbc_cts_decrypt \keylen -.endm - -// void aes_cbc_cts_crypt_zvkned(const struct crypto_aes_ctx *key, -// const u8 *in, u8 *out, size_t len, -// const u8 iv[16], bool enc); -// -// Encrypts or decrypts a message with the CS3 variant of AES-CBC-CTS. -// This is the variant that unconditionally swaps the last two blocks. -SYM_FUNC_START(aes_cbc_cts_crypt_zvkned) - aes_begin KEYP, 128f, 192f - aes_cbc_cts_crypt 256 -128: - aes_cbc_cts_crypt 128 -192: - aes_cbc_cts_crypt 192 -SYM_FUNC_END(aes_cbc_cts_crypt_zvkned) diff --git a/crypto/aes.c b/crypto/aes.c index 0e72351d7f71..e951f0e1fe5a 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -626,7 +626,8 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "cbc(aes)", .base.cra_driver_name =3D "cbc-aes-lib", - .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, + .base.cra_priority =3D (IS_ENABLED(CONFIG_RISCV) || + IS_ENABLED(CONFIG_X86)) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_key), .base.cra_module =3D THIS_MODULE, @@ -652,7 +653,8 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "cts(cbc(aes))", .base.cra_driver_name =3D "cts-cbc-aes-lib", - .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, + .base.cra_priority =3D (IS_ENABLED(CONFIG_RISCV) || + IS_ENABLED(CONFIG_X86)) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_key), .base.cra_module =3D THIS_MODULE, diff --git a/lib/crypto/riscv/aes-riscv64-zvkned.S b/lib/crypto/riscv/aes-r= iscv64-zvkned.S index b722bc90fd30..4a341cb83bda 100644 --- a/lib/crypto/riscv/aes-riscv64-zvkned.S +++ b/lib/crypto/riscv/aes-riscv64-zvkned.S @@ -132,3 +132,248 @@ SYM_FUNC_END(aes_ecb_encrypt_zvkned) SYM_FUNC_START(aes_ecb_decrypt_zvkned) aes_ecb_crypt 0 SYM_FUNC_END(aes_ecb_decrypt_zvkned) + +#undef DST +#undef SRC +#undef LEN +#undef KEYP + +#define DST a0 +#define SRC a1 +#define LEN a2 +#define IVP a3 +#define KEYP a4 + +.macro aes_cbc_encrypt keylen + vle32.v v16, (IVP) // Load IV +1: + vle32.v v17, (SRC) // Load plaintext block + vxor.vv v16, v16, v17 // XOR with IV or prev ciphertext block + aes_encrypt v16, \keylen // Encrypt + vse32.v v16, (DST) // Store ciphertext block + addi SRC, SRC, 16 + addi DST, DST, 16 + addi LEN, LEN, -16 + bnez LEN, 1b + + vse32.v v16, (IVP) // Store next IV + ret +.endm + +.macro aes_cbc_decrypt keylen + srli LEN, LEN, 2 // Convert LEN from bytes to words + vle32.v v16, (IVP) // Load IV +1: + vsetvli t0, LEN, e32, m4, ta, ma + vle32.v v20, (SRC) // Load ciphertext blocks + vslideup.vi v16, v20, 4 // Setup prev ciphertext blocks + addi t1, t0, -4 + vslidedown.vx v24, v20, t1 // Save last ciphertext block + aes_decrypt v20, \keylen // Decrypt the blocks + vxor.vv v20, v20, v16 // XOR with prev ciphertext blocks + vse32.v v20, (DST) // Store plaintext blocks + vmv.v.v v16, v24 // Next "IV" is last ciphertext block + slli t1, t0, 2 // Words to bytes + add SRC, SRC, t1 + add DST, DST, t1 + sub LEN, LEN, t0 + bnez LEN, 1b + + vsetivli zero, 4, e32, m1, ta, ma + vse32.v v16, (IVP) // Store next IV + ret +.endm + +// void aes_cbc_encrypt_zvkned(u8 *dst, const u8 *src, size_t len, +// u8 iv[AES_BLOCK_SIZE], +// const struct aes_enckey *key); +// +// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). +SYM_FUNC_START(aes_cbc_encrypt_zvkned) + aes_begin KEYP, 128f, 192f + aes_cbc_encrypt 256 +128: + aes_cbc_encrypt 128 +192: + aes_cbc_encrypt 192 +SYM_FUNC_END(aes_cbc_encrypt_zvkned) + +// void aes_cbc_decrypt_zvkned(u8 *dst, const u8 *src, size_t len, +// u8 iv[AES_BLOCK_SIZE], +// const struct aes_key *key); +// +// |len| must be nonzero and a multiple of 16 (AES_BLOCK_SIZE). +SYM_FUNC_START(aes_cbc_decrypt_zvkned) + aes_begin KEYP, 128f, 192f + aes_cbc_decrypt 256 +128: + aes_cbc_decrypt 128 +192: + aes_cbc_decrypt 192 +SYM_FUNC_END(aes_cbc_decrypt_zvkned) + +#undef DST +#undef SRC +#undef LEN +#undef IVP +#undef KEYP + +#define DST a0 +#define SRC a1 +#define LEN a2 +#define IVP a3 +#define KEYP a4 +#define ENC a5 + +.macro aes_cbc_cts_encrypt keylen + + // CBC-encrypt all blocks except the last. But don't store the + // second-to-last block to the output buffer yet, since it will be + // handled specially in the ciphertext stealing step. Exception: if the + // message is single-block, still encrypt the last (and only) block. + li t0, 16 + j 2f +1: + vse32.v v16, (DST) // Store ciphertext block + addi DST, DST, 16 +2: + vle32.v v17, (SRC) // Load plaintext block + vxor.vv v16, v16, v17 // XOR with IV or prev ciphertext block + aes_encrypt v16, \keylen // Encrypt + addi SRC, SRC, 16 + addi LEN, LEN, -16 + bgt LEN, t0, 1b // Repeat if more than one block remains + + // Special case: if the message is a single block, just do CBC. + beqz LEN, .Lcts_encrypt_done\@ + + // Encrypt the last two blocks using ciphertext stealing as follows: + // C[n-1] =3D Encrypt(Encrypt(P[n-1] ^ C[n-2]) ^ P[n]) + // C[n] =3D Encrypt(P[n-1] ^ C[n-2])[0..LEN] + // + // C[i] denotes the i'th ciphertext block, and likewise P[i] the i'th + // plaintext block. Block n, the last block, may be partial; its length + // is 1 <=3D LEN <=3D 16. If there are only 2 blocks, C[n-2] means the I= V. + // + // v16 already contains Encrypt(P[n-1] ^ C[n-2]). + // SRC points to P[n]. DST points to where C[n-1] should go. + // To support in-place encryption, load P[n] before storing C[n]. + addi t0, DST, 16 // Get pointer to where C[n] should go + vsetvli zero, LEN, e8, m1, tu, ma + vle8.v v17, (SRC) // Load P[n] + vse8.v v16, (t0) // Store C[n] + vxor.vv v16, v16, v17 // v16 =3D Encrypt(P[n-1] ^ C[n-2]) ^ P[n] + vsetivli zero, 4, e32, m1, ta, ma + aes_encrypt v16, \keylen +.Lcts_encrypt_done\@: + vse32.v v16, (DST) // Store C[n-1] (or C[n] in single-block case) + ret +.endm + +#define LEN32 t4 // Length of remaining full blocks in 32-bit words +#define LEN_MOD16 t5 // Length of message in bytes mod 16 + +.macro aes_cbc_cts_decrypt keylen + andi LEN32, LEN, ~15 + srli LEN32, LEN32, 2 + andi LEN_MOD16, LEN, 15 + + // Save C[n-2] in v28 so that it's available later during the ciphertext + // stealing step. If there are fewer than three blocks, C[n-2] means + // the IV, otherwise it means the third-to-last ciphertext block. + vmv.v.v v28, v16 // IV + add t0, LEN, -33 + bltz t0, .Lcts_decrypt_loop\@ + andi t0, t0, ~15 + add t0, t0, SRC + vle32.v v28, (t0) + + // CBC-decrypt all full blocks. For the last full block, or the last 2 + // full blocks if the message is block-aligned, this doesn't write the + // correct output blocks (unless the message is only a single block), + // because it XORs the wrong values with the raw AES plaintexts. But we + // fix this after this loop without redoing the AES decryptions. This + // approach allows more of the AES decryptions to be parallelized. +.Lcts_decrypt_loop\@: + vsetvli t0, LEN32, e32, m4, ta, ma + addi t1, t0, -4 + vle32.v v20, (SRC) // Load next set of ciphertext blocks + vmv.v.v v24, v16 // Get IV or last ciphertext block of prev set + vslideup.vi v24, v20, 4 // Setup prev ciphertext blocks + vslidedown.vx v16, v20, t1 // Save last ciphertext block of this set + aes_decrypt v20, \keylen // Decrypt this set of blocks + vxor.vv v24, v24, v20 // XOR prev ciphertext blocks with decrypted blocks + vse32.v v24, (DST) // Store this set of plaintext blocks + sub LEN32, LEN32, t0 + slli t0, t0, 2 // Words to bytes + add SRC, SRC, t0 + add DST, DST, t0 + bnez LEN32, .Lcts_decrypt_loop\@ + + vsetivli zero, 4, e32, m4, ta, ma + vslidedown.vx v20, v20, t1 // Extract raw plaintext of last full block + addi t0, DST, -16 // Get pointer to last full plaintext block + bnez LEN_MOD16, .Lcts_decrypt_non_block_aligned\@ + + // Special case: if the message is a single block, just do CBC. + li t1, 16 + beq LEN, t1, .Lcts_decrypt_done\@ + + // Block-aligned message. Just fix up the last 2 blocks. We need: + // + // P[n-1] =3D Decrypt(C[n]) ^ C[n-2] + // P[n] =3D Decrypt(C[n-1]) ^ C[n] + // + // We have C[n] in v16, Decrypt(C[n]) in v20, and C[n-2] in v28. + // Together with Decrypt(C[n-1]) ^ C[n-2] from the output buffer, this + // is everything needed to fix the output without re-decrypting blocks. + addi t1, DST, -32 // Get pointer to where P[n-1] should go + vxor.vv v20, v20, v28 // Decrypt(C[n]) ^ C[n-2] =3D=3D P[n-1] + vle32.v v24, (t1) // Decrypt(C[n-1]) ^ C[n-2] + vse32.v v20, (t1) // Store P[n-1] + vxor.vv v20, v24, v16 // Decrypt(C[n-1]) ^ C[n-2] ^ C[n] =3D=3D P[n] ^ C= [n-2] + j .Lcts_decrypt_finish\@ + +.Lcts_decrypt_non_block_aligned\@: + // Decrypt the last two blocks using ciphertext stealing as follows: + // + // P[n-1] =3D Decrypt(C[n] || Decrypt(C[n-1])[LEN_MOD16..16]) ^ C[n-2] + // P[n] =3D (Decrypt(C[n-1]) ^ C[n])[0..LEN_MOD16] + // + // We already have Decrypt(C[n-1]) in v20 and C[n-2] in v28. + vmv.v.v v16, v20 // v16 =3D Decrypt(C[n-1]) + vsetvli zero, LEN_MOD16, e8, m1, tu, ma + vle8.v v20, (SRC) // v20 =3D C[n] || Decrypt(C[n-1])[LEN_MOD16..16] + vxor.vv v16, v16, v20 // v16 =3D Decrypt(C[n-1]) ^ C[n] + vse8.v v16, (DST) // Store P[n] + vsetivli zero, 4, e32, m1, ta, ma + aes_decrypt v20, \keylen // v20 =3D Decrypt(C[n] || Decrypt(C[n-1])[LEN_M= OD16..16]) +.Lcts_decrypt_finish\@: + vxor.vv v20, v20, v28 // XOR with C[n-2] + vse32.v v20, (t0) // Store last full plaintext block +.Lcts_decrypt_done\@: + ret +.endm + +.macro aes_cbc_cts_crypt keylen + vle32.v v16, (IVP) // Load IV + beqz ENC, .Lcts_decrypt\@ + aes_cbc_cts_encrypt \keylen +.Lcts_decrypt\@: + aes_cbc_cts_decrypt \keylen +.endm + +// void aes_cbc_cts_crypt_zvkned(u8 *dst, const u8 *src, size_t len, +// const u8 iv[AES_BLOCK_SIZE], +// aes_encrypt_arg key, bool enc); +// +// Encrypts or decrypts a message with the CS3 variant of AES-CBC-CTS. +// This is the variant that unconditionally swaps the last two blocks. +SYM_FUNC_START(aes_cbc_cts_crypt_zvkned) + aes_begin KEYP, 128f, 192f + aes_cbc_cts_crypt 256 +128: + aes_cbc_cts_crypt 128 +192: + aes_cbc_cts_crypt 192 +SYM_FUNC_END(aes_cbc_cts_crypt_zvkned) diff --git a/lib/crypto/riscv/aes.h b/lib/crypto/riscv/aes.h index f97d27fa5985..e02f9343d67d 100644 --- a/lib/crypto/riscv/aes.h +++ b/lib/crypto/riscv/aes.h @@ -91,6 +91,74 @@ static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src,= size_t len, } #endif /* CONFIG_CRYPTO_LIB_AES_ECB */ =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CBC) +void aes_cbc_encrypt_zvkned(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], const struct aes_enckey *key); +void aes_cbc_decrypt_zvkned(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], const struct aes_key *key); +void aes_cbc_cts_crypt_zvkned(u8 *dst, const u8 *src, size_t len, + const u8 iv[AES_BLOCK_SIZE], + aes_encrypt_arg key, bool enc); + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_cbc_encrypt_arch aes_cbc_encrypt_arch +static bool aes_cbc_encrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + aes_cbc_encrypt_zvkned(dst, src, len, iv, key); + kernel_vector_end(); + return true; +} + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +#define aes_cbc_decrypt_arch aes_cbc_decrypt_arch +static bool aes_cbc_decrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_key *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + aes_cbc_decrypt_zvkned(dst, src, len, iv, key); + kernel_vector_end(); + return true; +} + +/* len can be any value greater than AES_BLOCK_SIZE here. */ +#define aes_cbc_cts_encrypt_arch aes_cbc_cts_encrypt_arch +static bool aes_cbc_cts_encrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + + kernel_vector_begin(); + aes_cbc_cts_crypt_zvkned(dst, src, len, iv, key, true); + kernel_vector_end(); + return true; +} + +/* len can be any value greater than AES_BLOCK_SIZE here. */ +#define aes_cbc_cts_decrypt_arch aes_cbc_cts_decrypt_arch +static bool aes_cbc_cts_decrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 iv[AES_BLOCK_SIZE], + const struct aes_key *key) +{ + if (!static_branch_likely(&have_zvkned) || unlikely(!may_use_simd())) + return false; + + kernel_vector_begin(); + aes_cbc_cts_crypt_zvkned(dst, src, len, iv, key, false); + kernel_vector_end(); + return true; +} +#endif /* CONFIG_CRYPTO_LIB_AES_CBC */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E22C3546CC; Mon, 21 Sep 2026 05:16:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967776; cv=none; b=aJEgVBEVfZlikaKtUgDsN1jBsaEs5yNK+GBeUJqE0cedogq2Z6ZBQAOTjVEi0fAWa84Zh5vg32/DZCYlF2q6+eyo5s8c4Hos4SoNrOxAgpJcBDrIEmkCWS2BkYyuFQV1VtrLDjhb3qjPCPXQeSN6EgFmvwnP55atKfnEef4M+xk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967776; c=relaxed/simple; bh=1r6hsE99SUFRqJO4POQHax1X+AYpbQeLVu1GvN9Mc84=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B1qoFDHVk4GpCRBKT/HRm+XTuZVmDoherwnj98nPxzArpMBCr9YQBz7N/iukg3qc6SBLlPEzhqMoFDdRv7xIKvAkVFQyiMko0EwszoU6tE/efsj4xIsnD4CoWp1Gw2WD+fna9YPJXns2rhd7xpdgzpgSdjgWGxk8PIXF5yEVg0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Km9EuofK; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Km9EuofK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D1F71F000FF; Mon, 21 Sep 2026 05:16:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967770; bh=vlT+J7esaslXXYemR4OArHYwe0CTdSuz4G5sivbKcpY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Km9EuofKDPOo2VpnsdxomeJcX1G59x46/4U0i5hePbSYsrciV+HwTY5M79rET52RW x1QrFfpWmPS6gFB4LhtGA5JtiHfxG1o3bxpTIBo6JqdjxbSffvaVShrEquMJCdt5SN RNbPwgFl7ZBE/FGz+wkH2Mwjw6FxoZpMPPQYGwxpg3v9JRs9WAelHc6xLriNyXqgwz rI4F+787EHCEMsyAnYY0kRB9jONQwG4R6FdXGiUnVBZyueblV2ODzYDVIZW83UiP9Z AtYFADVHpV2yoE3ldZby3eeT/kNTF5lPr1jN4G4kMr/xSCT6EHG/wwzbcM+CTZ+p+7 3uJ65DcrPNzTA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 19/20] lib/crypto: riscv/aes-ctr: Migrate optimized code into library Date: Sun, 20 Sep 2026 22:09:05 -0700 Message-ID: <20260921050910.296144-20-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of exposing the riscv-optimized AES-CTR code via a riscv-specific crypto_skcipher algorithm, just implement the AES-CTR library functions. This is simpler, it makes the AES-CTR library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-CTR code was disabled by default. AES-CTR support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it. To match what the library expects, update the assembly functions to operate on struct aes_enckey rather than struct crypto_aes_ctx, and adjust the argument order. Bump up the priority of the corresponding library-based algorithm on riscv now that it no longer has to be lower than arch/riscv/crypto/. Also re-enable the library-based "ccm(aes)" and "gcm(aes)". Signed-off-by: Eric Biggers --- arch/riscv/crypto/Kconfig | 5 +- arch/riscv/crypto/Makefile | 2 +- arch/riscv/crypto/aes-riscv64-glue.c | 114 +----------------- crypto/aes.c | 5 +- lib/crypto/Makefile | 4 + .../crypto/riscv}/aes-riscv64-zvkned-zvkb.S | 23 ++-- lib/crypto/riscv/aes.h | 69 ++++++++++- 7 files changed, 91 insertions(+), 131 deletions(-) rename {arch/riscv/crypto =3D> lib/crypto/riscv}/aes-riscv64-zvkned-zvkb.S= (93%) diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig index 0a3f87ad384e..0733d4894401 100644 --- a/arch/riscv/crypto/Kconfig +++ b/arch/riscv/crypto/Kconfig @@ -3,18 +3,17 @@ menu "Accelerated Cryptographic Algorithms for CPU (riscv)" =20 config CRYPTO_AES_RISCV64 - tristate "Ciphers: AES, modes: CTR, XTS" + tristate "Ciphers: AES, modes: XTS" depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \ RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS select CRYPTO_LIB_AES select CRYPTO_SKCIPHER help - Length-preserving ciphers: AES with CTR, XTS + Length-preserving ciphers: AES with XTS =20 Architecture: riscv64 using: - Zvkned vector crypto extension - Zvbb vector extension (XTS) - - Zvkb vector crypto extension (CTR) - Zvkg vector crypto extension (XTS) =20 config CRYPTO_SM4_RISCV64 diff --git a/arch/riscv/crypto/Makefile b/arch/riscv/crypto/Makefile index d8b85afa6d0b..08904603fc94 100644 --- a/arch/riscv/crypto/Makefile +++ b/arch/riscv/crypto/Makefile @@ -2,7 +2,7 @@ =20 obj-$(CONFIG_CRYPTO_AES_RISCV64) +=3D aes-riscv64.o aes-riscv64-y :=3D aes-riscv64-glue.o \ - aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o + aes-riscv64-zvkned-zvbb-zvkg.o =20 obj-$(CONFIG_CRYPTO_SM4_RISCV64) +=3D sm4-riscv64.o sm4-riscv64-y :=3D sm4-riscv64-glue.o sm4-riscv64-zvksed-zvkb.o diff --git a/arch/riscv/crypto/aes-riscv64-glue.c b/arch/riscv/crypto/aes-r= iscv64-glue.c index 97f5369d7e71..a7dcceb77c49 100644 --- a/arch/riscv/crypto/aes-riscv64-glue.c +++ b/arch/riscv/crypto/aes-riscv64-glue.c @@ -22,10 +22,6 @@ #include #include =20 -asmlinkage void aes_ctr32_crypt_zvkned_zvkb(const struct crypto_aes_ctx *k= ey, - const u8 *in, u8 *out, size_t len, - u8 iv[AES_BLOCK_SIZE]); - asmlinkage void aes_xts_encrypt_zvkned_zvbb_zvkg( const struct crypto_aes_ctx *key, const u8 *in, u8 *out, size_t len, @@ -62,75 +58,6 @@ static int riscv64_aes_setkey(struct crypto_aes_ctx *ctx, return aes_expandkey(ctx, key, keylen); } =20 -static int riscv64_aes_setkey_skcipher(struct crypto_skcipher *tfm, - const u8 *key, unsigned int keylen) -{ - struct crypto_aes_ctx *ctx =3D crypto_skcipher_ctx(tfm); - - return riscv64_aes_setkey(ctx, key, keylen); -} - -/* AES-CTR */ - -static int riscv64_aes_ctr_crypt(struct skcipher_request *req) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct crypto_aes_ctx *ctx =3D crypto_skcipher_ctx(tfm); - unsigned int nbytes, p1_nbytes; - struct skcipher_walk walk; - u32 ctr32, nblocks; - int err; - - /* Get the low 32-bit word of the 128-bit big endian counter. */ - ctr32 =3D get_unaligned_be32(req->iv + 12); - - err =3D skcipher_walk_virt(&walk, req, false); - while ((nbytes =3D walk.nbytes) !=3D 0) { - if (nbytes < walk.total) { - /* Not the end yet, so keep the length block-aligned. */ - nbytes =3D round_down(nbytes, AES_BLOCK_SIZE); - nblocks =3D nbytes / AES_BLOCK_SIZE; - } else { - /* It's the end, so include any final partial block. */ - nblocks =3D DIV_ROUND_UP(nbytes, AES_BLOCK_SIZE); - } - ctr32 +=3D nblocks; - - kernel_vector_begin(); - if (ctr32 >=3D nblocks) { - /* The low 32-bit word of the counter won't overflow. */ - aes_ctr32_crypt_zvkned_zvkb(ctx, walk.src.virt.addr, - walk.dst.virt.addr, nbytes, - req->iv); - } else { - /* - * The low 32-bit word of the counter will overflow. - * The assembly doesn't handle this case, so split the - * operation into two at the point where the overflow - * will occur. After the first part, add the carry bit. - */ - p1_nbytes =3D min(nbytes, (nblocks - ctr32) * AES_BLOCK_SIZE); - aes_ctr32_crypt_zvkned_zvkb(ctx, walk.src.virt.addr, - walk.dst.virt.addr, - p1_nbytes, req->iv); - crypto_inc(req->iv, 12); - - if (ctr32) { - aes_ctr32_crypt_zvkned_zvkb( - ctx, - walk.src.virt.addr + p1_nbytes, - walk.dst.virt.addr + p1_nbytes, - nbytes - p1_nbytes, req->iv); - } - } - kernel_vector_end(); - - err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); - } - - return err; -} - /* AES-XTS */ =20 struct riscv64_aes_xts_ctx { @@ -251,25 +178,6 @@ static int riscv64_aes_xts_decrypt(struct skcipher_req= uest *req) =20 /* Algorithm definitions */ =20 -static struct skcipher_alg riscv64_zvkned_zvkb_aes_skcipher_alg =3D { - .setkey =3D riscv64_aes_setkey_skcipher, - .encrypt =3D riscv64_aes_ctr_crypt, - .decrypt =3D riscv64_aes_ctr_crypt, - .min_keysize =3D AES_MIN_KEY_SIZE, - .max_keysize =3D AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .chunksize =3D AES_BLOCK_SIZE, - .walksize =3D 4 * AES_BLOCK_SIZE, /* matches LMUL=3D4 */ - .base =3D { - .cra_blocksize =3D 1, - .cra_ctxsize =3D sizeof(struct crypto_aes_ctx), - .cra_priority =3D 300, - .cra_name =3D "ctr(aes)", - .cra_driver_name =3D "ctr-aes-riscv64-zvkned-zvkb", - .cra_module =3D THIS_MODULE, - }, -}; - static struct skcipher_alg riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg =3D { .setkey =3D riscv64_aes_xts_setkey, .encrypt =3D riscv64_aes_xts_encrypt, @@ -302,43 +210,27 @@ static int __init riscv64_aes_mod_init(void) =20 if (riscv_isa_extension_available(NULL, ZVKNED) && riscv_vector_vlen() >=3D 128) { - if (riscv_isa_extension_available(NULL, ZVKB)) { - err =3D crypto_register_skcipher( - &riscv64_zvkned_zvkb_aes_skcipher_alg); - if (err) - return err; - } - if (riscv64_aes_xts_supported()) { err =3D crypto_register_skcipher( &riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg); if (err) - goto unregister_zvkned_zvkb_skcipher_alg; + return err; } } =20 return err; - -unregister_zvkned_zvkb_skcipher_alg: - if (riscv_isa_extension_available(NULL, ZVKB)) - crypto_unregister_skcipher(&riscv64_zvkned_zvkb_aes_skcipher_alg); - return err; } =20 static void __exit riscv64_aes_mod_exit(void) { - if (riscv64_aes_xts_supported()) - crypto_unregister_skcipher(&riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg); - if (riscv_isa_extension_available(NULL, ZVKB)) - crypto_unregister_skcipher(&riscv64_zvkned_zvkb_aes_skcipher_alg); + crypto_unregister_skcipher(&riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg); } =20 module_init(riscv64_aes_mod_init); module_exit(riscv64_aes_mod_exit); =20 -MODULE_DESCRIPTION("AES-CTR/XTS (RISC-V accelerated)"); +MODULE_DESCRIPTION("AES-XTS (RISC-V accelerated)"); MODULE_AUTHOR("Jerry Shih "); MODULE_LICENSE("GPL"); MODULE_ALIAS_CRYPTO("aes"); -MODULE_ALIAS_CRYPTO("ctr(aes)"); MODULE_ALIAS_CRYPTO("xts(aes)"); diff --git a/crypto/aes.c b/crypto/aes.c index e951f0e1fe5a..9990e5034d34 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -670,7 +670,8 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "ctr(aes)", .base.cra_driver_name =3D "ctr-aes-lib", - .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, + .base.cra_priority =3D (IS_ENABLED(CONFIG_RISCV) || + IS_ENABLED(CONFIG_X86)) ? 300 : 110, .base.cra_blocksize =3D 1, .base.cra_ctxsize =3D sizeof(struct aes_enckey), .base.cra_module =3D THIS_MODULE, @@ -1002,7 +1003,6 @@ static struct aead_alg aead_algs[] =3D { !(IS_ENABLED(CONFIG_ARM) || \ IS_ENABLED(CONFIG_ARM64) || \ IS_ENABLED(CONFIG_POWERPC) || \ - IS_ENABLED(CONFIG_RISCV) || \ IS_ENABLED(CONFIG_S390) || \ IS_ENABLED(CONFIG_SPARC)) { @@ -1045,7 +1045,6 @@ static struct aead_alg aead_algs[] =3D { !(IS_ENABLED(CONFIG_ARM) || \ IS_ENABLED(CONFIG_ARM64) || \ IS_ENABLED(CONFIG_POWERPC) || \ - IS_ENABLED(CONFIG_RISCV) || \ IS_ENABLED(CONFIG_S390) || \ IS_ENABLED(CONFIG_SPARC)) { diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile index 02d89a226377..ff34aeda37ba 100644 --- a/lib/crypto/Makefile +++ b/lib/crypto/Makefile @@ -51,6 +51,10 @@ endif # !CONFIG_SPE endif # CONFIG_PPC =20 libaes-$(CONFIG_RISCV) +=3D riscv/aes-riscv64-zvkned.o +ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),) +libaes-$(CONFIG_RISCV) +=3D riscv/aes-riscv64-zvkned-zvkb.o +endif + libaes-$(CONFIG_SPARC) +=3D sparc/aes_asm.o =20 libaes-$(CONFIG_X86) +=3D x86/aes-aesni.o diff --git a/arch/riscv/crypto/aes-riscv64-zvkned-zvkb.S b/lib/crypto/riscv= /aes-riscv64-zvkned-zvkb.S similarity index 93% rename from arch/riscv/crypto/aes-riscv64-zvkned-zvkb.S rename to lib/crypto/riscv/aes-riscv64-zvkned-zvkb.S index 9962d4500587..93747d4cb5f3 100644 --- a/arch/riscv/crypto/aes-riscv64-zvkned-zvkb.S +++ b/lib/crypto/riscv/aes-riscv64-zvkned-zvkb.S @@ -49,11 +49,11 @@ =20 #include "aes-macros.S" =20 -#define KEYP a0 -#define INP a1 -#define OUTP a2 -#define LEN a3 -#define IVP a4 +#define DST a0 +#define SRC a1 +#define LEN a2 +#define IVP a3 +#define KEYP a4 =20 #define LEN32 a5 #define VL_E32 a6 @@ -110,13 +110,13 @@ =20 // XOR the data with the keystream. vsetvli t0, LEN, e8, m4, ta, ma - vle8.v v20, (INP) + vle8.v v20, (SRC) vxor.vv v20, v20, v24 - vse8.v v20, (OUTP) + vse8.v v20, (DST) =20 // Advance the pointers and update the remaining length. - add INP, INP, t0 - add OUTP, OUTP, t0 + add SRC, SRC, t0 + add DST, DST, t0 sub LEN, LEN, t0 sub LEN32, LEN32, VL_E32 srli VL_BLOCKS, VL_E32, 2 @@ -133,9 +133,8 @@ ret .endm =20 -// void aes_ctr32_crypt_zvkned_zvkb(const struct crypto_aes_ctx *key, -// const u8 *in, u8 *out, size_t len, -// u8 iv[16]); +// void aes_ctr32_crypt_zvkned_zvkb(u8 *dst, const u8 *src, u32 len, u8 iv= [16], +// const struct aes_enckey *key); SYM_FUNC_START(aes_ctr32_crypt_zvkned_zvkb) aes_begin KEYP, 128f, 192f aes_ctr32_crypt 256 diff --git a/lib/crypto/riscv/aes.h b/lib/crypto/riscv/aes.h index e02f9343d67d..2c4d1e58c703 100644 --- a/lib/crypto/riscv/aes.h +++ b/lib/crypto/riscv/aes.h @@ -9,6 +9,7 @@ #include =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_zvkned); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_zvkned_zvkb); =20 /* The assembly code assumes the following offsets. */ static_assert(offsetof(struct aes_enckey, len) =3D=3D 0); @@ -159,10 +160,76 @@ static bool aes_cbc_cts_decrypt_arch(u8 *dst, const u= 8 *src, size_t len, } #endif /* CONFIG_CRYPTO_LIB_AES_CBC */ =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CTR) +void aes_ctr32_crypt_zvkned_zvkb(u8 *dst, const u8 *src, u32 len, + u8 iv[16], const struct aes_enckey *key); + +static void aes_ctr_riscv(u8 *dst, const u8 *src, u32 len, + u8 ctr[AES_BLOCK_SIZE], const struct aes_enckey *key) +{ + u32 ctr32 =3D get_unaligned_be32(&ctr[12]); + u32 part1_len; + u32 nblocks; + + nblocks =3D DIV_ROUND_UP(len, AES_BLOCK_SIZE); + ctr32 +=3D nblocks; + + if (likely(ctr32 >=3D nblocks)) { + /* The low 32 bits of the counter won't overflow. */ + aes_ctr32_crypt_zvkned_zvkb(dst, src, len, ctr, key); + } else { + /* + * The low 32 bits of the counter will overflow. The + * assembly doesn't handle this case, so split the + * operation into two at the point where the overflow + * will occur. After the first part, add the carry bit. + */ + part1_len =3D min(len, (nblocks - ctr32) * AES_BLOCK_SIZE); + aes_ctr32_crypt_zvkned_zvkb(dst, src, part1_len, ctr, key); + for (int i =3D AES_BLOCK_SIZE - 5; i >=3D 0; i--) { + if (++ctr[i] !=3D 0) + break; + } + if (part1_len < len) + aes_ctr32_crypt_zvkned_zvkb(dst + part1_len, + src + part1_len, + len - part1_len, ctr, key); + } +} + +#define aes_ctr_arch aes_ctr_arch +static bool aes_ctr_arch(u8 *dst, const u8 *src, size_t len, + u8 ctr[AES_BLOCK_SIZE], const struct aes_enckey *key) +{ + if (!static_branch_likely(&have_zvkned_zvkb) || + unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + while (len) { + /* + * Process at most a 32-bit len at a time, so that each step + * needs to handle at most 1 carry bit out of the low 32-bit + * word of the counter. + */ + u32 n =3D min(len, round_down(U32_MAX, AES_BLOCK_SIZE)); + + aes_ctr_riscv(dst, src, n, ctr, key); + dst +=3D n; + src +=3D n; + len -=3D n; + } + kernel_vector_end(); + return true; +} +#endif /* CONFIG_CRYPTO_LIB_AES_CTR */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { if (riscv_isa_extension_available(NULL, ZVKNED) && - riscv_vector_vlen() >=3D 128) + riscv_vector_vlen() >=3D 128) { static_branch_enable(&have_zvkned); + if (riscv_isa_extension_available(NULL, ZVKB)) + static_branch_enable(&have_zvkned_zvkb); + } } --=20 2.55.0 From nobody Thu Sep 24 20:31:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4206F364EB0; Mon, 21 Sep 2026 05:16:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967775; cv=none; b=jXK5LmfsjQ1qAaEbwDsVObP81lypuSWdFdb5PLhypKxSFEOdo/5zcJVanlR4cp7vZnnnJ9ZKjmva84Sh4e0xLz5j5iYzUmmpkB756DE5OD9OsLWVrq/V9vn9u3Y1h1jKJNmnal0ddHUNEUqKOnKinMh3yJ4zcwQGhR3kcsEcAU8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789967775; c=relaxed/simple; bh=i3Y/7w1J5Qa9h3tTbtwc43bFrrIUkSTfWgLXNrhdKMg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Mrtd69IzER6ltFOOwyOzgTv7Cs6UFl1oOQk/RbmQ/FH2xQR6WuWdAtanidSCLZDEXzgyx0v+hhR/LjXD3zw0aHmhzQZf8sNo37fpsLOAfd3P+sXdFNAMK/U1N3an5xvjMOIwIXOqVxm/cJsHW/E9nl+81ncNAd/N42BEpmv3vNc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=irQ2EgMt; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="irQ2EgMt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 97E101F0089D; Mon, 21 Sep 2026 05:16:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789967770; bh=MkyyNHKuYCDPCVmWsZf9HSuueJU5Xg70rl8FKthz/7g=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=irQ2EgMttOtWnDEVEMMRrZAWMY9y5rZvUzNRsWtg6aKFnuT3P/oOMgSVP8tnuheHa jO/a7Ae/duUzKi7YfMg1ccAH/xLZQXjejtR937gxZYmv53l1rE7iRUOALPsYHGgyRu EPQJ8j78ciwDoLdc22A8dDOweBJkY6oDQFUfCRPfindTkDu7j+9tnGkF7nfPfVNZS1 xmNz2/xZVqnfJUZ8fdiO/DLuU2JaGwSQFbli5tTPFJOLHSh1tKuWR14zPUZ6IU2PFS q2gyy8K37b1Ae3h/BoAZQuwfcbg52DHTN8yW/xYHd8pUyvja89cdJKuM73K7EVF/YG zr3cKACkl8lHw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , "Jason A . Donenfeld" , Herbert Xu , x86@kernel.org, linux-riscv@lists.infradead.org, Eric Biggers Subject: [PATCH 20/20] lib/crypto: riscv/aes-xts: Migrate optimized code into library Date: Sun, 20 Sep 2026 22:09:06 -0700 Message-ID: <20260921050910.296144-21-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260921050910.296144-1-ebiggers@kernel.org> References: <20260921050910.296144-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Instead of exposing the riscv-optimized AES-XTS code via a riscv-specific crypto_skcipher algorithm, just implement the AES-XTS library functions. This is simpler, it makes the AES-XTS library functions be riscv-optimized, and it also fixes the longstanding issue where the riscv-optimized AES-XTS code was disabled by default. AES-XTS support still remains available through crypto_skcipher via crypto/aes.c, but individual architectures no longer need to handle it. To match what the library expects, update the assembly functions to operate on struct aes_key rather than struct crypto_aes_ctx, adjust the argument order, and remove the redundant ciphertext stealing support which is already implemented in a generic way in the library. Bump up the priority of the corresponding library-based algorithm on riscv now that it no longer has to be lower than arch/riscv/crypto/. Signed-off-by: Eric Biggers --- arch/riscv/crypto/Kconfig | 14 -- arch/riscv/crypto/Makefile | 4 - arch/riscv/crypto/aes-macros.S | 166 ------------ arch/riscv/crypto/aes-riscv64-glue.c | 236 ------------------ crypto/aes.c | 3 +- lib/crypto/Makefile | 3 + .../riscv}/aes-riscv64-zvkned-zvbb-zvkg.S | 96 ++----- lib/crypto/riscv/aes.h | 53 ++++ 8 files changed, 74 insertions(+), 501 deletions(-) delete mode 100644 arch/riscv/crypto/aes-macros.S delete mode 100644 arch/riscv/crypto/aes-riscv64-glue.c rename {arch/riscv/crypto =3D> lib/crypto/riscv}/aes-riscv64-zvkned-zvbb-z= vkg.S (75%) diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig index 0733d4894401..614f93214862 100644 --- a/arch/riscv/crypto/Kconfig +++ b/arch/riscv/crypto/Kconfig @@ -2,20 +2,6 @@ =20 menu "Accelerated Cryptographic Algorithms for CPU (riscv)" =20 -config CRYPTO_AES_RISCV64 - tristate "Ciphers: AES, modes: XTS" - depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \ - RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS - select CRYPTO_LIB_AES - select CRYPTO_SKCIPHER - help - Length-preserving ciphers: AES with XTS - - Architecture: riscv64 using: - - Zvkned vector crypto extension - - Zvbb vector extension (XTS) - - Zvkg vector crypto extension (XTS) - config CRYPTO_SM4_RISCV64 tristate "Ciphers: SM4 (ShangMi 4)" depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \ diff --git a/arch/riscv/crypto/Makefile b/arch/riscv/crypto/Makefile index 08904603fc94..9f6956cf50b1 100644 --- a/arch/riscv/crypto/Makefile +++ b/arch/riscv/crypto/Makefile @@ -1,8 +1,4 @@ # SPDX-License-Identifier: GPL-2.0-only =20 -obj-$(CONFIG_CRYPTO_AES_RISCV64) +=3D aes-riscv64.o -aes-riscv64-y :=3D aes-riscv64-glue.o \ - aes-riscv64-zvkned-zvbb-zvkg.o - obj-$(CONFIG_CRYPTO_SM4_RISCV64) +=3D sm4-riscv64.o sm4-riscv64-y :=3D sm4-riscv64-glue.o sm4-riscv64-zvksed-zvkb.o diff --git a/arch/riscv/crypto/aes-macros.S b/arch/riscv/crypto/aes-macros.S deleted file mode 100644 index 1384164621a5..000000000000 --- a/arch/riscv/crypto/aes-macros.S +++ /dev/null @@ -1,166 +0,0 @@ -/* SPDX-License-Identifier: Apache-2.0 OR BSD-2-Clause */ -// -// This file is dual-licensed, meaning that you can use it under your -// choice of either of the following two licenses: -// -// Copyright 2023 The OpenSSL Project Authors. All Rights Reserved. -// -// Licensed under the Apache License 2.0 (the "License"). You can obtain -// a copy in the file LICENSE in the source distribution or at -// https://www.openssl.org/source/license.html -// -// or -// -// Copyright (c) 2023, Christoph M=C3=BCllner -// Copyright (c) 2023, Phoebe Chen -// Copyright (c) 2023, Jerry Shih -// Copyright 2024 Google LLC -// All rights reserved. -// -// Redistribution and use in source and binary forms, with or without -// modification, are permitted provided that the following conditions -// are met: -// 1. Redistributions of source code must retain the above copyright -// notice, this list of conditions and the following disclaimer. -// 2. Redistributions in binary form must reproduce the above copyright -// notice, this list of conditions and the following disclaimer in the -// documentation and/or other materials provided with the distribution. -// -// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT -// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, -// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT -// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, -// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY -// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT -// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE -// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -// This file contains macros that are shared by the other aes-*.S files. = The -// generated code of these macros depends on the following RISC-V extensio= ns: -// - RV64I -// - RISC-V Vector ('V') with VLEN >=3D 128 -// - RISC-V Vector AES block cipher extension ('Zvkned') - -// Loads the AES round keys from \keyp into vector registers and jumps to = code -// specific to the length of the key. Specifically: -// - If AES-128, loads round keys into v1-v11 and jumps to \label128. -// - If AES-192, loads round keys into v1-v13 and jumps to \label192. -// - If AES-256, loads round keys into v1-v15 and continues onwards. -// -// Also sets vl=3D4 and vtype=3De32,m1,ta,ma. Clobbers t0 and t1. -.macro aes_begin keyp, label128, label192, key_len -.ifb \key_len - lwu t0, 480(\keyp) // t0 =3D key length in bytes -.endif - li t1, 24 // t1 =3D key length for AES-192 - vsetivli zero, 4, e32, m1, ta, ma - vle32.v v1, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v2, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v3, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v4, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v5, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v6, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v7, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v8, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v9, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v10, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v11, (\keyp) -.ifb \key_len - blt t0, t1, \label128 // If AES-128, goto label128. -.else - blt \key_len, t1, \label128 // If AES-128, goto label128. -.endif - addi \keyp, \keyp, 16 - vle32.v v12, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v13, (\keyp) -.ifb \key_len - beq t0, t1, \label192 // If AES-192, goto label192. -.else - beq \key_len, t1, \label192 // If AES-192, goto label192. -.endif - // Else, it's AES-256. - addi \keyp, \keyp, 16 - vle32.v v14, (\keyp) - addi \keyp, \keyp, 16 - vle32.v v15, (\keyp) -.endm - -// Encrypts \data using zvkned instructions, using the round keys loaded i= nto -// v1-v11 (for AES-128), v1-v13 (for AES-192), or v1-v15 (for AES-256). \= keylen -// is the AES key length in bits. vl and vtype must already be set -// appropriately. Note that if vl > 4, multiple blocks are encrypted. -.macro aes_encrypt data, keylen - vaesz.vs \data, v1 - vaesem.vs \data, v2 - vaesem.vs \data, v3 - vaesem.vs \data, v4 - vaesem.vs \data, v5 - vaesem.vs \data, v6 - vaesem.vs \data, v7 - vaesem.vs \data, v8 - vaesem.vs \data, v9 - vaesem.vs \data, v10 -.if \keylen =3D=3D 128 - vaesef.vs \data, v11 -.elseif \keylen =3D=3D 192 - vaesem.vs \data, v11 - vaesem.vs \data, v12 - vaesef.vs \data, v13 -.else - vaesem.vs \data, v11 - vaesem.vs \data, v12 - vaesem.vs \data, v13 - vaesem.vs \data, v14 - vaesef.vs \data, v15 -.endif -.endm - -// Same as aes_encrypt, but decrypts instead of encrypts. -.macro aes_decrypt data, keylen -.if \keylen =3D=3D 128 - vaesz.vs \data, v11 -.elseif \keylen =3D=3D 192 - vaesz.vs \data, v13 - vaesdm.vs \data, v12 - vaesdm.vs \data, v11 -.else - vaesz.vs \data, v15 - vaesdm.vs \data, v14 - vaesdm.vs \data, v13 - vaesdm.vs \data, v12 - vaesdm.vs \data, v11 -.endif - vaesdm.vs \data, v10 - vaesdm.vs \data, v9 - vaesdm.vs \data, v8 - vaesdm.vs \data, v7 - vaesdm.vs \data, v6 - vaesdm.vs \data, v5 - vaesdm.vs \data, v4 - vaesdm.vs \data, v3 - vaesdm.vs \data, v2 - vaesdf.vs \data, v1 -.endm - -// Expands to aes_encrypt or aes_decrypt according to \enc, which is 1 or = 0. -.macro aes_crypt data, enc, keylen -.if \enc - aes_encrypt \data, \keylen -.else - aes_decrypt \data, \keylen -.endif -.endm diff --git a/arch/riscv/crypto/aes-riscv64-glue.c b/arch/riscv/crypto/aes-r= iscv64-glue.c deleted file mode 100644 index a7dcceb77c49..000000000000 --- a/arch/riscv/crypto/aes-riscv64-glue.c +++ /dev/null @@ -1,236 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * AES modes using the RISC-V vector crypto extensions - * - * Copyright (C) 2023 VRULL GmbH - * Author: Heiko Stuebner - * - * Copyright (C) 2023 SiFive, Inc. - * Author: Jerry Shih - * - * Copyright 2024 Google LLC - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -asmlinkage void aes_xts_encrypt_zvkned_zvbb_zvkg( - const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len, - u8 tweak[AES_BLOCK_SIZE]); - -asmlinkage void aes_xts_decrypt_zvkned_zvbb_zvkg( - const struct crypto_aes_ctx *key, - const u8 *in, u8 *out, size_t len, - u8 tweak[AES_BLOCK_SIZE]); - -static int riscv64_aes_setkey(struct crypto_aes_ctx *ctx, - const u8 *key, unsigned int keylen) -{ - /* - * For now we just use the generic key expansion, for these reasons: - * - * - zvkned's key expansion instructions don't support AES-192. - * So, non-zvkned fallback code would be needed anyway. - * - * - Users of AES in Linux usually don't change keys frequently. - * So, key expansion isn't performance-critical. - * - * - For single-block AES exposed as a "cipher" algorithm, it's - * necessary to use struct crypto_aes_ctx and initialize its 'key_dec' - * field with the round keys for the Equivalent Inverse Cipher. This - * is because with "cipher", decryption can be requested from a - * context where the vector unit isn't usable, necessitating a - * fallback to aes_decrypt(). But, zvkned can only generate and use - * the normal round keys. Of course, it's preferable to not have - * special code just for "cipher", as e.g. XTS also uses a - * single-block AES encryption. It's simplest to just use - * struct crypto_aes_ctx and aes_expandkey() everywhere. - */ - return aes_expandkey(ctx, key, keylen); -} - -/* AES-XTS */ - -struct riscv64_aes_xts_ctx { - struct crypto_aes_ctx ctx1; - struct aes_enckey tweak_key; -}; - -static int riscv64_aes_xts_setkey(struct crypto_skcipher *tfm, const u8 *k= ey, - unsigned int keylen) -{ - struct riscv64_aes_xts_ctx *ctx =3D crypto_skcipher_ctx(tfm); - - return xts_verify_key(tfm, key, keylen) ?: - riscv64_aes_setkey(&ctx->ctx1, key, keylen / 2) ?: - aes_prepareenckey(&ctx->tweak_key, key + keylen / 2, keylen / 2); -} - -static int riscv64_aes_xts_crypt(struct skcipher_request *req, bool enc) -{ - struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - const struct riscv64_aes_xts_ctx *ctx =3D crypto_skcipher_ctx(tfm); - int tail =3D req->cryptlen % AES_BLOCK_SIZE; - struct scatterlist sg_src[2], sg_dst[2]; - struct skcipher_request subreq; - struct scatterlist *src, *dst; - struct skcipher_walk walk; - int err; - - if (req->cryptlen < AES_BLOCK_SIZE) - return -EINVAL; - - /* Encrypt the IV with the tweak key to get the first tweak. */ - aes_encrypt(&ctx->tweak_key, req->iv, req->iv); - - err =3D skcipher_walk_virt(&walk, req, false); - - /* - * If the message length isn't divisible by the AES block size and the - * full message isn't available in one step of the scatterlist walk, - * then separate off the last full block and the partial block. This - * ensures that they are processed in the same call to the assembly - * function, which is required for ciphertext stealing. - */ - if (unlikely(tail > 0 && walk.nbytes < walk.total)) { - skcipher_walk_abort(&walk); - - skcipher_request_set_tfm(&subreq, tfm); - skcipher_request_set_callback(&subreq, - skcipher_request_flags(req), - NULL, NULL); - skcipher_request_set_crypt(&subreq, req->src, req->dst, - req->cryptlen - tail - AES_BLOCK_SIZE, - req->iv); - req =3D &subreq; - err =3D skcipher_walk_virt(&walk, req, false); - } else { - tail =3D 0; - } - - while (walk.nbytes) { - unsigned int nbytes =3D walk.nbytes; - - if (nbytes < walk.total) - nbytes =3D round_down(nbytes, AES_BLOCK_SIZE); - - kernel_vector_begin(); - if (enc) - aes_xts_encrypt_zvkned_zvbb_zvkg( - &ctx->ctx1, walk.src.virt.addr, - walk.dst.virt.addr, nbytes, req->iv); - else - aes_xts_decrypt_zvkned_zvbb_zvkg( - &ctx->ctx1, walk.src.virt.addr, - walk.dst.virt.addr, nbytes, req->iv); - kernel_vector_end(); - err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); - } - - if (err || likely(!tail)) - return err; - - /* Do ciphertext stealing with the last full block and partial block. */ - - dst =3D src =3D scatterwalk_ffwd(sg_src, req->src, req->cryptlen); - if (req->dst !=3D req->src) - dst =3D scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen); - - skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail, - req->iv); - - err =3D skcipher_walk_virt(&walk, req, false); - if (err) - return err; - - kernel_vector_begin(); - if (enc) - aes_xts_encrypt_zvkned_zvbb_zvkg( - &ctx->ctx1, walk.src.virt.addr, - walk.dst.virt.addr, walk.nbytes, req->iv); - else - aes_xts_decrypt_zvkned_zvbb_zvkg( - &ctx->ctx1, walk.src.virt.addr, - walk.dst.virt.addr, walk.nbytes, req->iv); - kernel_vector_end(); - - return skcipher_walk_done(&walk, 0); -} - -static int riscv64_aes_xts_encrypt(struct skcipher_request *req) -{ - return riscv64_aes_xts_crypt(req, true); -} - -static int riscv64_aes_xts_decrypt(struct skcipher_request *req) -{ - return riscv64_aes_xts_crypt(req, false); -} - -/* Algorithm definitions */ - -static struct skcipher_alg riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg =3D { - .setkey =3D riscv64_aes_xts_setkey, - .encrypt =3D riscv64_aes_xts_encrypt, - .decrypt =3D riscv64_aes_xts_decrypt, - .min_keysize =3D 2 * AES_MIN_KEY_SIZE, - .max_keysize =3D 2 * AES_MAX_KEY_SIZE, - .ivsize =3D AES_BLOCK_SIZE, - .chunksize =3D AES_BLOCK_SIZE, - .walksize =3D 4 * AES_BLOCK_SIZE, /* matches LMUL=3D4 */ - .base =3D { - .cra_blocksize =3D AES_BLOCK_SIZE, - .cra_ctxsize =3D sizeof(struct riscv64_aes_xts_ctx), - .cra_priority =3D 300, - .cra_name =3D "xts(aes)", - .cra_driver_name =3D "xts-aes-riscv64-zvkned-zvbb-zvkg", - .cra_module =3D THIS_MODULE, - }, -}; - -static inline bool riscv64_aes_xts_supported(void) -{ - return riscv_isa_extension_available(NULL, ZVBB) && - riscv_isa_extension_available(NULL, ZVKG) && - riscv_vector_vlen() < 2048 /* Implementation limitation */; -} - -static int __init riscv64_aes_mod_init(void) -{ - int err =3D -ENODEV; - - if (riscv_isa_extension_available(NULL, ZVKNED) && - riscv_vector_vlen() >=3D 128) { - if (riscv64_aes_xts_supported()) { - err =3D crypto_register_skcipher( - &riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg); - if (err) - return err; - } - } - - return err; -} - -static void __exit riscv64_aes_mod_exit(void) -{ - crypto_unregister_skcipher(&riscv64_zvkned_zvbb_zvkg_aes_skcipher_alg); -} - -module_init(riscv64_aes_mod_init); -module_exit(riscv64_aes_mod_exit); - -MODULE_DESCRIPTION("AES-XTS (RISC-V accelerated)"); -MODULE_AUTHOR("Jerry Shih "); -MODULE_LICENSE("GPL"); -MODULE_ALIAS_CRYPTO("aes"); -MODULE_ALIAS_CRYPTO("xts(aes)"); diff --git a/crypto/aes.c b/crypto/aes.c index 9990e5034d34..a5f34cbf6676 100644 --- a/crypto/aes.c +++ b/crypto/aes.c @@ -705,7 +705,8 @@ static struct skcipher_alg skcipher_algs[] =3D { { .base.cra_name =3D "xts(aes)", .base.cra_driver_name =3D "xts-aes-lib", - .base.cra_priority =3D IS_ENABLED(CONFIG_X86) ? 300 : 110, + .base.cra_priority =3D (IS_ENABLED(CONFIG_RISCV) || + IS_ENABLED(CONFIG_X86)) ? 300 : 110, .base.cra_blocksize =3D AES_BLOCK_SIZE, .base.cra_ctxsize =3D sizeof(struct aes_xts_key), .base.cra_module =3D THIS_MODULE, diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile index ff34aeda37ba..d683b8520f55 100644 --- a/lib/crypto/Makefile +++ b/lib/crypto/Makefile @@ -54,6 +54,9 @@ libaes-$(CONFIG_RISCV) +=3D riscv/aes-riscv64-zvkned.o ifneq ($(CONFIG_CRYPTO_LIB_AES_CTR),) libaes-$(CONFIG_RISCV) +=3D riscv/aes-riscv64-zvkned-zvkb.o endif +ifneq ($(CONFIG_CRYPTO_LIB_AES_XTS),) +libaes-$(CONFIG_RISCV) +=3D riscv/aes-riscv64-zvkned-zvbb-zvkg.o +endif =20 libaes-$(CONFIG_SPARC) +=3D sparc/aes_asm.o =20 diff --git a/arch/riscv/crypto/aes-riscv64-zvkned-zvbb-zvkg.S b/lib/crypto/= riscv/aes-riscv64-zvkned-zvbb-zvkg.S similarity index 75% rename from arch/riscv/crypto/aes-riscv64-zvkned-zvbb-zvkg.S rename to lib/crypto/riscv/aes-riscv64-zvkned-zvbb-zvkg.S index 146fc9cfb268..0a87e2666ae2 100644 --- a/arch/riscv/crypto/aes-riscv64-zvkned-zvbb-zvkg.S +++ b/lib/crypto/riscv/aes-riscv64-zvkned-zvbb-zvkg.S @@ -50,11 +50,11 @@ =20 #include "aes-macros.S" =20 -#define KEYP a0 -#define INP a1 -#define OUTP a2 -#define LEN a3 -#define TWEAKP a4 +#define DST a0 +#define SRC a1 +#define LEN a2 +#define TWEAKP a3 +#define KEYP a4 =20 #define LEN32 a5 #define TAIL_LEN a6 @@ -167,24 +167,21 @@ .endm =20 .macro __aes_xts_crypt enc, keylen - // With 16 < len <=3D 31, there's no main loop, just ciphertext stealing. - beqz LEN32, .Lcts_without_main_loop\@ - vsetvli VLMAX, zero, e32, m4, ta, ma 1: vsetvli VL, LEN32, e32, m4, ta, ma 2: // Encrypt or decrypt VL/4 blocks. - vle32.v TMP0, (INP) + vle32.v TMP0, (SRC) vxor.vv TMP0, TMP0, TWEAKS aes_crypt TMP0, \enc, \keylen vxor.vv TMP0, TMP0, TWEAKS - vse32.v TMP0, (OUTP) + vse32.v TMP0, (DST) =20 // Update the pointers and the remaining length. slli t0, VL, 2 - add INP, INP, t0 - add OUTP, OUTP, t0 + add SRC, SRC, t0 + add DST, DST, t0 sub LEN32, LEN32, VL =20 // Check whether more blocks remain. @@ -217,73 +214,14 @@ vsetivli zero, 4, e32, m1, ta, ma vgmul.vv TWEAKS_BREV, MULTS_BREV // Advance to next tweak =20 - bnez TAIL_LEN, .Lcts\@ - // Update *TWEAKP to contain the next tweak. vbrev8.v TWEAKS, TWEAKS_BREV vse32.v TWEAKS, (TWEAKP) ret - -.Lcts_without_main_loop\@: - load_x -.Lcts\@: - // TWEAKS_BREV now contains the next tweak. Compute the one after that. - vsetivli zero, 4, e32, m1, ta, ma - vmv.v.v TMP0, TWEAKS_BREV - vgmul.vv TMP0, MULTS_BREV - // Undo the bit reversal of the next two tweaks and store them in TMP1 - // and TMP2, such that TMP1 is the first needed and TMP2 the second. -.if \enc - vbrev8.v TMP1, TWEAKS_BREV - vbrev8.v TMP2, TMP0 -.else - vbrev8.v TMP1, TMP0 - vbrev8.v TMP2, TWEAKS_BREV -.endif - - // Encrypt/decrypt the last full block. - vle32.v TMP0, (INP) - vxor.vv TMP0, TMP0, TMP1 - aes_crypt TMP0, \enc, \keylen - vxor.vv TMP0, TMP0, TMP1 - - // Swap the first TAIL_LEN bytes of the above result with the tail. - // Note that to support in-place encryption/decryption, the load from - // the input tail must happen before the store to the output tail. - addi t0, INP, 16 - addi t1, OUTP, 16 - vmv.v.v TMP3, TMP0 - vsetvli zero, TAIL_LEN, e8, m1, tu, ma - vle8.v TMP0, (t0) - vse8.v TMP3, (t1) - - // Encrypt/decrypt again and store the last full block. - vsetivli zero, 4, e32, m1, ta, ma - vxor.vv TMP0, TMP0, TMP2 - aes_crypt TMP0, \enc, \keylen - vxor.vv TMP0, TMP0, TMP2 - vse32.v TMP0, (OUTP) - - ret .endm =20 .macro aes_xts_crypt enc - - // Check whether the length is a multiple of the AES block size. - andi TAIL_LEN, LEN, 15 - beqz TAIL_LEN, 1f - - // The length isn't a multiple of the AES block size, so ciphertext - // stealing will be required. Ciphertext stealing involves special - // handling of the partial block and the last full block, so subtract - // the length of both from the length to be processed in the main loop. - sub LEN, LEN, TAIL_LEN - addi LEN, LEN, -16 -1: srli LEN32, LEN, 2 - // LEN and LEN32 now contain the total length of the blocks that will be - // processed in the main loop, in bytes and 32-bit words respectively. - xts_init aes_begin KEYP, 128f, 192f __aes_xts_crypt \enc, 256 @@ -293,15 +231,13 @@ __aes_xts_crypt \enc, 192 .endm =20 -// void aes_xts_encrypt_zvkned_zvbb_zvkg(const struct crypto_aes_ctx *key, -// const u8 *in, u8 *out, size_t len, -// u8 tweak[16]); -// -// |key| is the data key. |tweak| contains the next tweak; the encryption= of -// the original IV with the tweak key was already done. This function sup= ports -// incremental computation, but |len| must always be >=3D 16 (AES_BLOCK_SI= ZE), and -// |len| must be a multiple of 16 except on the last call. If |len| is a -// multiple of 16, then this function updates |tweak| to contain the next = tweak. +// void aes_xts_encrypt_zvkned_zvbb_zvkg(u8 *dst, const u8 *src, size_t le= n, +// u8 tweak[AES_BLOCK_SIZE], +// const struct aes_key *key); + +// `tweak` must have already been encrypted by the tweak key; `key` is jus= t the +// main key. To allow incremental computation, this updates `tweak` to co= ntain +// the next tweak. SYM_FUNC_START(aes_xts_encrypt_zvkned_zvbb_zvkg) aes_xts_crypt 1 SYM_FUNC_END(aes_xts_encrypt_zvkned_zvbb_zvkg) diff --git a/lib/crypto/riscv/aes.h b/lib/crypto/riscv/aes.h index 2c4d1e58c703..1727302568f8 100644 --- a/lib/crypto/riscv/aes.h +++ b/lib/crypto/riscv/aes.h @@ -1,5 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0-only */ /* + * AES using the RISC-V vector crypto extensions + * * Copyright (C) 2023 VRULL GmbH * Copyright (C) 2023 SiFive, Inc. * Copyright 2024 Google LLC @@ -10,6 +12,7 @@ =20 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_zvkned); static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_zvkned_zvkb); +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_xts); =20 /* The assembly code assumes the following offsets. */ static_assert(offsetof(struct aes_enckey, len) =3D=3D 0); @@ -223,6 +226,52 @@ static bool aes_ctr_arch(u8 *dst, const u8 *src, size_= t len, } #endif /* CONFIG_CRYPTO_LIB_AES_CTR */ =20 +#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_XTS) +void aes_xts_encrypt_zvkned_zvbb_zvkg(u8 *dst, const u8 *src, size_t len, + u8 tweak[AES_BLOCK_SIZE], + const struct aes_key *key); +void aes_xts_decrypt_zvkned_zvbb_zvkg(u8 *dst, const u8 *src, size_t len, + u8 tweak[AES_BLOCK_SIZE], + const struct aes_key *key); + +/* len is always a positive multiple of AES_BLOCK_SIZE here. */ +static __always_inline bool +aes_xts_crypt_riscv(u8 *dst, const u8 *src, size_t len, + u8 tweak[AES_BLOCK_SIZE], + const struct aes_xts_key *key, bool cont, bool enc) +{ + if (!static_branch_likely(&have_xts) || unlikely(!may_use_simd())) + return false; + kernel_vector_begin(); + if (!cont) + aes_encrypt_zvkned(&key->tweak_key, tweak, tweak); + if (enc) + aes_xts_encrypt_zvkned_zvbb_zvkg(dst, src, len, tweak, + &key->main_key); + else + aes_xts_decrypt_zvkned_zvbb_zvkg(dst, src, len, tweak, + &key->main_key); + kernel_vector_end(); + return true; +} + +#define aes_xts_encrypt_arch aes_xts_encrypt_arch +static bool aes_xts_encrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 tweak[AES_BLOCK_SIZE], + const struct aes_xts_key *key, bool cont) +{ + return aes_xts_crypt_riscv(dst, src, len, tweak, key, cont, true); +} + +#define aes_xts_decrypt_arch aes_xts_decrypt_arch +static bool aes_xts_decrypt_arch(u8 *dst, const u8 *src, size_t len, + u8 tweak[AES_BLOCK_SIZE], + const struct aes_xts_key *key, bool cont) +{ + return aes_xts_crypt_riscv(dst, src, len, tweak, key, cont, false); +} +#endif /* CONFIG_CRYPTO_LIB_AES_XTS */ + #define aes_mod_init_arch aes_mod_init_arch static void aes_mod_init_arch(void) { @@ -231,5 +280,9 @@ static void aes_mod_init_arch(void) static_branch_enable(&have_zvkned); if (riscv_isa_extension_available(NULL, ZVKB)) static_branch_enable(&have_zvkned_zvkb); + if (riscv_isa_extension_available(NULL, ZVBB) && + riscv_isa_extension_available(NULL, ZVKG) && + riscv_vector_vlen() < 2048 /* Implementation limitation */) + static_branch_enable(&have_xts); } } --=20 2.55.0