From nobody Thu Oct 2 11:48:21 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CB26311C2C for ; Wed, 1 Oct 2025 21:04:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352650; cv=none; b=hR0slCpuLubvpH0RzmRPlkP/D6PA5zR1Vzabloam/ANH8yLi0GVlEBNCbZeTCInfXMk6XZaa06x5DLUoIfDkTcXTj/vLAQWQ1HeRCpkdFPdom0fnTJh16PVZSGSQe3LjdtU2zj5A4vzzApvCuE6n0JOqHlyHn+htALmj1xc3y34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759352650; c=relaxed/simple; bh=FsmKCKW0+3JBks3b7w6jj2RyXifJfqC/SFYqHtUwJ78=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g51wJJYdVEOHnBfrfwWflvYug9CWWn08NL501Ocye1wrMIBfGR2NgKQwJctycyCs2ozH8IYrB5oaJZ3z4wPNU0V1q9lxEgHGbxIcIFGBUk1pWG5Mq0MB/8vlF68HdFE1t22NtXoBWggATIsOnzZTd4oAfQEu8c9rAgsahgV4+xE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OIyTiwxX; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OIyTiwxX" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b3d525fb67eso31702466b.0 for ; Wed, 01 Oct 2025 14:04:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1759352645; x=1759957445; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2tGgNTZuDE42TuUV9n3Q45jrq8v49nbNgvB3NUvZDpE=; b=OIyTiwxXJqYc3bc9z1xzDbAx7FM9nAkzRPHyDtJIRdGq6ztkqjlpNChexJ8dz97+EZ xI6n+oe6pP2IjyjxwMAXY77W/LFD9T1rm5WosyZU8zKnkCfLAkE8YTMvUJxm07U6pVl+ Elkv4P3Ho8v4reQSSvv6b/sKTUOt+nqnYnualvVbdB/JIiKEFrbh6ff33OjmgtRQqvay ndyT0L4MAnrS8cvXIatnzQYAhtRiRsgul2ZVaIngLnZTr0fNCQwASQ5pjvaHwiTKWjd+ qKO05GNsTKkvpyIyhtkpNCYSCo/g8vMxjlNntRHhHXh93ewQs0pnso2BBpaVxaWfC7iy +pMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352645; x=1759957445; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2tGgNTZuDE42TuUV9n3Q45jrq8v49nbNgvB3NUvZDpE=; b=pHXMx+5zhvG89Zn5uxN6euC48wK5aSb0fhf8K8LOBP1Y50Pa4T4Y+V76TeEunowoTS 9hGrPlWwSFND2MbNJan2DwqAhMfmA3uKxQQ/JCe8uWlBPwh/gyt/QlnjqLl5AQ0OoYlr IKIV4nF/SjWEu4l33b8FxZ+iBYEUTbcf/+nOSZCgKFe3UVApxwjzZDOgOCuVhghk/6UU YjZj00r9N7vQQXKfrkqqX6lj++8igLmTFOSVdkIW9y5otcUbfd5WgYNUuZiZHjqSAxzE qUsVD/ma5UJuOnoqvPrX7Eh+oByD4/jbiGI+FVL5+rORE05gj4hMW41jxi+vwnesf5MO 7mnA== X-Forwarded-Encrypted: i=1; AJvYcCUB/wHUEdKRq7HxrGS9MJa+vaXZdu23o/BEwLcYr6s6Slv0/5U2nSltcysTyGbWkF4Kzn1QexPUUsl4s0U=@vger.kernel.org X-Gm-Message-State: AOJu0YyK3RXJK4PwWRjaKnindq4EK6KfMg0DhvAqZuE0FjKk3FFiVl0e dtNT+F7yqWuRVyQEMid4KoMdgr+1zn6D7J5HF1tQZsgSQ9rNAtiBjXxwwypSEOBdXMmgsGD3pQ= = X-Google-Smtp-Source: AGHT+IGRMPSEJRGhX6bzJNMLIpuTgPU8l+6UbKzu27Pb1Dg+NmBxLA1o3IMc5SIrwprjppFeYEZeOCGP X-Received: from edvd18.prod.google.com ([2002:aa7:ce12:0:b0:633:c29c:e02b]) (user=ardb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:1c91:b0:b45:b1f2:fab4 with SMTP id a640c23a62f3a-b46e4d7bd97mr600170166b.12.1759352644910; Wed, 01 Oct 2025 14:04:04 -0700 (PDT) Date: Wed, 1 Oct 2025 23:02:14 +0200 In-Reply-To: <20251001210201.838686-22-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251001210201.838686-22-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=20375; i=ardb@kernel.org; h=from:subject; bh=SDCOXpC6hIQ3gVfCMmGTmKQTQWVfl68LHJ4l6pbLEnM=; b=owGbwMvMwCVmkMcZplerG8N4Wi2JIePutL8d7XOnC9eGvfaNMHSbsXBGcCnntPo0b63wLwlpc 3qCixs6SlkYxLgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwETC1jAyTPwwP7vSzti9yfX9 Wf2bOtVNSdKrvtx8fV3Y+Jee5Ona+4wMx/oL2dSL8jZpOfsv57HlZ/N50jJtZ/XSnvK5sTf8bc1 5AA== X-Mailer: git-send-email 2.51.0.618.g983fd99d29-goog Message-ID: <20251001210201.838686-34-ardb+git@google.com> Subject: [PATCH v2 12/20] crypto/arm64: aes-blk - Switch to 'ksimd' scoped guard API From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, linux@armlinux.org.uk, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ard Biesheuvel Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-glue.c | 87 ++++++------ arch/arm64/crypto/aes-glue.c | 139 ++++++++---------- arch/arm64/crypto/aes-neonbs-glue.c | 150 ++++++++++---------- 3 files changed, 181 insertions(+), 195 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-glue.c b/arch/arm64/crypto/aes-ce-glu= e.c index 00b8749013c5..a4dad370991d 100644 --- a/arch/arm64/crypto/aes-ce-glue.c +++ b/arch/arm64/crypto/aes-ce-glue.c @@ -52,9 +52,8 @@ static void aes_cipher_encrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_encrypt(ctx->key_enc, dst, src, num_rounds(ctx)); } =20 static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8 dst[], u8 const = src[]) @@ -66,9 +65,8 @@ static void aes_cipher_decrypt(struct crypto_tfm *tfm, u8= dst[], u8 const src[]) return; } =20 - kernel_neon_begin(); - __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); - kernel_neon_end(); + scoped_ksimd() + __aes_ce_decrypt(ctx->key_dec, dst, src, num_rounds(ctx)); } =20 int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key, @@ -94,47 +92,48 @@ int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const = u8 *in_key, for (i =3D 0; i < kwords; i++) ctx->key_enc[i] =3D get_unaligned_le32(in_key + i * sizeof(u32)); =20 - kernel_neon_begin(); - for (i =3D 0; i < sizeof(rcon); i++) { - u32 *rki =3D ctx->key_enc + (i * kwords); - u32 *rko =3D rki + kwords; - - rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ rcon[i] ^ rki[0]; - rko[1] =3D rko[0] ^ rki[1]; - rko[2] =3D rko[1] ^ rki[2]; - rko[3] =3D rko[2] ^ rki[3]; - - if (key_len =3D=3D AES_KEYSIZE_192) { - if (i >=3D 7) - break; - rko[4] =3D rko[3] ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - } else if (key_len =3D=3D AES_KEYSIZE_256) { - if (i >=3D 6) - break; - rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; - rko[5] =3D rko[4] ^ rki[5]; - rko[6] =3D rko[5] ^ rki[6]; - rko[7] =3D rko[6] ^ rki[7]; + scoped_ksimd() { + for (i =3D 0; i < sizeof(rcon); i++) { + u32 *rki =3D ctx->key_enc + (i * kwords); + u32 *rko =3D rki + kwords; + + rko[0] =3D ror32(__aes_ce_sub(rki[kwords - 1]), 8) ^ + rcon[i] ^ rki[0]; + rko[1] =3D rko[0] ^ rki[1]; + rko[2] =3D rko[1] ^ rki[2]; + rko[3] =3D rko[2] ^ rki[3]; + + if (key_len =3D=3D AES_KEYSIZE_192) { + if (i >=3D 7) + break; + rko[4] =3D rko[3] ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + } else if (key_len =3D=3D AES_KEYSIZE_256) { + if (i >=3D 6) + break; + rko[4] =3D __aes_ce_sub(rko[3]) ^ rki[4]; + rko[5] =3D rko[4] ^ rki[5]; + rko[6] =3D rko[5] ^ rki[6]; + rko[7] =3D rko[6] ^ rki[7]; + } } - } =20 - /* - * Generate the decryption keys for the Equivalent Inverse Cipher. - * This involves reversing the order of the round keys, and applying - * the Inverse Mix Columns transformation on all but the first and - * the last one. - */ - key_enc =3D (struct aes_block *)ctx->key_enc; - key_dec =3D (struct aes_block *)ctx->key_dec; - j =3D num_rounds(ctx); - - key_dec[0] =3D key_enc[j]; - for (i =3D 1, j--; j > 0; i++, j--) - __aes_ce_invert(key_dec + i, key_enc + j); - key_dec[i] =3D key_enc[0]; + /* + * Generate the decryption keys for the Equivalent Inverse + * Cipher. This involves reversing the order of the round + * keys, and applying the Inverse Mix Columns transformation on + * all but the first and the last one. + */ + key_enc =3D (struct aes_block *)ctx->key_enc; + key_dec =3D (struct aes_block *)ctx->key_dec; + j =3D num_rounds(ctx); + + key_dec[0] =3D key_enc[j]; + for (i =3D 1, j--; j > 0; i++, j--) + __aes_ce_invert(key_dec + i, key_enc + j); + key_dec[i] =3D key_enc[0]; + } =20 - kernel_neon_end(); return 0; } EXPORT_SYMBOL(ce_aes_expandkey); diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c index 81560f722b9d..4b76d8dfa2c5 100644 --- a/arch/arm64/crypto/aes-glue.c +++ b/arch/arm64/crypto/aes-glue.c @@ -5,8 +5,6 @@ * Copyright (C) 2013 - 2017 Linaro Ltd */ =20 -#include -#include #include #include #include @@ -20,6 +18,9 @@ #include #include =20 +#include +#include + #include "aes-ce-setkey.h" =20 #ifdef USE_V8_CRYPTO_EXTENSIONS @@ -187,10 +188,9 @@ static int __maybe_unused ecb_encrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -207,10 +207,9 @@ static int __maybe_unused ecb_decrypt(struct skcipher_= request *req) err =3D skcipher_walk_virt(&walk, req, false); =20 while ((blocks =3D (walk.nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -225,10 +224,9 @@ static int cbc_encrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_enc, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_encrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_enc, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -254,10 +252,9 @@ static int cbc_decrypt_walk(struct skcipher_request *r= eq, unsigned int blocks; =20 while ((blocks =3D (walk->nbytes / AES_BLOCK_SIZE))) { - kernel_neon_begin(); - aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, - ctx->key_dec, rounds, blocks, walk->iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_decrypt(walk->dst.virt.addr, walk->src.virt.addr, + ctx->key_dec, rounds, blocks, walk->iv); err =3D skcipher_walk_done(walk, walk->nbytes % AES_BLOCK_SIZE); } return err; @@ -323,10 +320,9 @@ static int cts_cbc_encrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_enc, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_enc, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -380,10 +376,9 @@ static int cts_cbc_decrypt(struct skcipher_request *re= q) if (err) return err; =20 - kernel_neon_begin(); - aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key_dec, rounds, walk.nbytes, walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_cbc_cts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key_dec, rounds, walk.nbytes, walk.iv); =20 return skcipher_walk_done(&walk, 0); } @@ -416,11 +411,11 @@ static int __maybe_unused essiv_cbc_encrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_enc, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_encrypt_walk(req, &walk); @@ -438,11 +433,11 @@ static int __maybe_unused essiv_cbc_decrypt(struct sk= cipher_request *req) =20 blocks =3D walk.nbytes / AES_BLOCK_SIZE; if (blocks) { - kernel_neon_begin(); - aes_essiv_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, blocks, - req->iv, ctx->key2.key_enc); - kernel_neon_end(); + scoped_ksimd() + aes_essiv_cbc_decrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->key1.key_dec, rounds, blocks, + req->iv, ctx->key2.key_enc); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err ?: cbc_decrypt_walk(req, &walk); @@ -478,10 +473,9 @@ static int __maybe_unused xctr_encrypt(struct skcipher= _request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv, byte_ctr); - kernel_neon_end(); + scoped_ksimd() + aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv, byte_ctr); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -523,10 +517,9 @@ static int __maybe_unused ctr_encrypt(struct skcipher_= request *req) else if (nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aes_ctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes, + walk.iv); =20 if (unlikely(nbytes < AES_BLOCK_SIZE)) memcpy(walk.dst.virt.addr, @@ -579,11 +572,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -601,11 +593,10 @@ static int __maybe_unused xts_encrypt(struct skcipher= _request *req) if (err) return err; =20 - kernel_neon_begin(); - aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_enc, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_enc, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -651,11 +642,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) if (walk.nbytes < walk.total) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, nbytes, + ctx->key2.key_enc, walk.iv, first); err =3D skcipher_walk_done(&walk, walk.nbytes - nbytes); } =20 @@ -674,11 +664,10 @@ static int __maybe_unused xts_decrypt(struct skcipher= _request *req) return err; =20 =20 - kernel_neon_begin(); - aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key1.key_dec, rounds, walk.nbytes, - ctx->key2.key_enc, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() + aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key1.key_dec, rounds, walk.nbytes, + ctx->key2.key_enc, walk.iv, first); =20 return skcipher_walk_done(&walk, 0); } @@ -827,10 +816,9 @@ static int cmac_setkey(struct crypto_shash *tfm, const= u8 *in_key, return err; =20 /* encrypt the zero vector */ - kernel_neon_begin(); - aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, ctx->key.key_enc, - rounds, 1); - kernel_neon_end(); + scoped_ksimd() + aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){}, + ctx->key.key_enc, rounds, 1); =20 cmac_gf128_mul_by_x(consts, consts); cmac_gf128_mul_by_x(consts + 1, consts); @@ -856,10 +844,10 @@ static int xcbc_setkey(struct crypto_shash *tfm, cons= t u8 *in_key, if (err) return err; =20 - kernel_neon_begin(); - aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); - aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); - kernel_neon_end(); + scoped_ksimd() { + aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1); + aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2); + } =20 return cbcmac_setkey(tfm, key, sizeof(key)); } @@ -879,10 +867,9 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, = u8 const in[], int blocks, int rem; =20 do { - kernel_neon_begin(); - rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, - dg, enc_before, !enc_before); - kernel_neon_end(); + scoped_ksimd() + rem =3D aes_mac_update(in, ctx->key_enc, rounds, blocks, + dg, enc_before, !enc_before); in +=3D (blocks - rem) * AES_BLOCK_SIZE; blocks =3D rem; } while (blocks); diff --git a/arch/arm64/crypto/aes-neonbs-glue.c b/arch/arm64/crypto/aes-ne= onbs-glue.c index c4a623e86593..d496effb0a5b 100644 --- a/arch/arm64/crypto/aes-neonbs-glue.c +++ b/arch/arm64/crypto/aes-neonbs-glue.c @@ -85,9 +85,8 @@ static int aesbs_setkey(struct crypto_skcipher *tfm, cons= t u8 *in_key, =20 ctx->rounds =3D 6 + key_len / 4; =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->rk, rk.key_enc, ctx->rounds); =20 return 0; } @@ -110,10 +109,9 @@ static int __ecb_crypt(struct skcipher_request *req, blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, - ctx->rounds, blocks); - kernel_neon_end(); + scoped_ksimd() + fn(walk.dst.virt.addr, walk.src.virt.addr, ctx->rk, + ctx->rounds, blocks); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -146,9 +144,8 @@ static int aesbs_cbc_ctr_setkey(struct crypto_skcipher = *tfm, const u8 *in_key, =20 memcpy(ctx->enc, rk.key_enc, sizeof(ctx->enc)); =20 - kernel_neon_begin(); - aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); - kernel_neon_end(); + scoped_ksimd() + aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds); memzero_explicit(&rk, sizeof(rk)); =20 return 0; @@ -167,11 +164,11 @@ static int cbc_encrypt(struct skcipher_request *req) unsigned int blocks =3D walk.nbytes / AES_BLOCK_SIZE; =20 /* fall back to the non-bitsliced NEON implementation */ - kernel_neon_begin(); - neon_aes_cbc_encrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->enc, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + neon_aes_cbc_encrypt(walk.dst.virt.addr, + walk.src.virt.addr, + ctx->enc, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes % AES_BLOCK_SIZE); } return err; @@ -193,11 +190,10 @@ static int cbc_decrypt(struct skcipher_request *req) blocks =3D round_down(blocks, walk.stride / AES_BLOCK_SIZE); =20 - kernel_neon_begin(); - aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, - ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - kernel_neon_end(); + scoped_ksimd() + aesbs_cbc_decrypt(walk.dst.virt.addr, walk.src.virt.addr, + ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); err =3D skcipher_walk_done(&walk, walk.nbytes - blocks * AES_BLOCK_SIZE); } @@ -220,30 +216,32 @@ static int ctr_encrypt(struct skcipher_request *req) const u8 *src =3D walk.src.virt.addr; u8 *dst =3D walk.dst.virt.addr; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - aesbs_ctr_encrypt(dst, src, ctx->key.rk, ctx->key.rounds, - blocks, walk.iv); - dst +=3D blocks * AES_BLOCK_SIZE; - src +=3D blocks * AES_BLOCK_SIZE; - } - if (nbytes && walk.nbytes =3D=3D walk.total) { - u8 buf[AES_BLOCK_SIZE]; - u8 *d =3D dst; - - if (unlikely(nbytes < AES_BLOCK_SIZE)) - src =3D dst =3D memcpy(buf + sizeof(buf) - nbytes, - src, nbytes); - - neon_aes_ctr_encrypt(dst, src, ctx->enc, ctx->key.rounds, - nbytes, walk.iv); + scoped_ksimd() { + if (blocks >=3D 8) { + aesbs_ctr_encrypt(dst, src, ctx->key.rk, + ctx->key.rounds, blocks, + walk.iv); + dst +=3D blocks * AES_BLOCK_SIZE; + src +=3D blocks * AES_BLOCK_SIZE; + } + if (nbytes && walk.nbytes =3D=3D walk.total) { + u8 buf[AES_BLOCK_SIZE]; + u8 *d =3D dst; + + if (unlikely(nbytes < AES_BLOCK_SIZE)) + src =3D dst =3D memcpy(buf + sizeof(buf) - + nbytes, src, nbytes); + + neon_aes_ctr_encrypt(dst, src, ctx->enc, + ctx->key.rounds, nbytes, + walk.iv); =20 - if (unlikely(nbytes < AES_BLOCK_SIZE)) - memcpy(d, dst, nbytes); + if (unlikely(nbytes < AES_BLOCK_SIZE)) + memcpy(d, dst, nbytes); =20 - nbytes =3D 0; + nbytes =3D 0; + } } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } return err; @@ -320,33 +318,33 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (blocks >=3D 8) { - if (first =3D=3D 1) - neon_aes_ecb_encrypt(walk.iv, walk.iv, - ctx->twkey, - ctx->key.rounds, 1); - first =3D 2; - - fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, - walk.iv); - - out +=3D blocks * AES_BLOCK_SIZE; - in +=3D blocks * AES_BLOCK_SIZE; - nbytes -=3D blocks * AES_BLOCK_SIZE; + scoped_ksimd() { + if (blocks >=3D 8) { + if (first =3D=3D 1) + neon_aes_ecb_encrypt(walk.iv, walk.iv, + ctx->twkey, + ctx->key.rounds, 1); + first =3D 2; + + fn(out, in, ctx->key.rk, ctx->key.rounds, blocks, + walk.iv); + + out +=3D blocks * AES_BLOCK_SIZE; + in +=3D blocks * AES_BLOCK_SIZE; + nbytes -=3D blocks * AES_BLOCK_SIZE; + } + if (walk.nbytes =3D=3D walk.total && nbytes > 0) { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, + ctx->twkey, walk.iv, first); + nbytes =3D first =3D 0; + } } - if (walk.nbytes =3D=3D walk.total && nbytes > 0) { - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, - ctx->key.rounds, nbytes, - ctx->twkey, walk.iv, first); - nbytes =3D first =3D 0; - } - kernel_neon_end(); err =3D skcipher_walk_done(&walk, nbytes); } =20 @@ -369,14 +367,16 @@ static int __xts_crypt(struct skcipher_request *req, = bool encrypt, in =3D walk.src.virt.addr; nbytes =3D walk.nbytes; =20 - kernel_neon_begin(); - if (encrypt) - neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - else - neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, ctx->key.rounds, - nbytes, ctx->twkey, walk.iv, first); - kernel_neon_end(); + scoped_ksimd() { + if (encrypt) + neon_aes_xts_encrypt(out, in, ctx->cts.key_enc, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + else + neon_aes_xts_decrypt(out, in, ctx->cts.key_dec, + ctx->key.rounds, nbytes, ctx->twkey, + walk.iv, first); + } =20 return skcipher_walk_done(&walk, 0); } --=20 2.51.0.618.g983fd99d29-goog