From nobody Fri Feb 13 06:57:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4816CCE7B0A for ; Thu, 28 Sep 2023 07:39:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230502AbjI1HjJ (ORCPT ); Thu, 28 Sep 2023 03:39:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230294AbjI1HjB (ORCPT ); Thu, 28 Sep 2023 03:39:01 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 218DD9F; Thu, 28 Sep 2023 00:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695886740; x=1727422740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hTsR4bc4ZnGNeFWQnKoWjgv1oT3/1emfbnvRjX1H0pw=; b=ItpswK/xPLnNH0C9vlYV0o+OjmhZ7B6mS61K6zDH9Tgz3CCiQ2HeDh4L 2a0B51Gfmi89gWh6FNmtItlr8jeL3gpnaUKNVcVzMz0qUmN29iok+hqHX 6ygHdPkNSFvnYpuYFFRhg2lE1qrluZmv3qI0VXaGbV5R5va9DB1u4vghg NNZkX0GXdurqFlZuCdXnkc7J1/SrnK/842CGtOf1jQJT7D2NqEuMJZWyt vTxLYb53XvBQ7LPDnVS0F0ChL7T+rHilfdSU43EgDcrzoCsOhPGZGPcot N7E7QJ6sBIeWK1CS5xG/9wCurcR8lw31Gtah7zVHr7bJ6KHOJIiUb5jPo w==; X-IronPort-AV: E=McAfee;i="6600,9927,10846"; a="367064861" X-IronPort-AV: E=Sophos;i="6.03,183,1694761200"; d="scan'208";a="367064861" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Sep 2023 00:38:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10846"; a="742990421" X-IronPort-AV: E=Sophos;i="6.03,183,1694761200"; d="scan'208";a="742990421" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by orsmga007.jf.intel.com with ESMTP; 28 Sep 2023 00:38:58 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, davem@davemloft.net, ebiggers@kernel.org, x86@kernel.org, chang.seok.bae@intel.com Subject: [PATCH v2 1/3] crypto: x86/aesni - Refactor the common address alignment code Date: Thu, 28 Sep 2023 00:25:06 -0700 Message-Id: <20230928072508.218368-2-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230928072508.218368-1-chang.seok.bae@intel.com> References: <20230925151752.162449-1-chang.seok.bae@intel.com> <20230928072508.218368-1-chang.seok.bae@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The address alignment code has been duplicated for each mode. Instead of duplicating the same code, refactor the alignment code and simplify the alignment helpers. Suggested-by: Eric Biggers Link: https://lore.kernel.org/all/20230526065414.GB875@sol.localdomain/ Signed-off-by: Chang S. Bae Cc: linux-crypto@vger.kernel.org Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- V1->V2: drop the casts (Eric). --- arch/x86/crypto/aesni-intel_glue.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 39d6a62ac627..308deeb0c974 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -80,6 +80,13 @@ struct gcm_context_data { u8 hash_keys[GCM_BLOCK_LEN * 16]; }; =20 +static inline void *aes_align_addr(void *addr) +{ + if (crypto_tfm_ctx_alignment() >=3D AESNI_ALIGN) + return addr; + return PTR_ALIGN(addr, AESNI_ALIGN); +} + asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len); asmlinkage void aesni_enc(const void *ctx, u8 *out, const u8 *in); @@ -201,32 +208,19 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(gcm_us= e_avx2); static inline struct aesni_rfc4106_gcm_ctx *aesni_rfc4106_gcm_ctx_get(struct crypto_aead *tfm) { - unsigned long align =3D AESNI_ALIGN; - - if (align <=3D crypto_tfm_ctx_alignment()) - align =3D 1; - return PTR_ALIGN(crypto_aead_ctx(tfm), align); + return aes_align_addr(crypto_aead_ctx(tfm)); } =20 static inline struct generic_gcmaes_ctx *generic_gcmaes_ctx_get(struct crypto_aead *tfm) { - unsigned long align =3D AESNI_ALIGN; - - if (align <=3D crypto_tfm_ctx_alignment()) - align =3D 1; - return PTR_ALIGN(crypto_aead_ctx(tfm), align); + return aes_align_addr(crypto_aead_ctx(tfm)); } #endif =20 static inline struct crypto_aes_ctx *aes_ctx(void *raw_ctx) { - unsigned long addr =3D (unsigned long)raw_ctx; - unsigned long align =3D AESNI_ALIGN; - - if (align <=3D crypto_tfm_ctx_alignment()) - align =3D 1; - return (struct crypto_aes_ctx *)ALIGN(addr, align); + return aes_align_addr(raw_ctx); } =20 static int aes_set_key_common(struct crypto_aes_ctx *ctx, --=20 2.34.1 From nobody Fri Feb 13 06:57:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E4F5CE7B09 for ; Thu, 28 Sep 2023 07:39:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230488AbjI1HjH (ORCPT ); Thu, 28 Sep 2023 03:39:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230238AbjI1HjB (ORCPT ); Thu, 28 Sep 2023 03:39:01 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F6928F; Thu, 28 Sep 2023 00:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695886740; x=1727422740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F6NgVD08pFaNhfYaWXL6vs/EBfbIYirHydA4wVPI/g0=; b=X3MrSzlFxhQY55MBD6P+jYyXpKnprHj54fwBn8hn9u4U3VCwbhev3qna loIDgZzGblzrqBHPB9Y/eUT8zci5Tj13k7883a9lZrvcy38AgV84tNSt9 ozysf7GM02Kt4DTZonM2wTS9Il2tG7wOakrrZxg2pRvJlJv5+cPuJK70g uffN8I7sA8crFSDWYQJk2yhVsyxPYzH8Uur3lMpwqW5TtfTDG2YflmTri XZ1B/bgSzwysKzQpgBZojP85NFopR4PsS299Y1mBHSWFJI6Uft0L0uplK OAJ9gKfwkeEVW5jhJdWxmB+X0J2LAWd2CHojvJywYYGm/F7oC7S4MN4uU g==; X-IronPort-AV: E=McAfee;i="6600,9927,10846"; a="367064867" X-IronPort-AV: E=Sophos;i="6.03,183,1694761200"; d="scan'208";a="367064867" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Sep 2023 00:38:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10846"; a="742990426" X-IronPort-AV: E=Sophos;i="6.03,183,1694761200"; d="scan'208";a="742990426" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by orsmga007.jf.intel.com with ESMTP; 28 Sep 2023 00:38:58 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, davem@davemloft.net, ebiggers@kernel.org, x86@kernel.org, chang.seok.bae@intel.com Subject: [PATCH v2 2/3] crypto: x86/aesni - Correct the data type in struct aesni_xts_ctx Date: Thu, 28 Sep 2023 00:25:07 -0700 Message-Id: <20230928072508.218368-3-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230928072508.218368-1-chang.seok.bae@intel.com> References: <20230925151752.162449-1-chang.seok.bae@intel.com> <20230928072508.218368-1-chang.seok.bae@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, every field in struct aesni_xts_ctx is defined as a byte array of the same size as struct crypto_aes_ctx. This data type is obscure and the choice lacks justification. To rectify this, update the field type in struct aesni_xts_ctx to match its actual structure. Suggested-by: Eric Biggers Link: https://lore.kernel.org/all/ZFWQ4sZEVu%2FLHq+Q@gmail.com/ Signed-off-by: Chang S. Bae Cc: linux-crypto@vger.kernel.org Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- arch/x86/crypto/aesni-intel_glue.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 308deeb0c974..80e28a01aa3a 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -61,8 +61,8 @@ struct generic_gcmaes_ctx { }; =20 struct aesni_xts_ctx { - u8 raw_tweak_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; - u8 raw_crypt_ctx[sizeof(struct crypto_aes_ctx)] AESNI_ALIGN_ATTR; + struct crypto_aes_ctx tweak_ctx AESNI_ALIGN_ATTR; + struct crypto_aes_ctx crypt_ctx AESNI_ALIGN_ATTR; }; =20 #define GCM_BLOCK_LEN 16 @@ -885,13 +885,12 @@ static int xts_aesni_setkey(struct crypto_skcipher *t= fm, const u8 *key, keylen /=3D 2; =20 /* first half of xts-key is for crypt */ - err =3D aes_set_key_common(aes_ctx(ctx->raw_crypt_ctx), key, keylen); + err =3D aes_set_key_common(aes_ctx(&ctx->crypt_ctx), key, keylen); if (err) return err; =20 /* second half of xts-key is for tweak */ - return aes_set_key_common(aes_ctx(ctx->raw_tweak_ctx), key + keylen, - keylen); + return aes_set_key_common(aes_ctx(&ctx->tweak_ctx), key + keylen, keylen); } =20 static int xts_crypt(struct skcipher_request *req, bool encrypt) @@ -933,7 +932,7 @@ static int xts_crypt(struct skcipher_request *req, bool= encrypt) kernel_fpu_begin(); =20 /* calculate first value of T */ - aesni_enc(aes_ctx(ctx->raw_tweak_ctx), walk.iv, walk.iv); + aesni_enc(aes_ctx(&ctx->tweak_ctx), walk.iv, walk.iv); =20 while (walk.nbytes > 0) { int nbytes =3D walk.nbytes; @@ -942,11 +941,11 @@ static int xts_crypt(struct skcipher_request *req, bo= ol encrypt) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), + aesni_xts_encrypt(aes_ctx(&ctx->crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, nbytes, walk.iv); else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), + aesni_xts_decrypt(aes_ctx(&ctx->crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, nbytes, walk.iv); kernel_fpu_end(); @@ -974,11 +973,11 @@ static int xts_crypt(struct skcipher_request *req, bo= ol encrypt) =20 kernel_fpu_begin(); if (encrypt) - aesni_xts_encrypt(aes_ctx(ctx->raw_crypt_ctx), + aesni_xts_encrypt(aes_ctx(&ctx->crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes, walk.iv); else - aesni_xts_decrypt(aes_ctx(ctx->raw_crypt_ctx), + aesni_xts_decrypt(aes_ctx(&ctx->crypt_ctx), walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes, walk.iv); kernel_fpu_end(); --=20 2.34.1 From nobody Fri Feb 13 06:57:23 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB3BCCE7B09 for ; Thu, 28 Sep 2023 07:39:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230517AbjI1HjM (ORCPT ); Thu, 28 Sep 2023 03:39:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230437AbjI1HjE (ORCPT ); Thu, 28 Sep 2023 03:39:04 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC59B97; Thu, 28 Sep 2023 00:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695886741; x=1727422741; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mw6amrAjr2FBcC2ESow/ci1eT9Msi63HLvtWx6Vudl4=; b=VvcmoCVRbUaXbgt3gdty7cYGyRei8e9fcYOVm8z1B3HaIR8edMwN81Jv EqdiwOFyYxC2mq/gqGUhGB8KROFDhdh8vtNGrfjGdYox0RTsyoKxscNr7 pvDAwOjKyumjlVq6DVaOdIiGIUtoPqZfQqcbzwNcb/CqTNisfcSz+oKzD yYQjhebkK7Gsw0b2wmkr5LRw+P+8TuvsXELP2H8iYOS0t/NeDPY7HQINE gicQc+fdkGhejtDjDnBD9suxCeP/F4fBHv/Ipg2k3tnCRjogEOu+3tjfz eEsyvNgvZrGfU1lasK5v4edwaR8QJxe9CL/6G8EolBGHyLpJ32MePCQQZ Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10846"; a="367064871" X-IronPort-AV: E=Sophos;i="6.03,183,1694761200"; d="scan'208";a="367064871" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Sep 2023 00:38:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10846"; a="742990432" X-IronPort-AV: E=Sophos;i="6.03,183,1694761200"; d="scan'208";a="742990432" Received: from chang-linux-3.sc.intel.com ([172.25.66.173]) by orsmga007.jf.intel.com with ESMTP; 28 Sep 2023 00:38:58 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org Cc: herbert@gondor.apana.org.au, davem@davemloft.net, ebiggers@kernel.org, x86@kernel.org, chang.seok.bae@intel.com Subject: [PATCH v2 3/3] crypto: x86/aesni - Perform address alignment early for XTS mode Date: Thu, 28 Sep 2023 00:25:08 -0700 Message-Id: <20230928072508.218368-4-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230928072508.218368-1-chang.seok.bae@intel.com> References: <20230925151752.162449-1-chang.seok.bae@intel.com> <20230928072508.218368-1-chang.seok.bae@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, the alignment of each field in struct aesni_xts_ctx occurs right before every access. However, it's possible to perform this alignment ahead of time. Introduce a helper function that converts struct crypto_skcipher *tfm to struct aesni_xts_ctx *ctx and returns an aligned address. Utilize this helper function at the beginning of each XTS function and then eliminate redundant alignment code. Suggested-by: Eric Biggers Link: https://lore.kernel.org/all/ZFWQ4sZEVu%2FLHq+Q@gmail.com/ Signed-off-by: Chang S. Bae Cc: linux-crypto@vger.kernel.org Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org --- V1->V2: drop the cast (Eric). --- arch/x86/crypto/aesni-intel_glue.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index 80e28a01aa3a..b1d90c25975a 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -223,6 +223,11 @@ static inline struct crypto_aes_ctx *aes_ctx(void *raw= _ctx) return aes_align_addr(raw_ctx); } =20 +static inline struct aesni_xts_ctx *aes_xts_ctx(struct crypto_skcipher *tf= m) +{ + return aes_align_addr(crypto_skcipher_ctx(tfm)); +} + static int aes_set_key_common(struct crypto_aes_ctx *ctx, const u8 *in_key, unsigned int key_len) { @@ -875,7 +880,7 @@ static int helper_rfc4106_decrypt(struct aead_request *= req) static int xts_aesni_setkey(struct crypto_skcipher *tfm, const u8 *key, unsigned int keylen) { - struct aesni_xts_ctx *ctx =3D crypto_skcipher_ctx(tfm); + struct aesni_xts_ctx *ctx =3D aes_xts_ctx(tfm); int err; =20 err =3D xts_verify_key(tfm, key, keylen); @@ -885,18 +890,18 @@ static int xts_aesni_setkey(struct crypto_skcipher *t= fm, const u8 *key, keylen /=3D 2; =20 /* first half of xts-key is for crypt */ - err =3D aes_set_key_common(aes_ctx(&ctx->crypt_ctx), key, keylen); + err =3D aes_set_key_common(&ctx->crypt_ctx, key, keylen); if (err) return err; =20 /* second half of xts-key is for tweak */ - return aes_set_key_common(aes_ctx(&ctx->tweak_ctx), key + keylen, keylen); + return aes_set_key_common(&ctx->tweak_ctx, key + keylen, keylen); } =20 static int xts_crypt(struct skcipher_request *req, bool encrypt) { struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); - struct aesni_xts_ctx *ctx =3D crypto_skcipher_ctx(tfm); + struct aesni_xts_ctx *ctx =3D aes_xts_ctx(tfm); int tail =3D req->cryptlen % AES_BLOCK_SIZE; struct skcipher_request subreq; struct skcipher_walk walk; @@ -932,7 +937,7 @@ static int xts_crypt(struct skcipher_request *req, bool= encrypt) kernel_fpu_begin(); =20 /* calculate first value of T */ - aesni_enc(aes_ctx(&ctx->tweak_ctx), walk.iv, walk.iv); + aesni_enc(&ctx->tweak_ctx, walk.iv, walk.iv); =20 while (walk.nbytes > 0) { int nbytes =3D walk.nbytes; @@ -941,11 +946,11 @@ static int xts_crypt(struct skcipher_request *req, bo= ol encrypt) nbytes &=3D ~(AES_BLOCK_SIZE - 1); =20 if (encrypt) - aesni_xts_encrypt(aes_ctx(&ctx->crypt_ctx), + aesni_xts_encrypt(&ctx->crypt_ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes, walk.iv); else - aesni_xts_decrypt(aes_ctx(&ctx->crypt_ctx), + aesni_xts_decrypt(&ctx->crypt_ctx, walk.dst.virt.addr, walk.src.virt.addr, nbytes, walk.iv); kernel_fpu_end(); @@ -973,11 +978,11 @@ static int xts_crypt(struct skcipher_request *req, bo= ol encrypt) =20 kernel_fpu_begin(); if (encrypt) - aesni_xts_encrypt(aes_ctx(&ctx->crypt_ctx), + aesni_xts_encrypt(&ctx->crypt_ctx, walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes, walk.iv); else - aesni_xts_decrypt(aes_ctx(&ctx->crypt_ctx), + aesni_xts_decrypt(&ctx->crypt_ctx, walk.dst.virt.addr, walk.src.virt.addr, walk.nbytes, walk.iv); kernel_fpu_end(); --=20 2.34.1