From nobody Fri Dec 19 08:06:18 2025 Received: from fd01.gateway.ufhost.com (fd01.gateway.ufhost.com [61.152.239.71]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CA9B2D7A7; Sat, 16 Dec 2023 14:13:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com Received: from EXMBX166.cuchost.com (unknown [175.102.18.54]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "EXMBX166", Issuer "EXMBX166" (not verified)) by fd01.gateway.ufhost.com (Postfix) with ESMTP id 343F824DB84; Sat, 16 Dec 2023 22:12:54 +0800 (CST) Received: from EXMBX068.cuchost.com (172.16.6.68) by EXMBX166.cuchost.com (172.16.6.76) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:12:54 +0800 Received: from ubuntu.localdomain (161.142.156.108) by EXMBX068.cuchost.com (172.16.6.68) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:12:51 +0800 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Rob Herring , Krzysztof Kozlowski , Conor Dooley , , , Subject: [PATCH 1/5] dt-bindings: crypto: starfive: Add jh8100 compatible string Date: Sat, 16 Dec 2023 22:12:30 +0800 Message-ID: <20231216141234.417498-2-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231216141234.417498-1-jiajie.ho@starfivetech.com> References: <20231216141234.417498-1-jiajie.ho@starfivetech.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: EXCAS066.cuchost.com (172.16.6.26) To EXMBX068.cuchost.com (172.16.6.68) X-YovoleRuleAgent: yovoleflag Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add compatible string for StarFive JH8100 crypto hardware. Signed-off-by: Jia Jie Ho Acked-by: Conor Dooley --- .../devicetree/bindings/crypto/starfive,jh7110-crypto.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Documentation/devicetree/bindings/crypto/starfive,jh7110-crypt= o.yaml b/Documentation/devicetree/bindings/crypto/starfive,jh7110-crypto.ya= ml index 71a2876bd6e4..3b14320a107f 100644 --- a/Documentation/devicetree/bindings/crypto/starfive,jh7110-crypto.yaml +++ b/Documentation/devicetree/bindings/crypto/starfive,jh7110-crypto.yaml @@ -12,7 +12,11 @@ maintainers: =20 properties: compatible: - const: starfive,jh7110-crypto + oneOf: + - items: + - const: starfive,jh8100-crypto + - const: starfive,jh7110-crypto + - const: starfive,jh7110-crypto =20 reg: maxItems: 1 --=20 2.34.1 From nobody Fri Dec 19 08:06:18 2025 Received: from ex01.ufhost.com (ex01.ufhost.com [61.152.239.75]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EC1D2D7A6; Sat, 16 Dec 2023 14:13:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com Received: from EXMBX165.cuchost.com (unknown [175.102.18.54]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "EXMBX165", Issuer "EXMBX165" (not verified)) by ex01.ufhost.com (Postfix) with ESMTP id ECF4524DFCE; Sat, 16 Dec 2023 22:12:59 +0800 (CST) Received: from EXMBX068.cuchost.com (172.16.6.68) by EXMBX165.cuchost.com (172.16.6.75) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:12:59 +0800 Received: from ubuntu.localdomain (161.142.156.108) by EXMBX068.cuchost.com (172.16.6.68) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:12:54 +0800 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Rob Herring , Krzysztof Kozlowski , Conor Dooley , , , Subject: [PATCH 2/5] crypto: starfive: Update hash dma usage Date: Sat, 16 Dec 2023 22:12:31 +0800 Message-ID: <20231216141234.417498-3-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231216141234.417498-1-jiajie.ho@starfivetech.com> References: <20231216141234.417498-1-jiajie.ho@starfivetech.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: EXCAS066.cuchost.com (172.16.6.26) To EXMBX068.cuchost.com (172.16.6.68) X-YovoleRuleAgent: yovoleflag Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current hash uses sw fallback for non-word aligned input scatterlists. Add support for unaligned cases utilizing the data valid mask for dma. Signed-off-by: Jia Jie Ho --- drivers/crypto/starfive/jh7110-cryp.h | 1 + drivers/crypto/starfive/jh7110-hash.c | 257 ++++++++++---------------- 2 files changed, 100 insertions(+), 158 deletions(-) diff --git a/drivers/crypto/starfive/jh7110-cryp.h b/drivers/crypto/starfiv= e/jh7110-cryp.h index 6cdf6db5d904..4940cd1a3fbb 100644 --- a/drivers/crypto/starfive/jh7110-cryp.h +++ b/drivers/crypto/starfive/jh7110-cryp.h @@ -190,6 +190,7 @@ struct starfive_cryp_dev { struct crypto_engine *engine; struct tasklet_struct aes_done; struct tasklet_struct hash_done; + struct completion dma_done; size_t assoclen; size_t total_in; size_t total_out; diff --git a/drivers/crypto/starfive/jh7110-hash.c b/drivers/crypto/starfiv= e/jh7110-hash.c index b6d1808012ca..74e151b5f875 100644 --- a/drivers/crypto/starfive/jh7110-hash.c +++ b/drivers/crypto/starfive/jh7110-hash.c @@ -86,62 +86,31 @@ static int starfive_hash_hmac_key(struct starfive_cryp_= ctx *ctx) =20 static void starfive_hash_start(void *param) { - struct starfive_cryp_ctx *ctx =3D param; - struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; - struct starfive_cryp_dev *cryp =3D ctx->cryp; - union starfive_alg_cr alg_cr; + struct starfive_cryp_dev *cryp =3D param; union starfive_hash_csr csr; - u32 stat; - - dma_unmap_sg(cryp->dev, rctx->in_sg, rctx->in_sg_len, DMA_TO_DEVICE); - - alg_cr.v =3D 0; - alg_cr.clear =3D 1; - - writel(alg_cr.v, cryp->base + STARFIVE_ALG_CR_OFFSET); + u32 mask; =20 csr.v =3D readl(cryp->base + STARFIVE_HASH_SHACSR); csr.firstb =3D 0; csr.final =3D 1; - - stat =3D readl(cryp->base + STARFIVE_IE_MASK_OFFSET); - stat &=3D ~STARFIVE_IE_MASK_HASH_DONE; - writel(stat, cryp->base + STARFIVE_IE_MASK_OFFSET); + csr.ie =3D 1; writel(csr.v, cryp->base + STARFIVE_HASH_SHACSR); + + mask =3D readl(cryp->base + STARFIVE_IE_MASK_OFFSET); + mask &=3D ~STARFIVE_IE_MASK_HASH_DONE; + writel(mask, cryp->base + STARFIVE_IE_MASK_OFFSET); } =20 -static int starfive_hash_xmit_dma(struct starfive_cryp_ctx *ctx) +static void starfive_hash_dma_callback(void *param) { - struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; - struct starfive_cryp_dev *cryp =3D ctx->cryp; - struct dma_async_tx_descriptor *in_desc; - union starfive_alg_cr alg_cr; - int total_len; - int ret; - - if (!rctx->total) { - starfive_hash_start(ctx); - return 0; - } + struct starfive_cryp_dev *cryp =3D param; =20 - writel(rctx->total, cryp->base + STARFIVE_DMA_IN_LEN_OFFSET); - - total_len =3D rctx->total; - total_len =3D (total_len & 0x3) ? (((total_len >> 2) + 1) << 2) : total_l= en; - sg_dma_len(rctx->in_sg) =3D total_len; - - alg_cr.v =3D 0; - alg_cr.start =3D 1; - alg_cr.hash_dma_en =3D 1; - - writel(alg_cr.v, cryp->base + STARFIVE_ALG_CR_OFFSET); - - ret =3D dma_map_sg(cryp->dev, rctx->in_sg, rctx->in_sg_len, DMA_TO_DEVICE= ); - if (!ret) - return dev_err_probe(cryp->dev, -EINVAL, "dma_map_sg() error\n"); + complete(&cryp->dma_done); +} =20 - cryp->cfg_in.direction =3D DMA_MEM_TO_DEV; - cryp->cfg_in.src_addr_width =3D DMA_SLAVE_BUSWIDTH_4_BYTES; +static void starfive_hash_dma_init(struct starfive_cryp_dev *cryp) +{ + cryp->cfg_in.src_addr_width =3D DMA_SLAVE_BUSWIDTH_16_BYTES; cryp->cfg_in.dst_addr_width =3D DMA_SLAVE_BUSWIDTH_4_BYTES; cryp->cfg_in.src_maxburst =3D cryp->dma_maxburst; cryp->cfg_in.dst_maxburst =3D cryp->dma_maxburst; @@ -149,50 +118,48 @@ static int starfive_hash_xmit_dma(struct starfive_cry= p_ctx *ctx) =20 dmaengine_slave_config(cryp->tx, &cryp->cfg_in); =20 - in_desc =3D dmaengine_prep_slave_sg(cryp->tx, rctx->in_sg, - ret, DMA_MEM_TO_DEV, - DMA_PREP_INTERRUPT | DMA_CTRL_ACK); - - if (!in_desc) - return -EINVAL; - - in_desc->callback =3D starfive_hash_start; - in_desc->callback_param =3D ctx; - - dmaengine_submit(in_desc); - dma_async_issue_pending(cryp->tx); - - return 0; + init_completion(&cryp->dma_done); } =20 -static int starfive_hash_xmit(struct starfive_cryp_ctx *ctx) +static int starfive_hash_dma_xfer(struct starfive_cryp_dev *cryp, + struct scatterlist *sg) { - struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; - struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct dma_async_tx_descriptor *in_desc; + union starfive_alg_cr alg_cr; int ret =3D 0; =20 - rctx->csr.hash.v =3D 0; - rctx->csr.hash.reset =3D 1; - writel(rctx->csr.hash.v, cryp->base + STARFIVE_HASH_SHACSR); - - if (starfive_hash_wait_busy(ctx)) - return dev_err_probe(cryp->dev, -ETIMEDOUT, "Error resetting engine.\n"); + alg_cr.v =3D 0; + alg_cr.start =3D 1; + alg_cr.hash_dma_en =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_ALG_CR_OFFSET); =20 - rctx->csr.hash.v =3D 0; - rctx->csr.hash.mode =3D ctx->hash_mode; - rctx->csr.hash.ie =3D 1; + writel(sg_dma_len(sg), cryp->base + STARFIVE_DMA_IN_LEN_OFFSET); + sg_dma_len(sg) =3D ALIGN(sg_dma_len(sg), sizeof(u32)); =20 - if (ctx->is_hmac) { - ret =3D starfive_hash_hmac_key(ctx); - if (ret) - return ret; - } else { - rctx->csr.hash.start =3D 1; - rctx->csr.hash.firstb =3D 1; - writel(rctx->csr.hash.v, cryp->base + STARFIVE_HASH_SHACSR); + in_desc =3D dmaengine_prep_slave_sg(cryp->tx, sg, 1, DMA_MEM_TO_DEV, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!in_desc) { + ret =3D -EINVAL; + goto end; } =20 - return starfive_hash_xmit_dma(ctx); + reinit_completion(&cryp->dma_done); + in_desc->callback =3D starfive_hash_dma_callback; + in_desc->callback_param =3D cryp; + + dmaengine_submit(in_desc); + dma_async_issue_pending(cryp->tx); + + if (!wait_for_completion_timeout(&cryp->dma_done, + msecs_to_jiffies(1000))) + ret =3D -ETIMEDOUT; + +end: + alg_cr.v =3D 0; + alg_cr.clear =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_ALG_CR_OFFSET); + + return ret; } =20 static int starfive_hash_copy_hash(struct ahash_request *req) @@ -229,44 +196,56 @@ void starfive_hash_done_task(unsigned long param) crypto_finalize_hash_request(cryp->engine, cryp->req.hreq, err); } =20 -static int starfive_hash_check_aligned(struct scatterlist *sg, size_t tota= l, size_t align) +static int starfive_hash_one_request(struct crypto_engine *engine, void *a= req) { - int len =3D 0; - - if (!total) - return 0; + struct ahash_request *req =3D container_of(areq, struct ahash_request, + base); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(crypto_ahash_reqtfm(re= q)); + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct scatterlist *tsg; + int ret, src_nents, i; =20 - if (!IS_ALIGNED(total, align)) - return -EINVAL; + writel(STARFIVE_HASH_RESET, cryp->base + STARFIVE_HASH_SHACSR); =20 - while (sg) { - if (!IS_ALIGNED(sg->offset, sizeof(u32))) - return -EINVAL; + if (starfive_hash_wait_busy(ctx)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, "Error resetting hardware.\n= "); =20 - if (!IS_ALIGNED(sg->length, align)) - return -EINVAL; + rctx->csr.hash.v =3D 0; + rctx->csr.hash.mode =3D ctx->hash_mode; =20 - len +=3D sg->length; - sg =3D sg_next(sg); + if (ctx->is_hmac) { + ret =3D starfive_hash_hmac_key(ctx); + if (ret) + return ret; + } else { + rctx->csr.hash.start =3D 1; + rctx->csr.hash.firstb =3D 1; + writel(rctx->csr.hash.v, cryp->base + STARFIVE_HASH_SHACSR); } =20 - if (len !=3D total) - return -EINVAL; + /* No input message, get digest and end. */ + if (!rctx->total) + goto hash_start; =20 - return 0; -} + starfive_hash_dma_init(cryp); =20 -static int starfive_hash_one_request(struct crypto_engine *engine, void *a= req) -{ - struct ahash_request *req =3D container_of(areq, struct ahash_request, - base); - struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(crypto_ahash_reqtfm(re= q)); - struct starfive_cryp_dev *cryp =3D ctx->cryp; + for_each_sg(rctx->in_sg, tsg, rctx->in_sg_len, i) { + src_nents =3D dma_map_sg(cryp->dev, tsg, 1, DMA_TO_DEVICE); + if (src_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg error\n"); =20 - if (!cryp) - return -ENODEV; + ret =3D starfive_hash_dma_xfer(cryp, tsg); + dma_unmap_sg(cryp->dev, tsg, 1, DMA_TO_DEVICE); + if (ret) + return ret; + } + +hash_start: + starfive_hash_start(cryp); =20 - return starfive_hash_xmit(ctx); + return 0; } =20 static int starfive_hash_init(struct ahash_request *req) @@ -337,22 +316,6 @@ static int starfive_hash_finup(struct ahash_request *r= eq) return crypto_ahash_finup(&rctx->ahash_fbk_req); } =20 -static int starfive_hash_digest_fb(struct ahash_request *req) -{ - struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); - struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); - struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); - - ahash_request_set_tfm(&rctx->ahash_fbk_req, ctx->ahash_fbk); - ahash_request_set_callback(&rctx->ahash_fbk_req, req->base.flags, - req->base.complete, req->base.data); - - ahash_request_set_crypt(&rctx->ahash_fbk_req, req->src, - req->result, req->nbytes); - - return crypto_ahash_digest(&rctx->ahash_fbk_req); -} - static int starfive_hash_digest(struct ahash_request *req) { struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); @@ -370,9 +333,6 @@ static int starfive_hash_digest(struct ahash_request *r= eq) rctx->in_sg_len =3D sg_nents_for_len(rctx->in_sg, rctx->total); ctx->rctx =3D rctx; =20 - if (starfive_hash_check_aligned(rctx->in_sg, rctx->total, rctx->blksize)) - return starfive_hash_digest_fb(req); - return crypto_transfer_hash_request_to_engine(cryp->engine, req); } =20 @@ -406,7 +366,8 @@ static int starfive_hash_import(struct ahash_request *r= eq, const void *in) =20 static int starfive_hash_init_tfm(struct crypto_ahash *hash, const char *alg_name, - unsigned int mode) + unsigned int mode, + bool is_hmac) { struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); =20 @@ -426,7 +387,7 @@ static int starfive_hash_init_tfm(struct crypto_ahash *= hash, crypto_ahash_set_reqsize(hash, sizeof(struct starfive_cryp_request_ctx) + crypto_ahash_reqsize(ctx->ahash_fbk)); =20 - ctx->keylen =3D 0; + ctx->is_hmac =3D is_hmac; ctx->hash_mode =3D mode; =20 return 0; @@ -529,81 +490,61 @@ static int starfive_hash_setkey(struct crypto_ahash *= hash, static int starfive_sha224_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "sha224-generic", - STARFIVE_HASH_SHA224); + STARFIVE_HASH_SHA224, 0); } =20 static int starfive_sha256_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "sha256-generic", - STARFIVE_HASH_SHA256); + STARFIVE_HASH_SHA256, 0); } =20 static int starfive_sha384_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "sha384-generic", - STARFIVE_HASH_SHA384); + STARFIVE_HASH_SHA384, 0); } =20 static int starfive_sha512_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "sha512-generic", - STARFIVE_HASH_SHA512); + STARFIVE_HASH_SHA512, 0); } =20 static int starfive_sm3_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "sm3-generic", - STARFIVE_HASH_SM3); + STARFIVE_HASH_SM3, 0); } =20 static int starfive_hmac_sha224_init_tfm(struct crypto_ahash *hash) { - struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); - - ctx->is_hmac =3D true; - return starfive_hash_init_tfm(hash, "hmac(sha224-generic)", - STARFIVE_HASH_SHA224); + STARFIVE_HASH_SHA224, 1); } =20 static int starfive_hmac_sha256_init_tfm(struct crypto_ahash *hash) { - struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); - - ctx->is_hmac =3D true; - return starfive_hash_init_tfm(hash, "hmac(sha256-generic)", - STARFIVE_HASH_SHA256); + STARFIVE_HASH_SHA256, 1); } =20 static int starfive_hmac_sha384_init_tfm(struct crypto_ahash *hash) { - struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); - - ctx->is_hmac =3D true; - return starfive_hash_init_tfm(hash, "hmac(sha384-generic)", - STARFIVE_HASH_SHA384); + STARFIVE_HASH_SHA384, 1); } =20 static int starfive_hmac_sha512_init_tfm(struct crypto_ahash *hash) { - struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); - - ctx->is_hmac =3D true; - return starfive_hash_init_tfm(hash, "hmac(sha512-generic)", - STARFIVE_HASH_SHA512); + STARFIVE_HASH_SHA512, 1); } =20 static int starfive_hmac_sm3_init_tfm(struct crypto_ahash *hash) { - struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); - - ctx->is_hmac =3D true; - return starfive_hash_init_tfm(hash, "hmac(sm3-generic)", - STARFIVE_HASH_SM3); + STARFIVE_HASH_SM3, 1); } =20 static struct ahash_engine_alg algs_sha2_sm3[] =3D { --=20 2.34.1 From nobody Fri Dec 19 08:06:18 2025 Received: from ex01.ufhost.com (ex01.ufhost.com [61.152.239.75]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0F9530F88; Sat, 16 Dec 2023 14:13:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com Received: from EXMBX166.cuchost.com (unknown [175.102.18.54]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "EXMBX166", Issuer "EXMBX166" (not verified)) by ex01.ufhost.com (Postfix) with ESMTP id 3287D24E083; Sat, 16 Dec 2023 22:13:08 +0800 (CST) Received: from EXMBX068.cuchost.com (172.16.6.68) by EXMBX166.cuchost.com (172.16.6.76) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:13:08 +0800 Received: from ubuntu.localdomain (161.142.156.108) by EXMBX068.cuchost.com (172.16.6.68) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:12:59 +0800 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Rob Herring , Krzysztof Kozlowski , Conor Dooley , , , Subject: [PATCH 3/5] crypto: starfive: Use dma for aes requests Date: Sat, 16 Dec 2023 22:12:32 +0800 Message-ID: <20231216141234.417498-4-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231216141234.417498-1-jiajie.ho@starfivetech.com> References: <20231216141234.417498-1-jiajie.ho@starfivetech.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: EXCAS066.cuchost.com (172.16.6.26) To EXMBX068.cuchost.com (172.16.6.68) X-YovoleRuleAgent: yovoleflag Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert AES module to use dma for data transfers to reduce cpu load and compatible with future variants. Signed-off-by: Jia Jie Ho --- drivers/crypto/starfive/Kconfig | 4 + drivers/crypto/starfive/jh7110-aes.c | 583 +++++++++++++++++--------- drivers/crypto/starfive/jh7110-cryp.c | 9 - drivers/crypto/starfive/jh7110-cryp.h | 3 +- 4 files changed, 392 insertions(+), 207 deletions(-) diff --git a/drivers/crypto/starfive/Kconfig b/drivers/crypto/starfive/Kcon= fig index cb59357b58b2..0fe389e9f932 100644 --- a/drivers/crypto/starfive/Kconfig +++ b/drivers/crypto/starfive/Kconfig @@ -14,6 +14,10 @@ config CRYPTO_DEV_JH7110 select CRYPTO_RSA select CRYPTO_AES select CRYPTO_CCM + select CRYPTO_GCM + select CRYPTO_ECB + select CRYPTO_CBC + select CRYPTO_CTR help Support for StarFive JH7110 crypto hardware acceleration engine. This module provides acceleration for public key algo, diff --git a/drivers/crypto/starfive/jh7110-aes.c b/drivers/crypto/starfive= /jh7110-aes.c index 1ac15cc4ef3c..151d161459ff 100644 --- a/drivers/crypto/starfive/jh7110-aes.c +++ b/drivers/crypto/starfive/jh7110-aes.c @@ -78,7 +78,7 @@ static inline int is_gcm(struct starfive_cryp_dev *cryp) return (cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_AES_MODE_GCM; } =20 -static inline int is_encrypt(struct starfive_cryp_dev *cryp) +static inline bool is_encrypt(struct starfive_cryp_dev *cryp) { return cryp->flags & FLG_ENCRYPT; } @@ -103,16 +103,6 @@ static void starfive_aes_aead_hw_start(struct starfive= _cryp_ctx *ctx, u32 hw_mod } } =20 -static inline void starfive_aes_set_ivlen(struct starfive_cryp_ctx *ctx) -{ - struct starfive_cryp_dev *cryp =3D ctx->cryp; - - if (is_gcm(cryp)) - writel(GCM_AES_IV_SIZE, cryp->base + STARFIVE_AES_IVLEN); - else - writel(AES_BLOCK_SIZE, cryp->base + STARFIVE_AES_IVLEN); -} - static inline void starfive_aes_set_alen(struct starfive_cryp_ctx *ctx) { struct starfive_cryp_dev *cryp =3D ctx->cryp; @@ -261,7 +251,6 @@ static int starfive_aes_hw_init(struct starfive_cryp_ct= x *ctx) =20 rctx->csr.aes.mode =3D hw_mode; rctx->csr.aes.cmode =3D !is_encrypt(cryp); - rctx->csr.aes.ie =3D 1; rctx->csr.aes.stmode =3D STARFIVE_AES_MODE_XFB_1; =20 if (cryp->side_chan) { @@ -279,7 +268,7 @@ static int starfive_aes_hw_init(struct starfive_cryp_ct= x *ctx) case STARFIVE_AES_MODE_GCM: starfive_aes_set_alen(ctx); starfive_aes_set_mlen(ctx); - starfive_aes_set_ivlen(ctx); + writel(GCM_AES_IV_SIZE, cryp->base + STARFIVE_AES_IVLEN); starfive_aes_aead_hw_start(ctx, hw_mode); starfive_aes_write_iv(ctx, (void *)cryp->req.areq->iv); break; @@ -300,28 +289,30 @@ static int starfive_aes_hw_init(struct starfive_cryp_= ctx *ctx) return cryp->err; } =20 -static int starfive_aes_read_authtag(struct starfive_cryp_dev *cryp) +static int starfive_aes_read_authtag(struct starfive_cryp_ctx *ctx) { - int i, start_addr; + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + int i; =20 if (starfive_aes_wait_busy(cryp)) return dev_err_probe(cryp->dev, -ETIMEDOUT, "Timeout waiting for tag generation."); =20 - start_addr =3D STARFIVE_AES_NONCE0; - - if (is_gcm(cryp)) - for (i =3D 0; i < AES_BLOCK_32; i++, start_addr +=3D 4) - cryp->tag_out[i] =3D readl(cryp->base + start_addr); - else + if ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_AES_MODE_GCM) { + cryp->tag_out[0] =3D readl(cryp->base + STARFIVE_AES_NONCE0); + cryp->tag_out[1] =3D readl(cryp->base + STARFIVE_AES_NONCE1); + cryp->tag_out[2] =3D readl(cryp->base + STARFIVE_AES_NONCE2); + cryp->tag_out[3] =3D readl(cryp->base + STARFIVE_AES_NONCE3); + } else { for (i =3D 0; i < AES_BLOCK_32; i++) cryp->tag_out[i] =3D readl(cryp->base + STARFIVE_AES_AESDIO0R); + } =20 if (is_encrypt(cryp)) { - scatterwalk_copychunks(cryp->tag_out, &cryp->out_walk, cryp->authsize, 1= ); + scatterwalk_map_and_copy(cryp->tag_out, rctx->out_sg, + cryp->total_in, cryp->authsize, 1); } else { - scatterwalk_copychunks(cryp->tag_in, &cryp->in_walk, cryp->authsize, 0); - if (crypto_memneq(cryp->tag_in, cryp->tag_out, cryp->authsize)) return dev_err_probe(cryp->dev, -EBADMSG, "Failed tag verification\n"); } @@ -329,23 +320,18 @@ static int starfive_aes_read_authtag(struct starfive_= cryp_dev *cryp) return 0; } =20 -static void starfive_aes_finish_req(struct starfive_cryp_dev *cryp) +static void starfive_aes_finish_req(struct starfive_cryp_ctx *ctx) { - union starfive_aes_csr csr; + struct starfive_cryp_dev *cryp =3D ctx->cryp; int err =3D cryp->err; =20 if (!err && cryp->authsize) - err =3D starfive_aes_read_authtag(cryp); + err =3D starfive_aes_read_authtag(ctx); =20 if (!err && ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_AES_MODE_CBC || (cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_AES_MODE_CTR)) starfive_aes_get_iv(cryp, (void *)cryp->req.sreq->iv); =20 - /* reset irq flags*/ - csr.v =3D 0; - csr.aesrst =3D 1; - writel(csr.v, cryp->base + STARFIVE_AES_CSR); - if (cryp->authsize) crypto_finalize_aead_request(cryp->engine, cryp->req.areq, err); else @@ -353,39 +339,6 @@ static void starfive_aes_finish_req(struct starfive_cr= yp_dev *cryp) err); } =20 -void starfive_aes_done_task(unsigned long param) -{ - struct starfive_cryp_dev *cryp =3D (struct starfive_cryp_dev *)param; - u32 block[AES_BLOCK_32]; - u32 stat; - int i; - - for (i =3D 0; i < AES_BLOCK_32; i++) - block[i] =3D readl(cryp->base + STARFIVE_AES_AESDIO0R); - - scatterwalk_copychunks(block, &cryp->out_walk, min_t(size_t, AES_BLOCK_SI= ZE, - cryp->total_out), 1); - - cryp->total_out -=3D min_t(size_t, AES_BLOCK_SIZE, cryp->total_out); - - if (!cryp->total_out) { - starfive_aes_finish_req(cryp); - return; - } - - memset(block, 0, AES_BLOCK_SIZE); - scatterwalk_copychunks(block, &cryp->in_walk, min_t(size_t, AES_BLOCK_SIZ= E, - cryp->total_in), 0); - cryp->total_in -=3D min_t(size_t, AES_BLOCK_SIZE, cryp->total_in); - - for (i =3D 0; i < AES_BLOCK_32; i++) - writel(block[i], cryp->base + STARFIVE_AES_AESDIO0R); - - stat =3D readl(cryp->base + STARFIVE_IE_MASK_OFFSET); - stat &=3D ~STARFIVE_IE_MASK_AES_DONE; - writel(stat, cryp->base + STARFIVE_IE_MASK_OFFSET); -} - static int starfive_aes_gcm_write_adata(struct starfive_cryp_ctx *ctx) { struct starfive_cryp_dev *cryp =3D ctx->cryp; @@ -451,60 +404,165 @@ static int starfive_aes_ccm_write_adata(struct starf= ive_cryp_ctx *ctx) return 0; } =20 -static int starfive_aes_prepare_req(struct skcipher_request *req, - struct aead_request *areq) +static void starfive_aes_dma_done(void *param) { - struct starfive_cryp_ctx *ctx; - struct starfive_cryp_request_ctx *rctx; - struct starfive_cryp_dev *cryp; + struct starfive_cryp_dev *cryp =3D param; =20 - if (!req && !areq) - return -EINVAL; + complete(&cryp->dma_done); +} =20 - ctx =3D req ? crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)) : - crypto_aead_ctx(crypto_aead_reqtfm(areq)); +static void starfive_aes_dma_init(struct starfive_cryp_dev *cryp) +{ + cryp->cfg_in.direction =3D DMA_MEM_TO_DEV; + cryp->cfg_in.src_addr_width =3D DMA_SLAVE_BUSWIDTH_16_BYTES; + cryp->cfg_in.dst_addr_width =3D DMA_SLAVE_BUSWIDTH_4_BYTES; + cryp->cfg_in.src_maxburst =3D cryp->dma_maxburst; + cryp->cfg_in.dst_maxburst =3D cryp->dma_maxburst; + cryp->cfg_in.dst_addr =3D cryp->phys_base + STARFIVE_ALG_FIFO_OFFSET; =20 - cryp =3D ctx->cryp; - rctx =3D req ? skcipher_request_ctx(req) : aead_request_ctx(areq); + dmaengine_slave_config(cryp->tx, &cryp->cfg_in); =20 - if (req) { - cryp->req.sreq =3D req; - cryp->total_in =3D req->cryptlen; - cryp->total_out =3D req->cryptlen; - cryp->assoclen =3D 0; - cryp->authsize =3D 0; - } else { - cryp->req.areq =3D areq; - cryp->assoclen =3D areq->assoclen; - cryp->authsize =3D crypto_aead_authsize(crypto_aead_reqtfm(areq)); - if (is_encrypt(cryp)) { - cryp->total_in =3D areq->cryptlen; - cryp->total_out =3D areq->cryptlen; - } else { - cryp->total_in =3D areq->cryptlen - cryp->authsize; - cryp->total_out =3D cryp->total_in; - } - } + cryp->cfg_out.direction =3D DMA_DEV_TO_MEM; + cryp->cfg_out.src_addr_width =3D DMA_SLAVE_BUSWIDTH_4_BYTES; + cryp->cfg_out.dst_addr_width =3D DMA_SLAVE_BUSWIDTH_16_BYTES; + cryp->cfg_out.src_maxburst =3D 4; + cryp->cfg_out.dst_maxburst =3D 4; + cryp->cfg_out.src_addr =3D cryp->phys_base + STARFIVE_ALG_FIFO_OFFSET; =20 - rctx->in_sg =3D req ? req->src : areq->src; - scatterwalk_start(&cryp->in_walk, rctx->in_sg); + dmaengine_slave_config(cryp->rx, &cryp->cfg_out); =20 - rctx->out_sg =3D req ? req->dst : areq->dst; - scatterwalk_start(&cryp->out_walk, rctx->out_sg); + init_completion(&cryp->dma_done); +} =20 - if (cryp->assoclen) { - rctx->adata =3D kzalloc(cryp->assoclen + AES_BLOCK_SIZE, GFP_KERNEL); - if (!rctx->adata) - return dev_err_probe(cryp->dev, -ENOMEM, - "Failed to alloc memory for adata"); +static int starfive_aes_dma_xfer(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst, + int len) +{ + struct dma_async_tx_descriptor *in_desc, *out_desc; + union starfive_alg_cr alg_cr; + int ret =3D 0, in_save, out_save; + + alg_cr.v =3D 0; + alg_cr.start =3D 1; + alg_cr.aes_dma_en =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_ALG_CR_OFFSET); =20 - scatterwalk_copychunks(rctx->adata, &cryp->in_walk, cryp->assoclen, 0); - scatterwalk_copychunks(NULL, &cryp->out_walk, cryp->assoclen, 2); + in_save =3D sg_dma_len(src); + out_save =3D sg_dma_len(dst); + + writel(ALIGN(len, AES_BLOCK_SIZE), cryp->base + STARFIVE_DMA_IN_LEN_OFFSE= T); + writel(ALIGN(len, AES_BLOCK_SIZE), cryp->base + STARFIVE_DMA_OUT_LEN_OFFS= ET); + + sg_dma_len(src) =3D ALIGN(len, AES_BLOCK_SIZE); + sg_dma_len(dst) =3D ALIGN(len, AES_BLOCK_SIZE); + + out_desc =3D dmaengine_prep_slave_sg(cryp->rx, dst, 1, DMA_DEV_TO_MEM, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!out_desc) { + ret =3D -EINVAL; + goto dma_err; } =20 - ctx->rctx =3D rctx; + out_desc->callback =3D starfive_aes_dma_done; + out_desc->callback_param =3D cryp; + + reinit_completion(&cryp->dma_done); + dmaengine_submit(out_desc); + dma_async_issue_pending(cryp->rx); + + in_desc =3D dmaengine_prep_slave_sg(cryp->tx, src, 1, DMA_MEM_TO_DEV, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!in_desc) { + ret =3D -EINVAL; + goto dma_err; + } + + dmaengine_submit(in_desc); + dma_async_issue_pending(cryp->tx); + + if (!wait_for_completion_timeout(&cryp->dma_done, + msecs_to_jiffies(1000))) + ret =3D -ETIMEDOUT; =20 - return starfive_aes_hw_init(ctx); +dma_err: + sg_dma_len(src) =3D in_save; + sg_dma_len(dst) =3D out_save; + + alg_cr.v =3D 0; + alg_cr.clear =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_ALG_CR_OFFSET); + + return ret; +} + +static int starfive_aes_map_sg(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst) +{ + struct scatterlist *stsg, *dtsg; + struct scatterlist _src[2], _dst[2]; + unsigned int remain =3D cryp->total_in; + unsigned int len, src_nents, dst_nents; + int ret; + + if (src =3D=3D dst) { + for (stsg =3D src, dtsg =3D dst; remain > 0; + stsg =3D sg_next(stsg), dtsg =3D sg_next(dtsg)) { + src_nents =3D dma_map_sg(cryp->dev, stsg, 1, DMA_BIDIRECTIONAL); + if (src_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg error\n"); + + dst_nents =3D src_nents; + len =3D min(sg_dma_len(stsg), remain); + + ret =3D starfive_aes_dma_xfer(cryp, stsg, dtsg, len); + dma_unmap_sg(cryp->dev, stsg, 1, DMA_BIDIRECTIONAL); + if (ret) + return ret; + + remain -=3D len; + } + } else { + for (stsg =3D src, dtsg =3D dst;;) { + src_nents =3D dma_map_sg(cryp->dev, stsg, 1, DMA_TO_DEVICE); + if (src_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg src error\n"); + + dst_nents =3D dma_map_sg(cryp->dev, dtsg, 1, DMA_FROM_DEVICE); + if (dst_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg dst error\n"); + + len =3D min(sg_dma_len(stsg), sg_dma_len(dtsg)); + len =3D min(len, remain); + + ret =3D starfive_aes_dma_xfer(cryp, stsg, dtsg, len); + dma_unmap_sg(cryp->dev, stsg, 1, DMA_TO_DEVICE); + dma_unmap_sg(cryp->dev, dtsg, 1, DMA_FROM_DEVICE); + if (ret) + return ret; + + remain -=3D len; + if (remain =3D=3D 0) + break; + + if (sg_dma_len(stsg) - len) { + stsg =3D scatterwalk_ffwd(_src, stsg, len); + dtsg =3D sg_next(dtsg); + } else if (sg_dma_len(dtsg) - len) { + dtsg =3D scatterwalk_ffwd(_dst, dtsg, len); + stsg =3D sg_next(stsg); + } else { + stsg =3D sg_next(stsg); + dtsg =3D sg_next(dtsg); + } + } + } + + return 0; } =20 static int starfive_aes_do_one_req(struct crypto_engine *engine, void *are= q) @@ -513,35 +571,38 @@ static int starfive_aes_do_one_req(struct crypto_engi= ne *engine, void *areq) container_of(areq, struct skcipher_request, base); struct starfive_cryp_ctx *ctx =3D crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)); + struct starfive_cryp_request_ctx *rctx =3D skcipher_request_ctx(req); struct starfive_cryp_dev *cryp =3D ctx->cryp; - u32 block[AES_BLOCK_32]; - u32 stat; - int err; - int i; + int ret; =20 - err =3D starfive_aes_prepare_req(req, NULL); - if (err) - return err; + cryp->req.sreq =3D req; + cryp->total_in =3D req->cryptlen; + cryp->total_out =3D req->cryptlen; + cryp->assoclen =3D 0; + cryp->authsize =3D 0; =20 - /* - * Write first plain/ciphertext block to start the module - * then let irq tasklet handle the rest of the data blocks. - */ - scatterwalk_copychunks(block, &cryp->in_walk, min_t(size_t, AES_BLOCK_SIZ= E, - cryp->total_in), 0); - cryp->total_in -=3D min_t(size_t, AES_BLOCK_SIZE, cryp->total_in); + rctx->in_sg =3D req->src; + rctx->out_sg =3D req->dst; + + ctx->rctx =3D rctx; + + ret =3D starfive_aes_hw_init(ctx); + if (ret) + return ret; =20 - for (i =3D 0; i < AES_BLOCK_32; i++) - writel(block[i], cryp->base + STARFIVE_AES_AESDIO0R); + starfive_aes_dma_init(cryp); =20 - stat =3D readl(cryp->base + STARFIVE_IE_MASK_OFFSET); - stat &=3D ~STARFIVE_IE_MASK_AES_DONE; - writel(stat, cryp->base + STARFIVE_IE_MASK_OFFSET); + ret =3D starfive_aes_map_sg(cryp, rctx->in_sg, rctx->out_sg); + if (ret) + return ret; + + starfive_aes_finish_req(ctx); =20 return 0; } =20 -static int starfive_aes_init_tfm(struct crypto_skcipher *tfm) +static int starfive_aes_init_tfm(struct crypto_skcipher *tfm, + const char *alg_name) { struct starfive_cryp_ctx *ctx =3D crypto_skcipher_ctx(tfm); =20 @@ -549,12 +610,26 @@ static int starfive_aes_init_tfm(struct crypto_skciph= er *tfm) if (!ctx->cryp) return -ENODEV; =20 + ctx->skcipher_fbk =3D crypto_alloc_skcipher(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + if (IS_ERR(ctx->skcipher_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->skcipher_fbk), + "%s() failed to allocate fallback for %s\n", + __func__, alg_name); + crypto_skcipher_set_reqsize(tfm, sizeof(struct starfive_cryp_request_ctx)= + sizeof(struct skcipher_request)); =20 return 0; } =20 +static void starfive_aes_exit_tfm(struct crypto_skcipher *tfm) +{ + struct starfive_cryp_ctx *ctx =3D crypto_skcipher_ctx(tfm); + + crypto_free_skcipher(ctx->skcipher_fbk); +} + static int starfive_aes_aead_do_one_req(struct crypto_engine *engine, void= *areq) { struct aead_request *req =3D @@ -562,76 +637,96 @@ static int starfive_aes_aead_do_one_req(struct crypto= _engine *engine, void *areq struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(crypto_aead_reqtfm(req)); struct starfive_cryp_dev *cryp =3D ctx->cryp; - struct starfive_cryp_request_ctx *rctx; - u32 block[AES_BLOCK_32]; - u32 stat; - int err; - int i; + struct starfive_cryp_request_ctx *rctx =3D aead_request_ctx(req); + struct scatterlist _src[2], _dst[2]; + int ret; + + cryp->req.areq =3D req; + cryp->assoclen =3D req->assoclen; + cryp->authsize =3D crypto_aead_authsize(crypto_aead_reqtfm(req)); + + rctx->in_sg =3D scatterwalk_ffwd(_src, req->src, cryp->assoclen); + if (req->src =3D=3D req->dst) + rctx->out_sg =3D rctx->in_sg; + else + rctx->out_sg =3D scatterwalk_ffwd(_dst, req->dst, cryp->assoclen); =20 - err =3D starfive_aes_prepare_req(NULL, req); - if (err) - return err; + if (is_encrypt(cryp)) { + cryp->total_in =3D req->cryptlen; + cryp->total_out =3D req->cryptlen; + } else { + cryp->total_in =3D req->cryptlen - cryp->authsize; + cryp->total_out =3D cryp->total_in; + scatterwalk_map_and_copy(cryp->tag_in, req->src, + cryp->total_in + cryp->assoclen, + cryp->authsize, 0); + } =20 - rctx =3D ctx->rctx; + if (cryp->assoclen) { + rctx->adata =3D kzalloc(cryp->assoclen + AES_BLOCK_SIZE, GFP_KERNEL); + if (!rctx->adata) + return dev_err_probe(cryp->dev, -ENOMEM, + "Failed to alloc memory for adata"); + + if (sg_copy_to_buffer(req->src, sg_nents_for_len(req->src, cryp->assocle= n), + rctx->adata, cryp->assoclen) !=3D cryp->assoclen) + return -EINVAL; + } + + if (cryp->total_in) + sg_zero_buffer(rctx->in_sg, sg_nents(rctx->in_sg), + sg_dma_len(rctx->in_sg) - cryp->total_in, + cryp->total_in); + + ctx->rctx =3D rctx; + + ret =3D starfive_aes_hw_init(ctx); + if (ret) + return ret; =20 if (!cryp->assoclen) goto write_text; =20 if ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_AES_MODE_CCM) - cryp->err =3D starfive_aes_ccm_write_adata(ctx); + ret =3D starfive_aes_ccm_write_adata(ctx); else - cryp->err =3D starfive_aes_gcm_write_adata(ctx); + ret =3D starfive_aes_gcm_write_adata(ctx); =20 kfree(rctx->adata); =20 - if (cryp->err) - return cryp->err; + if (ret) + return ret; =20 write_text: if (!cryp->total_in) goto finish_req; =20 - /* - * Write first plain/ciphertext block to start the module - * then let irq tasklet handle the rest of the data blocks. - */ - scatterwalk_copychunks(block, &cryp->in_walk, min_t(size_t, AES_BLOCK_SIZ= E, - cryp->total_in), 0); - cryp->total_in -=3D min_t(size_t, AES_BLOCK_SIZE, cryp->total_in); - - for (i =3D 0; i < AES_BLOCK_32; i++) - writel(block[i], cryp->base + STARFIVE_AES_AESDIO0R); + starfive_aes_dma_init(cryp); =20 - stat =3D readl(cryp->base + STARFIVE_IE_MASK_OFFSET); - stat &=3D ~STARFIVE_IE_MASK_AES_DONE; - writel(stat, cryp->base + STARFIVE_IE_MASK_OFFSET); - - return 0; + ret =3D starfive_aes_map_sg(cryp, rctx->in_sg, rctx->out_sg); + if (ret) + return ret; =20 finish_req: - starfive_aes_finish_req(cryp); + starfive_aes_finish_req(ctx); return 0; } =20 -static int starfive_aes_aead_init_tfm(struct crypto_aead *tfm) +static int starfive_aes_aead_init_tfm(struct crypto_aead *tfm, + const char *alg_name) { struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(tfm); - struct starfive_cryp_dev *cryp =3D ctx->cryp; - struct crypto_tfm *aead =3D crypto_aead_tfm(tfm); - struct crypto_alg *alg =3D aead->__crt_alg; =20 ctx->cryp =3D starfive_cryp_find_dev(ctx); if (!ctx->cryp) return -ENODEV; =20 - if (alg->cra_flags & CRYPTO_ALG_NEED_FALLBACK) { - ctx->aead_fbk =3D crypto_alloc_aead(alg->cra_name, 0, - CRYPTO_ALG_NEED_FALLBACK); - if (IS_ERR(ctx->aead_fbk)) - return dev_err_probe(cryp->dev, PTR_ERR(ctx->aead_fbk), - "%s() failed to allocate fallback for %s\n", - __func__, alg->cra_name); - } + ctx->aead_fbk =3D crypto_alloc_aead(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + if (IS_ERR(ctx->aead_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->aead_fbk), + "%s() failed to allocate fallback for %s\n", + __func__, alg_name); =20 crypto_aead_set_reqsize(tfm, sizeof(struct starfive_cryp_ctx) + sizeof(struct aead_request)); @@ -646,6 +741,44 @@ static void starfive_aes_aead_exit_tfm(struct crypto_a= ead *tfm) crypto_free_aead(ctx->aead_fbk); } =20 +static bool starfive_aes_check_unaligned(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst) +{ + struct scatterlist *tsg; + int i; + + for_each_sg(src, tsg, sg_nents(src), i) + if (!IS_ALIGNED(tsg->length, AES_BLOCK_SIZE) && + !sg_is_last(tsg)) + return true; + + if (src !=3D dst) + for_each_sg(dst, tsg, sg_nents(dst), i) + if (!IS_ALIGNED(tsg->length, AES_BLOCK_SIZE) && + !sg_is_last(tsg)) + return true; + + return false; +} + +static int starfive_aes_do_fallback(struct skcipher_request *req, bool enc) +{ + struct starfive_cryp_ctx *ctx =3D + crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)); + struct skcipher_request *subreq =3D skcipher_request_ctx(req); + + skcipher_request_set_tfm(subreq, ctx->skcipher_fbk); + skcipher_request_set_callback(subreq, req->base.flags, + req->base.complete, + req->base.data); + skcipher_request_set_crypt(subreq, req->src, req->dst, + req->cryptlen, req->iv); + + return enc ? crypto_skcipher_encrypt(subreq) : + crypto_skcipher_decrypt(subreq); +} + static int starfive_aes_crypt(struct skcipher_request *req, unsigned long = flags) { struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); @@ -660,32 +793,54 @@ static int starfive_aes_crypt(struct skcipher_request= *req, unsigned long flags) if (req->cryptlen & blocksize_align) return -EINVAL; =20 + if (starfive_aes_check_unaligned(cryp, req->src, req->dst)) + return starfive_aes_do_fallback(req, is_encrypt(cryp)); + return crypto_transfer_skcipher_request_to_engine(cryp->engine, req); } =20 +static int starfive_aes_aead_do_fallback(struct aead_request *req, bool en= c) +{ + struct starfive_cryp_ctx *ctx =3D + crypto_aead_ctx(crypto_aead_reqtfm(req)); + struct aead_request *subreq =3D aead_request_ctx(req); + + aead_request_set_tfm(subreq, ctx->aead_fbk); + aead_request_set_callback(subreq, req->base.flags, + req->base.complete, + req->base.data); + aead_request_set_crypt(subreq, req->src, req->dst, + req->cryptlen, req->iv); + aead_request_set_ad(subreq, req->assoclen); + + return enc ? crypto_aead_encrypt(subreq) : + crypto_aead_decrypt(subreq); +} + static int starfive_aes_aead_crypt(struct aead_request *req, unsigned long= flags) { struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(crypto_aead_reqtfm(req)= ); struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct scatterlist *src, *dst, _src[2], _dst[2]; =20 cryp->flags =3D flags; =20 - /* - * HW engine could not perform CCM tag verification on - * non-blocksize aligned text, use fallback algo instead + /* aes-ccm does not support tag verification for non-aligned text, + * use fallback for ccm decryption instead. */ - if (ctx->aead_fbk && !is_encrypt(cryp)) { - struct aead_request *subreq =3D aead_request_ctx(req); + if (((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_AES_MODE_CCM) && + !is_encrypt(cryp)) + return starfive_aes_aead_do_fallback(req, 0); =20 - aead_request_set_tfm(subreq, ctx->aead_fbk); - aead_request_set_callback(subreq, req->base.flags, - req->base.complete, req->base.data); - aead_request_set_crypt(subreq, req->src, - req->dst, req->cryptlen, req->iv); - aead_request_set_ad(subreq, req->assoclen); + src =3D scatterwalk_ffwd(_src, req->src, req->assoclen); =20 - return crypto_aead_decrypt(subreq); - } + if (req->src =3D=3D req->dst) + dst =3D src; + else + dst =3D scatterwalk_ffwd(_dst, req->dst, req->assoclen); + + if (starfive_aes_check_unaligned(cryp, src, dst)) + return starfive_aes_aead_do_fallback(req, is_encrypt(cryp)); =20 return crypto_transfer_aead_request_to_engine(cryp->engine, req); } @@ -706,7 +861,7 @@ static int starfive_aes_setkey(struct crypto_skcipher *= tfm, const u8 *key, memcpy(ctx->key, key, keylen); ctx->keylen =3D keylen; =20 - return 0; + return crypto_skcipher_setkey(ctx->skcipher_fbk, key, keylen); } =20 static int starfive_aes_aead_setkey(struct crypto_aead *tfm, const u8 *key, @@ -725,16 +880,20 @@ static int starfive_aes_aead_setkey(struct crypto_aea= d *tfm, const u8 *key, memcpy(ctx->key, key, keylen); ctx->keylen =3D keylen; =20 - if (ctx->aead_fbk) - return crypto_aead_setkey(ctx->aead_fbk, key, keylen); - - return 0; + return crypto_aead_setkey(ctx->aead_fbk, key, keylen); } =20 static int starfive_aes_gcm_setauthsize(struct crypto_aead *tfm, unsigned int authsize) { - return crypto_gcm_check_authsize(authsize); + struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(tfm); + int ret; + + ret =3D crypto_gcm_check_authsize(authsize); + if (ret) + return ret; + + return crypto_aead_setauthsize(ctx->aead_fbk, authsize); } =20 static int starfive_aes_ccm_setauthsize(struct crypto_aead *tfm, @@ -820,9 +979,35 @@ static int starfive_aes_ccm_decrypt(struct aead_reques= t *req) return starfive_aes_aead_crypt(req, STARFIVE_AES_MODE_CCM); } =20 +static int starfive_aes_ecb_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_aes_init_tfm(tfm, "ecb(aes-generic)"); +} + +static int starfive_aes_cbc_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_aes_init_tfm(tfm, "cbc(aes-generic)"); +} + +static int starfive_aes_ctr_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_aes_init_tfm(tfm, "ctr(aes-generic)"); +} + +static int starfive_aes_ccm_init_tfm(struct crypto_aead *tfm) +{ + return starfive_aes_aead_init_tfm(tfm, "ccm_base(ctr(aes-generic),cbcmac(= aes-generic))"); +} + +static int starfive_aes_gcm_init_tfm(struct crypto_aead *tfm) +{ + return starfive_aes_aead_init_tfm(tfm, "gcm_base(ctr(aes-generic),ghash-g= eneric)"); +} + static struct skcipher_engine_alg skcipher_algs[] =3D { { - .base.init =3D starfive_aes_init_tfm, + .base.init =3D starfive_aes_ecb_init_tfm, + .base.exit =3D starfive_aes_exit_tfm, .base.setkey =3D starfive_aes_setkey, .base.encrypt =3D starfive_aes_ecb_encrypt, .base.decrypt =3D starfive_aes_ecb_decrypt, @@ -832,7 +1017,8 @@ static struct skcipher_engine_alg skcipher_algs[] =3D { .cra_name =3D "ecb(aes)", .cra_driver_name =3D "starfive-ecb-aes", .cra_priority =3D 200, - .cra_flags =3D CRYPTO_ALG_ASYNC, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, .cra_blocksize =3D AES_BLOCK_SIZE, .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), .cra_alignmask =3D 0xf, @@ -842,7 +1028,8 @@ static struct skcipher_engine_alg skcipher_algs[] =3D { .do_one_request =3D starfive_aes_do_one_req, }, }, { - .base.init =3D starfive_aes_init_tfm, + .base.init =3D starfive_aes_cbc_init_tfm, + .base.exit =3D starfive_aes_exit_tfm, .base.setkey =3D starfive_aes_setkey, .base.encrypt =3D starfive_aes_cbc_encrypt, .base.decrypt =3D starfive_aes_cbc_decrypt, @@ -853,7 +1040,8 @@ static struct skcipher_engine_alg skcipher_algs[] =3D { .cra_name =3D "cbc(aes)", .cra_driver_name =3D "starfive-cbc-aes", .cra_priority =3D 200, - .cra_flags =3D CRYPTO_ALG_ASYNC, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, .cra_blocksize =3D AES_BLOCK_SIZE, .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), .cra_alignmask =3D 0xf, @@ -863,7 +1051,8 @@ static struct skcipher_engine_alg skcipher_algs[] =3D { .do_one_request =3D starfive_aes_do_one_req, }, }, { - .base.init =3D starfive_aes_init_tfm, + .base.init =3D starfive_aes_ctr_init_tfm, + .base.exit =3D starfive_aes_exit_tfm, .base.setkey =3D starfive_aes_setkey, .base.encrypt =3D starfive_aes_ctr_encrypt, .base.decrypt =3D starfive_aes_ctr_decrypt, @@ -874,7 +1063,8 @@ static struct skcipher_engine_alg skcipher_algs[] =3D { .cra_name =3D "ctr(aes)", .cra_driver_name =3D "starfive-ctr-aes", .cra_priority =3D 200, - .cra_flags =3D CRYPTO_ALG_ASYNC, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, .cra_blocksize =3D 1, .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), .cra_alignmask =3D 0xf, @@ -892,7 +1082,7 @@ static struct aead_engine_alg aead_algs[] =3D { .base.setauthsize =3D starfive_aes_gcm_setauthsize, .base.encrypt =3D starfive_aes_gcm_encrypt, .base.decrypt =3D starfive_aes_gcm_decrypt, - .base.init =3D starfive_aes_aead_init_tfm, + .base.init =3D starfive_aes_gcm_init_tfm, .base.exit =3D starfive_aes_aead_exit_tfm, .base.ivsize =3D GCM_AES_IV_SIZE, .base.maxauthsize =3D AES_BLOCK_SIZE, @@ -900,7 +1090,8 @@ static struct aead_engine_alg aead_algs[] =3D { .cra_name =3D "gcm(aes)", .cra_driver_name =3D "starfive-gcm-aes", .cra_priority =3D 200, - .cra_flags =3D CRYPTO_ALG_ASYNC, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, .cra_blocksize =3D 1, .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), .cra_alignmask =3D 0xf, @@ -914,7 +1105,7 @@ static struct aead_engine_alg aead_algs[] =3D { .base.setauthsize =3D starfive_aes_ccm_setauthsize, .base.encrypt =3D starfive_aes_ccm_encrypt, .base.decrypt =3D starfive_aes_ccm_decrypt, - .base.init =3D starfive_aes_aead_init_tfm, + .base.init =3D starfive_aes_ccm_init_tfm, .base.exit =3D starfive_aes_aead_exit_tfm, .base.ivsize =3D AES_BLOCK_SIZE, .base.maxauthsize =3D AES_BLOCK_SIZE, diff --git a/drivers/crypto/starfive/jh7110-cryp.c b/drivers/crypto/starfiv= e/jh7110-cryp.c index 425fddf3a8ab..fe33e87f25ab 100644 --- a/drivers/crypto/starfive/jh7110-cryp.c +++ b/drivers/crypto/starfive/jh7110-cryp.c @@ -97,12 +97,6 @@ static irqreturn_t starfive_cryp_irq(int irq, void *priv) =20 mask =3D readl(cryp->base + STARFIVE_IE_MASK_OFFSET); status =3D readl(cryp->base + STARFIVE_IE_FLAG_OFFSET); - if (status & STARFIVE_IE_FLAG_AES_DONE) { - mask |=3D STARFIVE_IE_MASK_AES_DONE; - writel(mask, cryp->base + STARFIVE_IE_MASK_OFFSET); - tasklet_schedule(&cryp->aes_done); - } - if (status & STARFIVE_IE_FLAG_HASH_DONE) { mask |=3D STARFIVE_IE_MASK_HASH_DONE; writel(mask, cryp->base + STARFIVE_IE_MASK_OFFSET); @@ -131,7 +125,6 @@ static int starfive_cryp_probe(struct platform_device *= pdev) return dev_err_probe(&pdev->dev, PTR_ERR(cryp->base), "Error remapping memory for platform device\n"); =20 - tasklet_init(&cryp->aes_done, starfive_aes_done_task, (unsigned long)cryp= ); tasklet_init(&cryp->hash_done, starfive_hash_done_task, (unsigned long)cr= yp); =20 cryp->phys_base =3D res->start; @@ -219,7 +212,6 @@ static int starfive_cryp_probe(struct platform_device *= pdev) clk_disable_unprepare(cryp->ahb); reset_control_assert(cryp->rst); =20 - tasklet_kill(&cryp->aes_done); tasklet_kill(&cryp->hash_done); =20 return ret; @@ -233,7 +225,6 @@ static void starfive_cryp_remove(struct platform_device= *pdev) starfive_hash_unregister_algs(); starfive_rsa_unregister_algs(); =20 - tasklet_kill(&cryp->aes_done); tasklet_kill(&cryp->hash_done); =20 crypto_engine_stop(cryp->engine); diff --git a/drivers/crypto/starfive/jh7110-cryp.h b/drivers/crypto/starfiv= e/jh7110-cryp.h index 4940cd1a3fbb..3f8f0b001d41 100644 --- a/drivers/crypto/starfive/jh7110-cryp.h +++ b/drivers/crypto/starfive/jh7110-cryp.h @@ -168,6 +168,7 @@ struct starfive_cryp_ctx { struct crypto_akcipher *akcipher_fbk; struct crypto_ahash *ahash_fbk; struct crypto_aead *aead_fbk; + struct crypto_skcipher *skcipher_fbk; }; =20 struct starfive_cryp_dev { @@ -188,7 +189,6 @@ struct starfive_cryp_dev { struct scatter_walk in_walk; struct scatter_walk out_walk; struct crypto_engine *engine; - struct tasklet_struct aes_done; struct tasklet_struct hash_done; struct completion dma_done; size_t assoclen; @@ -239,5 +239,4 @@ int starfive_aes_register_algs(void); void starfive_aes_unregister_algs(void); =20 void starfive_hash_done_task(unsigned long param); -void starfive_aes_done_task(unsigned long param); #endif --=20 2.34.1 From nobody Fri Dec 19 08:06:18 2025 Received: from fd01.gateway.ufhost.com (fd01.gateway.ufhost.com [61.152.239.71]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A2DC31597; Sat, 16 Dec 2023 14:13:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com Received: from EXMBX165.cuchost.com (unknown [175.102.18.54]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "EXMBX165", Issuer "EXMBX165" (not verified)) by fd01.gateway.ufhost.com (Postfix) with ESMTP id 3CAF024DC13; Sat, 16 Dec 2023 22:13:14 +0800 (CST) Received: from EXMBX068.cuchost.com (172.16.6.68) by EXMBX165.cuchost.com (172.16.6.75) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:13:14 +0800 Received: from ubuntu.localdomain (161.142.156.108) by EXMBX068.cuchost.com (172.16.6.68) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:13:08 +0800 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Rob Herring , Krzysztof Kozlowski , Conor Dooley , , , Subject: [PATCH 4/5] crypto: starfive: Add sm3 support for JH8100 Date: Sat, 16 Dec 2023 22:12:33 +0800 Message-ID: <20231216141234.417498-5-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231216141234.417498-1-jiajie.ho@starfivetech.com> References: <20231216141234.417498-1-jiajie.ho@starfivetech.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: EXCAS066.cuchost.com (172.16.6.26) To EXMBX068.cuchost.com (172.16.6.68) X-YovoleRuleAgent: yovoleflag Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add driver support for SM3 module in JH8100 SoC. Updated dedicated dma allows access to 64-bit memory address space. Signed-off-by: Jia Jie Ho --- drivers/crypto/starfive/Kconfig | 23 ++ drivers/crypto/starfive/Makefile | 3 + drivers/crypto/starfive/jh7110-cryp.c | 48 ++- drivers/crypto/starfive/jh7110-cryp.h | 59 +++ drivers/crypto/starfive/jh7110-hash.c | 20 +- drivers/crypto/starfive/jh8100-sm3.c | 532 ++++++++++++++++++++++++++ 6 files changed, 676 insertions(+), 9 deletions(-) create mode 100644 drivers/crypto/starfive/jh8100-sm3.c diff --git a/drivers/crypto/starfive/Kconfig b/drivers/crypto/starfive/Kcon= fig index 0fe389e9f932..e05990f90685 100644 --- a/drivers/crypto/starfive/Kconfig +++ b/drivers/crypto/starfive/Kconfig @@ -24,3 +24,26 @@ config CRYPTO_DEV_JH7110 skciphers, AEAD and hash functions. =20 If you choose 'M' here, this module will be called jh7110-crypto. + +config CRYPTO_DEV_JH8100 + tristate "StarFive JH8100 cryptographic engine drivers" + depends on (SOC_STARFIVE && DW_AXI_DMAC) || COMPILE_TEST + depends on HAS_DMA + select CRYPTO_ENGINE + select CRYPTO_HMAC + select CRYPTO_SHA256 + select CRYPTO_SHA512 + select CRYPTO_SM3_GENERIC + select CRYPTO_RSA + select CRYPTO_AES + select CRYPTO_CCM + select CRYPTO_GCM + select CRYPTO_CBC + select CRYPTO_ECB + select CRYPTO_CTR + help + Support for StarFive JH8100 crypto hardware acceleration engine. + This module provides additional support for SM2 signature verification, + SM3 hash/hmac functions and SM4 skcipher. + + If you choose 'M' here, this module will be called jh8100-crypto. diff --git a/drivers/crypto/starfive/Makefile b/drivers/crypto/starfive/Mak= efile index 8c137afe58ad..67717fca3f5d 100644 --- a/drivers/crypto/starfive/Makefile +++ b/drivers/crypto/starfive/Makefile @@ -2,3 +2,6 @@ =20 obj-$(CONFIG_CRYPTO_DEV_JH7110) +=3D jh7110-crypto.o jh7110-crypto-objs :=3D jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-ae= s.o + +obj-$(CONFIG_CRYPTO_DEV_JH8100) +=3D jh8100-crypto.o +jh8100-crypto-objs :=3D jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-ae= s.o jh8100-sm3.o diff --git a/drivers/crypto/starfive/jh7110-cryp.c b/drivers/crypto/starfiv= e/jh7110-cryp.c index fe33e87f25ab..fb7c19705fbf 100644 --- a/drivers/crypto/starfive/jh7110-cryp.c +++ b/drivers/crypto/starfive/jh7110-cryp.c @@ -106,6 +106,26 @@ static irqreturn_t starfive_cryp_irq(int irq, void *pr= iv) return IRQ_HANDLED; } =20 +#ifdef CONFIG_CRYPTO_DEV_JH8100 +static irqreturn_t starfive_cryp_irq1(int irq, void *priv) +{ + u32 status; + u32 mask; + struct starfive_cryp_dev *cryp =3D (struct starfive_cryp_dev *)priv; + + mask =3D readl(cryp->base + STARFIVE_SM_IE_MASK_OFFSET); + status =3D readl(cryp->base + STARFIVE_SM_IE_FLAG_OFFSET); + + if (status & STARFIVE_SM_IE_FLAG_SM3_DONE) { + mask |=3D STARFIVE_SM_IE_MASK_SM3_DONE; + writel(mask, cryp->base + STARFIVE_SM_IE_MASK_OFFSET); + tasklet_schedule(&cryp->sm3_done); + } + + return IRQ_HANDLED; +} +#endif + static int starfive_cryp_probe(struct platform_device *pdev) { struct starfive_cryp_dev *cryp; @@ -156,6 +176,16 @@ static int starfive_cryp_probe(struct platform_device = *pdev) return dev_err_probe(&pdev->dev, ret, "Failed to register interrupt handler\n"); =20 +#ifdef CONFIG_CRYPTO_DEV_JH8100 + tasklet_init(&cryp->sm3_done, starfive_sm3_done_task, (unsigned long)cryp= ); + + irq =3D platform_get_irq(pdev, 1); + if (irq < 0) + return irq; + + ret =3D devm_request_irq(&pdev->dev, irq, starfive_cryp_irq1, 0, + pdev->name, (void *)cryp); +#endif clk_prepare_enable(cryp->hclk); clk_prepare_enable(cryp->ahb); reset_control_deassert(cryp->rst); @@ -191,8 +221,17 @@ static int starfive_cryp_probe(struct platform_device = *pdev) if (ret) goto err_algs_rsa; =20 +#ifdef CONFIG_CRYPTO_DEV_JH8100 + ret =3D starfive_sm3_register_algs(); + if (ret) + goto err_algs_sm3; +#endif return 0; =20 +#ifdef CONFIG_CRYPTO_DEV_JH8100 +err_algs_sm3: + starfive_rsa_unregister_algs(); +#endif err_algs_rsa: starfive_hash_unregister_algs(); err_algs_hash: @@ -213,7 +252,9 @@ static int starfive_cryp_probe(struct platform_device *= pdev) reset_control_assert(cryp->rst); =20 tasklet_kill(&cryp->hash_done); - +#ifdef CONFIG_CRYPTO_DEV_JH8100 + tasklet_kill(&cryp->sm3_done); +#endif return ret; } =20 @@ -226,7 +267,10 @@ static void starfive_cryp_remove(struct platform_devic= e *pdev) starfive_rsa_unregister_algs(); =20 tasklet_kill(&cryp->hash_done); - +#ifdef CONFIG_CRYPTO_DEV_JH8100 + starfive_sm3_unregister_algs(); + tasklet_kill(&cryp->sm3_done); +#endif crypto_engine_stop(cryp->engine); crypto_engine_exit(cryp->engine); =20 diff --git a/drivers/crypto/starfive/jh7110-cryp.h b/drivers/crypto/starfiv= e/jh7110-cryp.h index 3f8f0b001d41..e6c8496acbc9 100644 --- a/drivers/crypto/starfive/jh7110-cryp.h +++ b/drivers/crypto/starfive/jh7110-cryp.h @@ -19,12 +19,22 @@ #define STARFIVE_DMA_IN_LEN_OFFSET 0x10 #define STARFIVE_DMA_OUT_LEN_OFFSET 0x14 =20 +#define STARFIVE_SM_ALG_CR_OFFSET 0x4000 +#define STARFIVE_SM_IE_MASK_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x4) +#define STARFIVE_SM_IE_FLAG_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x8) +#define STARFIVE_SM_DMA_IN_LEN_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0xc) +#define STARFIVE_SM_DMA_OUT_LEN_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x10) +#define STARFIVE_SM_ALG_FIFO_IN_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x20) +#define STARFIVE_SM_ALG_FIFO_OUT_OFFSET (STARFIVE_SM_ALG_CR_OFFSET + 0x28) + #define STARFIVE_IE_MASK_AES_DONE 0x1 #define STARFIVE_IE_MASK_HASH_DONE 0x4 #define STARFIVE_IE_MASK_PKA_DONE 0x8 #define STARFIVE_IE_FLAG_AES_DONE 0x1 #define STARFIVE_IE_FLAG_HASH_DONE 0x4 #define STARFIVE_IE_FLAG_PKA_DONE 0x8 +#define STARFIVE_SM_IE_MASK_SM3_DONE 0x2 +#define STARFIVE_SM_IE_FLAG_SM3_DONE 0x2 =20 #define STARFIVE_MSG_BUFFER_SIZE SZ_16K #define MAX_KEY_SIZE SHA512_BLOCK_SIZE @@ -68,6 +78,20 @@ union starfive_aes_csr { }; }; =20 +union starfive_sm_alg_cr { + u32 v; + struct { + u32 start :1; + u32 sm4_dma_en :1; + u32 sm3_dma_en :1; + u32 rsvd_0 :1; + u32 alg_done :1; + u32 rsvd_1 :3; + u32 clear :1; + u32 rsvd_2 :23; + }; +}; + union starfive_hash_csr { u32 v; struct { @@ -132,6 +156,32 @@ union starfive_pka_casr { }; }; =20 +union starfive_sm3_csr { + u32 v; + struct { + u32 start :1; + u32 reset :1; + u32 ie :1; + u32 firstb :1; +#define STARFIVE_SM3_MODE 0x0 + u32 mode :3; + u32 rsvd_0 :1; + u32 final :1; + u32 rsvd_1 :2; +#define STARFIVE_SM3_HMAC_FLAGS 0x800 + u32 hmac :1; + u32 rsvd_2 :1; +#define STARFIVE_SM3_KEY_DONE BIT(13) + u32 key_done :1; + u32 key_flag :1; + u32 hmac_done :1; +#define STARFIVE_SM3_BUSY BIT(16) + u32 busy :1; + u32 hashdone :1; + u32 rsvd_3 :14; + }; +}; + struct starfive_rsa_key { u8 *n; u8 *e; @@ -190,6 +240,7 @@ struct starfive_cryp_dev { struct scatter_walk out_walk; struct crypto_engine *engine; struct tasklet_struct hash_done; + struct tasklet_struct sm3_done; struct completion dma_done; size_t assoclen; size_t total_in; @@ -213,6 +264,7 @@ struct starfive_cryp_request_ctx { union starfive_hash_csr hash; union starfive_pka_cacr pka; union starfive_aes_csr aes; + union starfive_sm3_csr sm3; } csr; =20 struct scatterlist *in_sg; @@ -239,4 +291,11 @@ int starfive_aes_register_algs(void); void starfive_aes_unregister_algs(void); =20 void starfive_hash_done_task(unsigned long param); + +#ifdef CONFIG_CRYPTO_DEV_JH8100 +int starfive_sm3_register_algs(void); +void starfive_sm3_unregister_algs(void); + +void starfive_sm3_done_task(unsigned long param); +#endif #endif diff --git a/drivers/crypto/starfive/jh7110-hash.c b/drivers/crypto/starfiv= e/jh7110-hash.c index 74e151b5f875..45cf82e64fb8 100644 --- a/drivers/crypto/starfive/jh7110-hash.c +++ b/drivers/crypto/starfive/jh7110-hash.c @@ -511,12 +511,6 @@ static int starfive_sha512_init_tfm(struct crypto_ahas= h *hash) STARFIVE_HASH_SHA512, 0); } =20 -static int starfive_sm3_init_tfm(struct crypto_ahash *hash) -{ - return starfive_hash_init_tfm(hash, "sm3-generic", - STARFIVE_HASH_SM3, 0); -} - static int starfive_hmac_sha224_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "hmac(sha224-generic)", @@ -541,11 +535,19 @@ static int starfive_hmac_sha512_init_tfm(struct crypt= o_ahash *hash) STARFIVE_HASH_SHA512, 1); } =20 +#ifndef CONFIG_CRYPTO_DEV_JH8100 +static int starfive_sm3_init_tfm(struct crypto_ahash *hash) +{ + return starfive_hash_init_tfm(hash, "sm3-generic", + STARFIVE_HASH_SM3, 0); +} + static int starfive_hmac_sm3_init_tfm(struct crypto_ahash *hash) { return starfive_hash_init_tfm(hash, "hmac(sm3-generic)", STARFIVE_HASH_SM3, 1); } +#endif =20 static struct ahash_engine_alg algs_sha2_sm3[] =3D { { @@ -776,7 +778,10 @@ static struct ahash_engine_alg algs_sha2_sm3[] =3D { .op =3D { .do_one_request =3D starfive_hash_one_request, }, -}, { +}, + +#ifndef CONFIG_CRYPTO_DEV_JH8100 +{ .base.init =3D starfive_hash_init, .base.update =3D starfive_hash_update, .base.final =3D starfive_hash_final, @@ -834,6 +839,7 @@ static struct ahash_engine_alg algs_sha2_sm3[] =3D { .do_one_request =3D starfive_hash_one_request, }, }, +#endif }; =20 int starfive_hash_register_algs(void) diff --git a/drivers/crypto/starfive/jh8100-sm3.c b/drivers/crypto/starfive= /jh8100-sm3.c new file mode 100644 index 000000000000..7289c5fba0d8 --- /dev/null +++ b/drivers/crypto/starfive/jh8100-sm3.c @@ -0,0 +1,532 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SM3 Hash function and HMAC support for StarFive driver + * + * Copyright (c) 2022 - 2023 StarFive Technology + * + */ + +#include +#include +#include +#include +#include "jh7110-cryp.h" +#include +#include +#include + +#define STARFIVE_SM3_REGS_OFFSET 0x4200 +#define STARFIVE_SM3_CSR (STARFIVE_SM3_REGS_OFFSET + 0x0) +#define STARFIVE_SM3_WDR (STARFIVE_SM3_REGS_OFFSET + 0x4) +#define STARFIVE_SM3_RDR (STARFIVE_SM3_REGS_OFFSET + 0x8) +#define STARFIVE_SM3_WSR (STARFIVE_SM3_REGS_OFFSET + 0xC) +#define STARFIVE_SM3_WLEN3 (STARFIVE_SM3_REGS_OFFSET + 0x10) +#define STARFIVE_SM3_WLEN2 (STARFIVE_SM3_REGS_OFFSET + 0x14) +#define STARFIVE_SM3_WLEN1 (STARFIVE_SM3_REGS_OFFSET + 0x18) +#define STARFIVE_SM3_WLEN0 (STARFIVE_SM3_REGS_OFFSET + 0x1C) +#define STARFIVE_SM3_WKR (STARFIVE_SM3_REGS_OFFSET + 0x20) +#define STARFIVE_SM3_WKLEN (STARFIVE_SM3_REGS_OFFSET + 0x24) + +#define STARFIVE_SM3_BUFLEN SHA512_BLOCK_SIZE +#define STARFIVE_SM3_RESET 0x2 + +static inline int starfive_sm3_wait_busy(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM3_CSR, status, + !(status & STARFIVE_SM3_BUSY), 10, 100000); +} + +static inline int starfive_sm3_wait_key_done(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM3_CSR, status, + (status & STARFIVE_SM3_KEY_DONE), 10, 100000); +} + +static int starfive_sm3_hmac_key(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + struct starfive_cryp_dev *cryp =3D ctx->cryp; + int klen =3D ctx->keylen, loop; + unsigned int *key =3D (unsigned int *)ctx->key; + unsigned char *cl; + + writel(ctx->keylen, cryp->base + STARFIVE_SM3_WKLEN); + + rctx->csr.sm3.hmac =3D 1; + rctx->csr.sm3.key_flag =3D 1; + + writel(rctx->csr.sm3.v, cryp->base + STARFIVE_SM3_CSR); + + for (loop =3D 0; loop < klen / sizeof(unsigned int); loop++, key++) + writel(*key, cryp->base + STARFIVE_SM3_WKR); + + if (klen & 0x3) { + cl =3D (unsigned char *)key; + for (loop =3D 0; loop < (klen & 0x3); loop++, cl++) + writeb(*cl, cryp->base + STARFIVE_SM3_WKR); + } + + if (starfive_sm3_wait_key_done(ctx)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, + "starfive_sm3_wait_key_done error\n"); + + return 0; +} + +static void starfive_sm3_start(struct starfive_cryp_dev *cryp) +{ + union starfive_sm3_csr csr; + u32 mask; + + csr.v =3D readl(cryp->base + STARFIVE_SM3_CSR); + csr.firstb =3D 0; + csr.final =3D 1; + csr.ie =3D 1; + writel(csr.v, cryp->base + STARFIVE_SM3_CSR); + + mask =3D readl(cryp->base + STARFIVE_SM_IE_MASK_OFFSET); + mask &=3D ~STARFIVE_SM_IE_MASK_SM3_DONE; + writel(mask, cryp->base + STARFIVE_SM_IE_MASK_OFFSET); +} + +static void starfive_sm3_dma_callback(void *param) +{ + struct starfive_cryp_dev *cryp =3D param; + + complete(&cryp->dma_done); +} + +static void starfive_sm3_dma_init(struct starfive_cryp_dev *cryp) +{ + cryp->cfg_in.direction =3D DMA_MEM_TO_DEV; + cryp->cfg_in.src_addr_width =3D DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.dst_addr_width =3D DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.src_maxburst =3D cryp->dma_maxburst; + cryp->cfg_in.dst_maxburst =3D cryp->dma_maxburst; + cryp->cfg_in.dst_addr =3D cryp->phys_base + STARFIVE_SM_ALG_FIFO_IN_OFFSE= T; + + dmaengine_slave_config(cryp->tx, &cryp->cfg_in); + + init_completion(&cryp->dma_done); +} + +static int starfive_sm3_dma_xfer(struct starfive_cryp_dev *cryp, + struct scatterlist *sg) +{ + struct dma_async_tx_descriptor *in_desc; + union starfive_sm_alg_cr alg_cr; + int ret =3D 0; + + alg_cr.v =3D 0; + alg_cr.start =3D 1; + alg_cr.sm3_dma_en =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + writel(sg_dma_len(sg), cryp->base + STARFIVE_SM_DMA_IN_LEN_OFFSET); + sg_dma_len(sg) =3D ALIGN(sg_dma_len(sg), sizeof(u32)); + + in_desc =3D dmaengine_prep_slave_sg(cryp->tx, sg, 1, DMA_MEM_TO_DEV, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!in_desc) { + ret =3D -EINVAL; + goto end; + } + + reinit_completion(&cryp->dma_done); + in_desc->callback =3D starfive_sm3_dma_callback; + in_desc->callback_param =3D cryp; + + dmaengine_submit(in_desc); + dma_async_issue_pending(cryp->tx); + + if (!wait_for_completion_timeout(&cryp->dma_done, + msecs_to_jiffies(1000))) + ret =3D -ETIMEDOUT; + +end: + alg_cr.v =3D 0; + alg_cr.clear =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + return ret; +} + +static int starfive_sm3_copy_hash(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(crypto_ahash_reqtfm(re= q)); + int count, *data; + int mlen; + + if (!req->result) + return 0; + + mlen =3D rctx->digsize / sizeof(u32); + data =3D (u32 *)req->result; + + for (count =3D 0; count < mlen; count++) + data[count] =3D readl(ctx->cryp->base + STARFIVE_SM3_RDR); + + return 0; +} + +void starfive_sm3_done_task(unsigned long param) +{ + struct starfive_cryp_dev *cryp =3D (struct starfive_cryp_dev *)param; + int err; + + err =3D starfive_sm3_copy_hash(cryp->req.hreq); + + /* Reset to clear hash_done in irq register*/ + writel(STARFIVE_SM3_RESET, cryp->base + STARFIVE_SM3_CSR); + + crypto_finalize_hash_request(cryp->engine, cryp->req.hreq, err); +} + +static int starfive_sm3_one_request(struct crypto_engine *engine, void *ar= eq) +{ + struct ahash_request *req =3D + container_of(areq, struct ahash_request, base); + struct starfive_cryp_ctx *ctx =3D + crypto_ahash_ctx(crypto_ahash_reqtfm(req)); + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + struct scatterlist *tsg; + int ret, src_nents, i; + + rctx->csr.sm3.v =3D 0; + rctx->csr.sm3.reset =3D 1; + + writel(rctx->csr.sm3.v, cryp->base + STARFIVE_SM3_CSR); + + if (starfive_sm3_wait_busy(ctx)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, + "Error resetting engine.\n"); + + rctx->csr.sm3.v =3D 0; + rctx->csr.sm3.mode =3D ctx->hash_mode; + + if (ctx->is_hmac) { + ret =3D starfive_sm3_hmac_key(ctx); + if (ret) + return ret; + } else { + rctx->csr.sm3.start =3D 1; + rctx->csr.sm3.firstb =3D 1; + writel(rctx->csr.sm3.v, cryp->base + STARFIVE_SM3_CSR); + } + + /* No input message, get digest and end. */ + if (!rctx->total) + goto hash_start; + + starfive_sm3_dma_init(cryp); + + for_each_sg(rctx->in_sg, tsg, rctx->in_sg_len, i) { + src_nents =3D dma_map_sg(cryp->dev, tsg, 1, DMA_TO_DEVICE); + if (src_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg error\n"); + + ret =3D starfive_sm3_dma_xfer(cryp, tsg); + dma_unmap_sg(cryp->dev, tsg, 1, DMA_TO_DEVICE); + if (ret) + return ret; + } + +hash_start: + starfive_sm3_start(cryp); + + return 0; +} + +static void starfive_sm3_set_ahash(struct ahash_request *req, + struct starfive_cryp_ctx *ctx, + struct starfive_cryp_request_ctx *rctx) +{ + ahash_request_set_tfm(&rctx->ahash_fbk_req, ctx->ahash_fbk); + ahash_request_set_callback(&rctx->ahash_fbk_req, + req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP, + req->base.complete, req->base.data); + ahash_request_set_crypt(&rctx->ahash_fbk_req, req->src, + req->result, req->nbytes); +} + +static int starfive_sm3_init(struct ahash_request *req) +{ + struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_init(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_update(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_update(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_final(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_final(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_finup(struct ahash_request *req) +{ + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); + + starfive_sm3_set_ahash(req, ctx, rctx); + + return crypto_ahash_finup(&rctx->ahash_fbk_req); +} + +static int starfive_sm3_digest(struct ahash_request *req) +{ + struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct starfive_cryp_dev *cryp =3D ctx->cryp; + + memset(rctx, 0, sizeof(struct starfive_cryp_request_ctx)); + + cryp->req.hreq =3D req; + rctx->total =3D req->nbytes; + rctx->in_sg =3D req->src; + rctx->blksize =3D crypto_tfm_alg_blocksize(crypto_ahash_tfm(tfm)); + rctx->digsize =3D crypto_ahash_digestsize(tfm); + rctx->in_sg_len =3D sg_nents_for_len(rctx->in_sg, rctx->total); + ctx->rctx =3D rctx; + + return crypto_transfer_hash_request_to_engine(cryp->engine, req); +} + +static int starfive_sm3_export(struct ahash_request *req, void *out) +{ + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); + + ahash_request_set_tfm(&rctx->ahash_fbk_req, ctx->ahash_fbk); + ahash_request_set_callback(&rctx->ahash_fbk_req, + req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP, + req->base.complete, req->base.data); + + return crypto_ahash_export(&rctx->ahash_fbk_req, out); +} + +static int starfive_sm3_import(struct ahash_request *req, const void *in) +{ + struct starfive_cryp_request_ctx *rctx =3D ahash_request_ctx(req); + struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(req); + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(tfm); + + ahash_request_set_tfm(&rctx->ahash_fbk_req, ctx->ahash_fbk); + ahash_request_set_callback(&rctx->ahash_fbk_req, + req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP, + req->base.complete, req->base.data); + + return crypto_ahash_import(&rctx->ahash_fbk_req, in); +} + +static int starfive_sm3_init_algo(struct crypto_ahash *hash, + const char *alg_name, + bool is_hmac) +{ + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); + + ctx->cryp =3D starfive_cryp_find_dev(ctx); + if (!ctx->cryp) + return -ENODEV; + + ctx->ahash_fbk =3D crypto_alloc_ahash(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + + if (IS_ERR(ctx->ahash_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->ahash_fbk), + "starfive-sm3: Could not load fallback driver.\n"); + + crypto_ahash_set_statesize(hash, crypto_ahash_statesize(ctx->ahash_fbk)); + crypto_ahash_set_reqsize(hash, sizeof(struct starfive_cryp_request_ctx) + + crypto_ahash_reqsize(ctx->ahash_fbk)); + + ctx->keylen =3D 0; + ctx->hash_mode =3D STARFIVE_SM3_MODE; + ctx->is_hmac =3D is_hmac; + + return 0; +} + +static void starfive_sm3_exit_tfm(struct crypto_ahash *hash) +{ + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); + + crypto_free_ahash(ctx->ahash_fbk); +} + +static int starfive_sm3_long_setkey(struct starfive_cryp_ctx *ctx, + const u8 *key, unsigned int keylen) +{ + struct crypto_wait wait; + struct ahash_request *req; + struct scatterlist sg; + struct crypto_ahash *ahash_tfm; + struct starfive_cryp_dev *cryp =3D ctx->cryp; + u8 *buf; + int ret; + + ahash_tfm =3D crypto_alloc_ahash("sm3-starfive", 0, 0); + if (IS_ERR(ahash_tfm)) + return PTR_ERR(ahash_tfm); + + req =3D ahash_request_alloc(ahash_tfm, GFP_KERNEL); + if (!req) { + ret =3D -ENOMEM; + goto err_free_ahash; + } + + crypto_init_wait(&wait); + ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, &wait); + crypto_ahash_clear_flags(ahash_tfm, ~0); + + buf =3D devm_kzalloc(cryp->dev, keylen + STARFIVE_SM3_BUFLEN, GFP_KERNEL); + if (!buf) { + ret =3D -ENOMEM; + goto err_free_req; + } + + memcpy(buf, key, keylen); + sg_init_one(&sg, buf, keylen); + ahash_request_set_crypt(req, &sg, ctx->key, keylen); + + ret =3D crypto_wait_req(crypto_ahash_digest(req), &wait); + +err_free_req: + ahash_request_free(req); +err_free_ahash: + crypto_free_ahash(ahash_tfm); + return ret; +} + +static int starfive_sm3_setkey(struct crypto_ahash *hash, + const u8 *key, unsigned int keylen) +{ + struct starfive_cryp_ctx *ctx =3D crypto_ahash_ctx(hash); + unsigned int digestsize =3D crypto_ahash_digestsize(hash); + unsigned int blocksize =3D crypto_ahash_blocksize(hash); + + crypto_ahash_setkey(ctx->ahash_fbk, key, keylen); + + if (keylen <=3D blocksize) { + memcpy(ctx->key, key, keylen); + ctx->keylen =3D keylen; + return 0; + } + + ctx->keylen =3D digestsize; + + return starfive_sm3_long_setkey(ctx, key, keylen); +} + +static int starfive_sm3_init_tfm(struct crypto_ahash *hash) +{ + return starfive_sm3_init_algo(hash, "sm3-generic", 0); +} + +static int starfive_hmac_sm3_init_tfm(struct crypto_ahash *hash) +{ + return starfive_sm3_init_algo(hash, "hmac(sm3-generic)", 1); +} + +static struct ahash_engine_alg algs_sm3[] =3D { +{ + .base.init =3D starfive_sm3_init, + .base.update =3D starfive_sm3_update, + .base.final =3D starfive_sm3_final, + .base.finup =3D starfive_sm3_finup, + .base.digest =3D starfive_sm3_digest, + .base.export =3D starfive_sm3_export, + .base.import =3D starfive_sm3_import, + .base.init_tfm =3D starfive_sm3_init_tfm, + .base.exit_tfm =3D starfive_sm3_exit_tfm, + .base.halg =3D { + .digestsize =3D SM3_DIGEST_SIZE, + .statesize =3D sizeof(struct sm3_state), + .base =3D { + .cra_name =3D "sm3", + .cra_driver_name =3D "sm3-starfive", + .cra_priority =3D 200, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_TYPE_AHASH | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize =3D SM3_BLOCK_SIZE, + .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), + .cra_module =3D THIS_MODULE, + } + }, + .op =3D { + .do_one_request =3D starfive_sm3_one_request, + }, +}, { + .base.init =3D starfive_sm3_init, + .base.update =3D starfive_sm3_update, + .base.final =3D starfive_sm3_final, + .base.finup =3D starfive_sm3_finup, + .base.digest =3D starfive_sm3_digest, + .base.export =3D starfive_sm3_export, + .base.import =3D starfive_sm3_import, + .base.init_tfm =3D starfive_hmac_sm3_init_tfm, + .base.exit_tfm =3D starfive_sm3_exit_tfm, + .base.setkey =3D starfive_sm3_setkey, + .base.halg =3D { + .digestsize =3D SM3_DIGEST_SIZE, + .statesize =3D sizeof(struct sm3_state), + .base =3D { + .cra_name =3D "hmac(sm3)", + .cra_driver_name =3D "sm3-hmac-starfive", + .cra_priority =3D 200, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_TYPE_AHASH | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize =3D SM3_BLOCK_SIZE, + .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), + .cra_module =3D THIS_MODULE, + } + }, + .op =3D { + .do_one_request =3D starfive_sm3_one_request, + }, +}, +}; + +int starfive_sm3_register_algs(void) +{ + return crypto_engine_register_ahashes(algs_sm3, ARRAY_SIZE(algs_sm3)); +} + +void starfive_sm3_unregister_algs(void) +{ + crypto_engine_unregister_ahashes(algs_sm3, ARRAY_SIZE(algs_sm3)); +} --=20 2.34.1 From nobody Fri Dec 19 08:06:18 2025 Received: from fd01.gateway.ufhost.com (fd01.gateway.ufhost.com [61.152.239.71]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A6CE3174A; Sat, 16 Dec 2023 14:13:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=starfivetech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=starfivetech.com Received: from EXMBX166.cuchost.com (unknown [175.102.18.54]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "EXMBX166", Issuer "EXMBX166" (not verified)) by fd01.gateway.ufhost.com (Postfix) with ESMTP id 4DC5A24DC28; Sat, 16 Dec 2023 22:13:22 +0800 (CST) Received: from EXMBX068.cuchost.com (172.16.6.68) by EXMBX166.cuchost.com (172.16.6.76) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:13:22 +0800 Received: from ubuntu.localdomain (161.142.156.108) by EXMBX068.cuchost.com (172.16.6.68) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sat, 16 Dec 2023 22:13:14 +0800 From: Jia Jie Ho To: Herbert Xu , "David S . Miller" , Rob Herring , Krzysztof Kozlowski , Conor Dooley , , , Subject: [PATCH 5/5] crypto: starfive: Add sm4 support for JH8100 Date: Sat, 16 Dec 2023 22:12:34 +0800 Message-ID: <20231216141234.417498-6-jiajie.ho@starfivetech.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231216141234.417498-1-jiajie.ho@starfivetech.com> References: <20231216141234.417498-1-jiajie.ho@starfivetech.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: EXCAS066.cuchost.com (172.16.6.26) To EXMBX068.cuchost.com (172.16.6.68) X-YovoleRuleAgent: yovoleflag Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add driver support for sm4 skcipher and aead for StarFive JH8100 SoC. Signed-off-by: Jia Jie Ho --- drivers/crypto/starfive/Kconfig | 1 + drivers/crypto/starfive/Makefile | 2 +- drivers/crypto/starfive/jh7110-cryp.c | 8 + drivers/crypto/starfive/jh7110-cryp.h | 39 + drivers/crypto/starfive/jh8100-sm4.c | 1107 +++++++++++++++++++++++++ 5 files changed, 1156 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/starfive/jh8100-sm4.c diff --git a/drivers/crypto/starfive/Kconfig b/drivers/crypto/starfive/Kcon= fig index e05990f90685..90160676160a 100644 --- a/drivers/crypto/starfive/Kconfig +++ b/drivers/crypto/starfive/Kconfig @@ -34,6 +34,7 @@ config CRYPTO_DEV_JH8100 select CRYPTO_SHA256 select CRYPTO_SHA512 select CRYPTO_SM3_GENERIC + select CRYPTO_SM4_GENERIC select CRYPTO_RSA select CRYPTO_AES select CRYPTO_CCM diff --git a/drivers/crypto/starfive/Makefile b/drivers/crypto/starfive/Mak= efile index 67717fca3f5d..8370f20427fd 100644 --- a/drivers/crypto/starfive/Makefile +++ b/drivers/crypto/starfive/Makefile @@ -4,4 +4,4 @@ obj-$(CONFIG_CRYPTO_DEV_JH7110) +=3D jh7110-crypto.o jh7110-crypto-objs :=3D jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-ae= s.o =20 obj-$(CONFIG_CRYPTO_DEV_JH8100) +=3D jh8100-crypto.o -jh8100-crypto-objs :=3D jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-ae= s.o jh8100-sm3.o +jh8100-crypto-objs :=3D jh7110-cryp.o jh7110-hash.o jh7110-rsa.o jh7110-ae= s.o jh8100-sm3.o jh8100-sm4.o diff --git a/drivers/crypto/starfive/jh7110-cryp.c b/drivers/crypto/starfiv= e/jh7110-cryp.c index fb7c19705fbf..63b801cd6555 100644 --- a/drivers/crypto/starfive/jh7110-cryp.c +++ b/drivers/crypto/starfive/jh7110-cryp.c @@ -225,10 +225,17 @@ static int starfive_cryp_probe(struct platform_device= *pdev) ret =3D starfive_sm3_register_algs(); if (ret) goto err_algs_sm3; + + ret =3D starfive_sm4_register_algs(); + if (ret) + goto err_algs_sm4; #endif + return 0; =20 #ifdef CONFIG_CRYPTO_DEV_JH8100 +err_algs_sm4: + starfive_sm3_unregister_algs(); err_algs_sm3: starfive_rsa_unregister_algs(); #endif @@ -269,6 +276,7 @@ static void starfive_cryp_remove(struct platform_device= *pdev) tasklet_kill(&cryp->hash_done); #ifdef CONFIG_CRYPTO_DEV_JH8100 starfive_sm3_unregister_algs(); + starfive_sm4_unregister_algs(); tasklet_kill(&cryp->sm3_done); #endif crypto_engine_stop(cryp->engine); diff --git a/drivers/crypto/starfive/jh7110-cryp.h b/drivers/crypto/starfiv= e/jh7110-cryp.h index e6c8496acbc9..8cc941f90a7e 100644 --- a/drivers/crypto/starfive/jh7110-cryp.h +++ b/drivers/crypto/starfive/jh7110-cryp.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -182,6 +183,40 @@ union starfive_sm3_csr { }; }; =20 +union starfive_sm4_csr { + u32 v; + struct { + u32 cmode :1; + u32 rsvd_0 :1; + u32 ie :1; + u32 sm4rst :1; + u32 rsvd_1 :1; +#define STARFIVE_SM4_DONE BIT(5) + u32 sm4done :1; +#define STARFIVE_SM4_KEY_DONE BIT(6) + u32 krdy :1; + u32 busy :1; + u32 vsm4_start :1; + u32 delay_sm4 :1; +#define STARFIVE_SM4_CCM_START BIT(10) + u32 ccm_start :1; +#define STARFIVE_SM4_GCM_START BIT(11) + u32 gcm_start :1; + u32 rsvd_2 :4; +#define STARFIVE_SM4_MODE_XFB_1 0x0 +#define STARFIVE_SM4_MODE_XFB_128 0x5 + u32 stmode :3; + u32 rsvd_3 :2; +#define STARFIVE_SM4_MODE_ECB 0x0 +#define STARFIVE_SM4_MODE_CBC 0x1 +#define STARFIVE_SM4_MODE_CTR 0x4 +#define STARFIVE_SM4_MODE_CCM 0x5 +#define STARFIVE_SM4_MODE_GCM 0x6 + u32 mode :3; + u32 rsvd_4 :8; + }; +}; + struct starfive_rsa_key { u8 *n; u8 *e; @@ -265,6 +300,7 @@ struct starfive_cryp_request_ctx { union starfive_pka_cacr pka; union starfive_aes_csr aes; union starfive_sm3_csr sm3; + union starfive_sm4_csr sm4; } csr; =20 struct scatterlist *in_sg; @@ -296,6 +332,9 @@ void starfive_hash_done_task(unsigned long param); int starfive_sm3_register_algs(void); void starfive_sm3_unregister_algs(void); =20 +int starfive_sm4_register_algs(void); +void starfive_sm4_unregister_algs(void); + void starfive_sm3_done_task(unsigned long param); #endif #endif diff --git a/drivers/crypto/starfive/jh8100-sm4.c b/drivers/crypto/starfive= /jh8100-sm4.c new file mode 100644 index 000000000000..ccde5fb793cc --- /dev/null +++ b/drivers/crypto/starfive/jh8100-sm4.c @@ -0,0 +1,1107 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * StarFive SM4 acceleration driver + * + * Copyright (c) 2022 - 2023 StarFive Technology + */ + +#include +#include +#include +#include +#include +#include "jh7110-cryp.h" +#include + +#define STARFIVE_SM4_REGS_OFFSET 0x4100 +#define STARFIVE_SM4_SM4DIO0R (STARFIVE_SM4_REGS_OFFSET + 0x0) +#define STARFIVE_SM4_KEY0 (STARFIVE_SM4_REGS_OFFSET + 0x4) +#define STARFIVE_SM4_KEY1 (STARFIVE_SM4_REGS_OFFSET + 0x8) +#define STARFIVE_SM4_KEY2 (STARFIVE_SM4_REGS_OFFSET + 0xC) +#define STARFIVE_SM4_KEY3 (STARFIVE_SM4_REGS_OFFSET + 0x10) +#define STARFIVE_SM4_IV0 (STARFIVE_SM4_REGS_OFFSET + 0x14) +#define STARFIVE_SM4_IV1 (STARFIVE_SM4_REGS_OFFSET + 0x18) +#define STARFIVE_SM4_IV2 (STARFIVE_SM4_REGS_OFFSET + 0x1c) +#define STARFIVE_SM4_IV3 (STARFIVE_SM4_REGS_OFFSET + 0x20) +#define STARFIVE_SM4_CSR (STARFIVE_SM4_REGS_OFFSET + 0x24) +#define STARFIVE_SM4_NONCE0 (STARFIVE_SM4_REGS_OFFSET + 0x30) +#define STARFIVE_SM4_NONCE1 (STARFIVE_SM4_REGS_OFFSET + 0x34) +#define STARFIVE_SM4_NONCE2 (STARFIVE_SM4_REGS_OFFSET + 0x38) +#define STARFIVE_SM4_NONCE3 (STARFIVE_SM4_REGS_OFFSET + 0x3c) +#define STARFIVE_SM4_ALEN0 (STARFIVE_SM4_REGS_OFFSET + 0x40) +#define STARFIVE_SM4_ALEN1 (STARFIVE_SM4_REGS_OFFSET + 0x44) +#define STARFIVE_SM4_MLEN0 (STARFIVE_SM4_REGS_OFFSET + 0x48) +#define STARFIVE_SM4_MLEN1 (STARFIVE_SM4_REGS_OFFSET + 0x4c) +#define STARFIVE_SM4_IVLEN (STARFIVE_SM4_REGS_OFFSET + 0x50) + +#define FLG_MODE_MASK GENMASK(2, 0) +#define FLG_ENCRYPT BIT(4) + +/* Misc */ +#define CCM_B0_ADATA 0x40 +#define SM4_BLOCK_32 (SM4_BLOCK_SIZE / sizeof(u32)) + +static inline int starfive_sm4_wait_done(struct starfive_cryp_dev *cryp) +{ + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM4_CSR, status, + status & STARFIVE_SM4_DONE, 10, 100000); +} + +static inline int starfive_sm4_wait_keydone(struct starfive_cryp_dev *cryp) +{ + u32 status; + + return readl_relaxed_poll_timeout(cryp->base + STARFIVE_SM4_CSR, status, + status & STARFIVE_SM4_KEY_DONE, 10, 100000); +} + +static inline int is_encrypt(struct starfive_cryp_dev *cryp) +{ + return cryp->flags & FLG_ENCRYPT; +} + +static int starfive_sm4_aead_write_key(struct starfive_cryp_ctx *ctx, u32 = hw_mode) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + unsigned int value; + u32 *key =3D (u32 *)ctx->key; + + writel(key[0], cryp->base + STARFIVE_SM4_KEY0); + writel(key[1], cryp->base + STARFIVE_SM4_KEY1); + writel(key[2], cryp->base + STARFIVE_SM4_KEY2); + writel(key[3], cryp->base + STARFIVE_SM4_KEY3); + + value =3D readl(ctx->cryp->base + STARFIVE_SM4_CSR); + + if (hw_mode =3D=3D STARFIVE_SM4_MODE_GCM) + value |=3D STARFIVE_SM4_GCM_START; + else + value |=3D STARFIVE_SM4_CCM_START; + + writel(value, cryp->base + STARFIVE_SM4_CSR); + + if (starfive_sm4_wait_keydone(cryp)) + return -ETIMEDOUT; + + return 0; +} + +static inline void starfive_sm4_set_alen(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + + writel(upper_32_bits(cryp->assoclen), cryp->base + STARFIVE_SM4_ALEN0); + writel(lower_32_bits(cryp->assoclen), cryp->base + STARFIVE_SM4_ALEN1); +} + +static inline void starfive_sm4_set_mlen(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + + writel(upper_32_bits(cryp->total_in), cryp->base + STARFIVE_SM4_MLEN0); + writel(lower_32_bits(cryp->total_in), cryp->base + STARFIVE_SM4_MLEN1); +} + +static inline int starfive_sm4_ccm_check_iv(const u8 *iv) +{ + /* 2 <=3D L <=3D 8, so 1 <=3D L' <=3D 7. */ + if (iv[0] < 1 || iv[0] > 7) + return -EINVAL; + + return 0; +} + +static inline void starfive_sm4_write_iv(struct starfive_cryp_ctx *ctx, u3= 2 *iv) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + + writel(iv[0], cryp->base + STARFIVE_SM4_IV0); + writel(iv[1], cryp->base + STARFIVE_SM4_IV1); + writel(iv[2], cryp->base + STARFIVE_SM4_IV2); + writel(iv[3], cryp->base + STARFIVE_SM4_IV3); +} + +static inline void starfive_sm4_get_iv(struct starfive_cryp_dev *cryp, u32= *iv) +{ + iv[0] =3D readl(cryp->base + STARFIVE_SM4_IV0); + iv[1] =3D readl(cryp->base + STARFIVE_SM4_IV1); + iv[2] =3D readl(cryp->base + STARFIVE_SM4_IV2); + iv[3] =3D readl(cryp->base + STARFIVE_SM4_IV3); +} + +static inline void starfive_sm4_write_nonce(struct starfive_cryp_ctx *ctx,= u32 *nonce) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + + writel(nonce[0], cryp->base + STARFIVE_SM4_NONCE0); + writel(nonce[1], cryp->base + STARFIVE_SM4_NONCE1); + writel(nonce[2], cryp->base + STARFIVE_SM4_NONCE2); + writel(nonce[3], cryp->base + STARFIVE_SM4_NONCE3); +} + +static int starfive_sm4_write_key(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + u32 *key =3D (u32 *)ctx->key; + + writel(key[0], cryp->base + STARFIVE_SM4_KEY0); + writel(key[1], cryp->base + STARFIVE_SM4_KEY1); + writel(key[2], cryp->base + STARFIVE_SM4_KEY2); + writel(key[3], cryp->base + STARFIVE_SM4_KEY3); + + if (starfive_sm4_wait_keydone(cryp)) + return -ETIMEDOUT; + + return 0; +} + +static int starfive_sm4_ccm_init(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + u8 iv[SM4_BLOCK_SIZE], b0[SM4_BLOCK_SIZE]; + unsigned int textlen; + + memcpy(iv, cryp->req.areq->iv, SM4_BLOCK_SIZE); + memset(iv + SM4_BLOCK_SIZE - 1 - iv[0], 0, iv[0] + 1); + + /* Build B0 */ + memcpy(b0, iv, SM4_BLOCK_SIZE); + + b0[0] |=3D (8 * ((cryp->authsize - 2) / 2)); + + if (cryp->assoclen) + b0[0] |=3D CCM_B0_ADATA; + + textlen =3D cryp->total_in; + + b0[SM4_BLOCK_SIZE - 2] =3D textlen >> 8; + b0[SM4_BLOCK_SIZE - 1] =3D textlen & 0xFF; + + starfive_sm4_write_nonce(ctx, (u32 *)b0); + + return 0; +} + +static int starfive_sm4_hw_init(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + struct starfive_cryp_dev *cryp =3D ctx->cryp; + u32 hw_mode; + int ret =3D 0; + + /* reset */ + rctx->csr.sm4.v =3D 0; + rctx->csr.sm4.sm4rst =3D 1; + writel(rctx->csr.sm4.v, cryp->base + STARFIVE_SM4_CSR); + + /* csr setup */ + hw_mode =3D cryp->flags & FLG_MODE_MASK; + + rctx->csr.sm4.v =3D 0; + rctx->csr.sm4.mode =3D hw_mode; + rctx->csr.sm4.cmode =3D !is_encrypt(cryp); + rctx->csr.sm4.stmode =3D STARFIVE_SM4_MODE_XFB_1; + + if (cryp->side_chan) { + rctx->csr.sm4.delay_sm4 =3D 1; + rctx->csr.sm4.vsm4_start =3D 1; + } + + writel(rctx->csr.sm4.v, cryp->base + STARFIVE_SM4_CSR); + + switch (hw_mode) { + case STARFIVE_SM4_MODE_GCM: + starfive_sm4_set_alen(ctx); + starfive_sm4_set_mlen(ctx); + writel(GCM_AES_IV_SIZE, cryp->base + STARFIVE_SM4_IVLEN); + ret =3D starfive_sm4_aead_write_key(ctx, hw_mode); + if (ret) + return ret; + + starfive_sm4_write_iv(ctx, (void *)cryp->req.areq->iv); + break; + case STARFIVE_SM4_MODE_CCM: + starfive_sm4_set_alen(ctx); + starfive_sm4_set_mlen(ctx); + starfive_sm4_ccm_init(ctx); + ret =3D starfive_sm4_aead_write_key(ctx, hw_mode); + if (ret) + return ret; + break; + case STARFIVE_SM4_MODE_CBC: + case STARFIVE_SM4_MODE_CTR: + starfive_sm4_write_iv(ctx, (void *)cryp->req.sreq->iv); + ret =3D starfive_sm4_write_key(ctx); + if (ret) + return ret; + break; + case STARFIVE_SM4_MODE_ECB: + ret =3D starfive_sm4_write_key(ctx); + if (ret) + return ret; + break; + default: + return -EINVAL; + } + + return 0; +} + +static int starfive_sm4_read_authtag(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + int i; + + if ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_GCM) { + cryp->tag_out[0] =3D readl(cryp->base + STARFIVE_SM4_NONCE0); + cryp->tag_out[1] =3D readl(cryp->base + STARFIVE_SM4_NONCE1); + cryp->tag_out[2] =3D readl(cryp->base + STARFIVE_SM4_NONCE2); + cryp->tag_out[3] =3D readl(cryp->base + STARFIVE_SM4_NONCE3); + } else { + for (i =3D 0; i < SM4_BLOCK_32; i++) + cryp->tag_out[i] =3D readl(cryp->base + STARFIVE_SM4_SM4DIO0R); + } + + if (is_encrypt(cryp)) { + scatterwalk_map_and_copy(cryp->tag_out, rctx->out_sg, + cryp->total_in, cryp->authsize, 1); + } else { + if (crypto_memneq(cryp->tag_in, cryp->tag_out, cryp->authsize)) + return dev_err_probe(cryp->dev, -EBADMSG, + "Failed tag verification\n"); + } + + return 0; +} + +static void starfive_sm4_finish_req(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + int err =3D 0; + + if (cryp->authsize) + err =3D starfive_sm4_read_authtag(ctx); + + if ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_CBC || + (cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_CTR) + starfive_sm4_get_iv(cryp, (void *)cryp->req.sreq->iv); + + if (cryp->authsize) + crypto_finalize_aead_request(cryp->engine, cryp->req.areq, err); + else + crypto_finalize_skcipher_request(cryp->engine, cryp->req.sreq, + err); +} + +static int starfive_sm4_gcm_write_adata(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + u32 *buffer; + int total_len, loop; + + total_len =3D ALIGN(cryp->assoclen, SM4_BLOCK_SIZE) / sizeof(unsigned int= ); + buffer =3D (u32 *)rctx->adata; + + for (loop =3D 0; loop < total_len; loop +=3D 4) { + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE0); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE1); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE2); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_NONCE3); + buffer++; + + if (starfive_sm4_wait_done(cryp)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, + "Timeout processing gcm aad block"); + } + + return 0; +} + +static int starfive_sm4_ccm_write_adata(struct starfive_cryp_ctx *ctx) +{ + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct starfive_cryp_request_ctx *rctx =3D ctx->rctx; + u32 *buffer; + int total_len, loop; + + buffer =3D (u32 *)rctx->adata; + total_len =3D ALIGN(cryp->assoclen + 2, SM4_BLOCK_SIZE) / sizeof(unsigned= int); + + for (loop =3D 0; loop < total_len; loop +=3D 4) { + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + writel(*buffer, cryp->base + STARFIVE_SM4_SM4DIO0R); + buffer++; + + if (starfive_sm4_wait_done(cryp)) + return dev_err_probe(cryp->dev, -ETIMEDOUT, + "Timeout processing ccm aad block"); + } + + return 0; +} + +static void starfive_sm4_dma_done(void *param) +{ + struct starfive_cryp_dev *cryp =3D param; + + complete(&cryp->dma_done); +} + +static void starfive_sm4_dma_init(struct starfive_cryp_dev *cryp) +{ + cryp->cfg_in.direction =3D DMA_MEM_TO_DEV; + cryp->cfg_in.src_addr_width =3D DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.dst_addr_width =3D DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_in.src_maxburst =3D cryp->dma_maxburst; + cryp->cfg_in.dst_maxburst =3D cryp->dma_maxburst; + cryp->cfg_in.dst_addr =3D cryp->phys_base + STARFIVE_SM_ALG_FIFO_IN_OFFSE= T; + + dmaengine_slave_config(cryp->tx, &cryp->cfg_in); + + cryp->cfg_out.direction =3D DMA_DEV_TO_MEM; + cryp->cfg_out.src_addr_width =3D DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_out.dst_addr_width =3D DMA_SLAVE_BUSWIDTH_8_BYTES; + cryp->cfg_out.src_maxburst =3D cryp->dma_maxburst; + cryp->cfg_out.dst_maxburst =3D cryp->dma_maxburst; + cryp->cfg_out.src_addr =3D cryp->phys_base + STARFIVE_SM_ALG_FIFO_OUT_OFF= SET; + + dmaengine_slave_config(cryp->rx, &cryp->cfg_out); + + init_completion(&cryp->dma_done); +} + +static int starfive_sm4_dma_xfer(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst, + int len) +{ + struct dma_async_tx_descriptor *in_desc, *out_desc; + union starfive_sm_alg_cr alg_cr; + int ret =3D 0, in_save, out_save; + + alg_cr.v =3D 0; + alg_cr.start =3D 1; + alg_cr.sm4_dma_en =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + in_save =3D sg_dma_len(src); + out_save =3D sg_dma_len(dst); + + writel(ALIGN(len, SM4_BLOCK_SIZE), cryp->base + STARFIVE_SM_DMA_IN_LEN_OF= FSET); + writel(ALIGN(len, SM4_BLOCK_SIZE), cryp->base + STARFIVE_SM_DMA_OUT_LEN_O= FFSET); + + sg_dma_len(src) =3D ALIGN(len, SM4_BLOCK_SIZE); + sg_dma_len(dst) =3D ALIGN(len, SM4_BLOCK_SIZE); + + out_desc =3D dmaengine_prep_slave_sg(cryp->rx, dst, 1, DMA_DEV_TO_MEM, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!out_desc) { + ret =3D -EINVAL; + goto dma_err; + } + + out_desc->callback =3D starfive_sm4_dma_done; + out_desc->callback_param =3D cryp; + + reinit_completion(&cryp->dma_done); + dmaengine_submit(out_desc); + dma_async_issue_pending(cryp->rx); + + in_desc =3D dmaengine_prep_slave_sg(cryp->tx, src, 1, DMA_MEM_TO_DEV, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + if (!in_desc) { + ret =3D -EINVAL; + goto dma_err; + } + + dmaengine_submit(in_desc); + dma_async_issue_pending(cryp->tx); + + if (!wait_for_completion_timeout(&cryp->dma_done, + msecs_to_jiffies(1000))) + ret =3D -ETIMEDOUT; + +dma_err: + sg_dma_len(src) =3D in_save; + sg_dma_len(dst) =3D out_save; + + alg_cr.v =3D 0; + alg_cr.clear =3D 1; + writel(alg_cr.v, cryp->base + STARFIVE_SM_ALG_CR_OFFSET); + + return ret; +} + +static int starfive_sm4_map_sg(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst) +{ + struct scatterlist *stsg, *dtsg; + struct scatterlist _src[2], _dst[2]; + unsigned int remain =3D cryp->total_in; + unsigned int len, src_nents, dst_nents; + int ret; + + if (src =3D=3D dst) { + for (stsg =3D src, dtsg =3D dst; remain > 0; + stsg =3D sg_next(stsg), dtsg =3D sg_next(dtsg)) { + src_nents =3D dma_map_sg(cryp->dev, stsg, 1, DMA_BIDIRECTIONAL); + if (src_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg error\n"); + + dst_nents =3D src_nents; + + len =3D min(sg_dma_len(stsg), remain); + + ret =3D starfive_sm4_dma_xfer(cryp, stsg, dtsg, len); + dma_unmap_sg(cryp->dev, stsg, 1, DMA_BIDIRECTIONAL); + if (ret) + return ret; + + remain -=3D len; + } + } else { + for (stsg =3D src, dtsg =3D dst;;) { + src_nents =3D dma_map_sg(cryp->dev, stsg, 1, DMA_TO_DEVICE); + if (src_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg src error\n"); + + dst_nents =3D dma_map_sg(cryp->dev, dtsg, 1, DMA_FROM_DEVICE); + if (dst_nents =3D=3D 0) + return dev_err_probe(cryp->dev, -ENOMEM, + "dma_map_sg dst error\n"); + + len =3D min(sg_dma_len(stsg), sg_dma_len(dtsg)); + len =3D min(len, remain); + + ret =3D starfive_sm4_dma_xfer(cryp, stsg, dtsg, len); + dma_unmap_sg(cryp->dev, stsg, 1, DMA_TO_DEVICE); + dma_unmap_sg(cryp->dev, dtsg, 1, DMA_FROM_DEVICE); + if (ret) + return ret; + + remain -=3D len; + if (remain =3D=3D 0) + break; + + if (sg_dma_len(stsg) - len) { + stsg =3D scatterwalk_ffwd(_src, stsg, len); + dtsg =3D sg_next(dtsg); + } else if (sg_dma_len(dtsg) - len) { + dtsg =3D scatterwalk_ffwd(_dst, dtsg, len); + stsg =3D sg_next(stsg); + } else { + stsg =3D sg_next(stsg); + dtsg =3D sg_next(dtsg); + } + } + } + + return 0; +} + +static int starfive_sm4_do_one_req(struct crypto_engine *engine, void *are= q) +{ + struct skcipher_request *req =3D + container_of(areq, struct skcipher_request, base); + struct starfive_cryp_ctx *ctx =3D + crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)); + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct starfive_cryp_request_ctx *rctx =3D skcipher_request_ctx(req); + int ret; + + cryp->req.sreq =3D req; + cryp->total_in =3D req->cryptlen; + cryp->total_out =3D req->cryptlen; + cryp->assoclen =3D 0; + cryp->authsize =3D 0; + + rctx->in_sg =3D req->src; + rctx->out_sg =3D req->dst; + + ctx->rctx =3D rctx; + + ret =3D starfive_sm4_hw_init(ctx); + if (ret) + return ret; + + starfive_sm4_dma_init(cryp); + + ret =3D starfive_sm4_map_sg(cryp, rctx->in_sg, rctx->out_sg); + if (ret) + return ret; + + starfive_sm4_finish_req(ctx); + + return 0; +} + +static int starfive_sm4_init_tfm(struct crypto_skcipher *tfm, + const char *alg_name) +{ + struct starfive_cryp_ctx *ctx =3D crypto_skcipher_ctx(tfm); + + ctx->cryp =3D starfive_cryp_find_dev(ctx); + if (!ctx->cryp) + return -ENODEV; + + ctx->skcipher_fbk =3D crypto_alloc_skcipher(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + if (IS_ERR(ctx->skcipher_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->skcipher_fbk), + "%s() failed to allocate fallback for %s\n", + __func__, alg_name); + + crypto_skcipher_set_reqsize(tfm, sizeof(struct starfive_cryp_request_ctx)= + + sizeof(struct skcipher_request)); + + return 0; +} + +static void starfive_sm4_exit_tfm(struct crypto_skcipher *tfm) +{ + struct starfive_cryp_ctx *ctx =3D crypto_skcipher_ctx(tfm); + + crypto_free_skcipher(ctx->skcipher_fbk); +} + +static int starfive_sm4_aead_do_one_req(struct crypto_engine *engine, void= *areq) +{ + struct aead_request *req =3D + container_of(areq, struct aead_request, base); + struct starfive_cryp_ctx *ctx =3D + crypto_aead_ctx(crypto_aead_reqtfm(req)); + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct starfive_cryp_request_ctx *rctx =3D aead_request_ctx(req); + struct scatterlist _dst[2], _src[2]; + int ret; + + cryp->req.areq =3D req; + cryp->assoclen =3D req->assoclen; + cryp->authsize =3D crypto_aead_authsize(crypto_aead_reqtfm(req)); + + if (is_encrypt(cryp)) { + cryp->total_in =3D req->cryptlen; + cryp->total_out =3D req->cryptlen; + } else { + cryp->total_in =3D req->cryptlen - cryp->authsize; + cryp->total_out =3D cryp->total_in; + scatterwalk_map_and_copy(cryp->tag_in, req->src, + cryp->total_in + cryp->assoclen, + cryp->authsize, 0); + } + + if (cryp->assoclen) { + if ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_CCM) { + rctx->adata =3D kzalloc(cryp->assoclen + 2 + SM4_BLOCK_SIZE, GFP_KERNEL= ); + if (!rctx->adata) + return -ENOMEM; + + /* Append 2 bytes zeroes at the start of ccm aad */ + rctx->adata[0] =3D 0; + rctx->adata[1] =3D 0; + + sg_copy_to_buffer(req->src, + sg_nents_for_len(req->src, cryp->assoclen), + &rctx->adata[2], cryp->assoclen); + } else { + rctx->adata =3D kzalloc(cryp->assoclen + SM4_BLOCK_SIZE, GFP_KERNEL); + if (!rctx->adata) + return dev_err_probe(cryp->dev, -ENOMEM, + "Failed to alloc memory for adata"); + + sg_copy_to_buffer(req->src, + sg_nents_for_len(req->src, cryp->assoclen), + rctx->adata, cryp->assoclen); + } + } + + rctx->in_sg =3D scatterwalk_ffwd(_src, req->src, cryp->assoclen); + if (req->src =3D=3D req->dst) + rctx->out_sg =3D rctx->in_sg; + else + rctx->out_sg =3D scatterwalk_ffwd(_dst, req->dst, cryp->assoclen); + + if (cryp->total_in) + sg_zero_buffer(rctx->in_sg, sg_nents(rctx->in_sg), + sg_dma_len(rctx->in_sg) - cryp->total_in, + cryp->total_in); + + ctx->rctx =3D rctx; + + ret =3D starfive_sm4_hw_init(ctx); + if (ret) + return ret; + + if (!cryp->assoclen) + goto write_text; + + if ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_CCM) + ret =3D starfive_sm4_ccm_write_adata(ctx); + else + ret =3D starfive_sm4_gcm_write_adata(ctx); + + kfree(rctx->adata); + + if (ret) + return ret; + +write_text: + if (!cryp->total_in) + goto finish_req; + + starfive_sm4_dma_init(cryp); + + ret =3D starfive_sm4_map_sg(cryp, rctx->in_sg, rctx->out_sg); + if (ret) + return ret; + +finish_req: + starfive_sm4_finish_req(ctx); + return 0; +} + +static int starfive_sm4_aead_init_tfm(struct crypto_aead *tfm, + const char *alg_name) +{ + struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(tfm); + + ctx->cryp =3D starfive_cryp_find_dev(ctx); + if (!ctx->cryp) + return -ENODEV; + + ctx->aead_fbk =3D crypto_alloc_aead(alg_name, 0, + CRYPTO_ALG_NEED_FALLBACK); + if (IS_ERR(ctx->aead_fbk)) + return dev_err_probe(ctx->cryp->dev, PTR_ERR(ctx->aead_fbk), + "%s() failed to allocate fallback for %s\n", + __func__, alg_name); + + crypto_aead_set_reqsize(tfm, sizeof(struct starfive_cryp_ctx) + + sizeof(struct aead_request)); + + return 0; +} + +static void starfive_sm4_aead_exit_tfm(struct crypto_aead *tfm) +{ + struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(tfm); + + crypto_free_aead(ctx->aead_fbk); +} + +static bool starfive_sm4_check_unaligned(struct starfive_cryp_dev *cryp, + struct scatterlist *src, + struct scatterlist *dst) +{ + struct scatterlist *tsg; + int i; + + for_each_sg(src, tsg, sg_nents(src), i) + if (!IS_ALIGNED(tsg->length, SM4_BLOCK_SIZE) && + !sg_is_last(tsg)) + return true; + + if (src !=3D dst) + for_each_sg(dst, tsg, sg_nents(dst), i) + if (!IS_ALIGNED(tsg->length, SM4_BLOCK_SIZE) && + !sg_is_last(tsg)) + return true; + + return false; +} + +static int starfive_sm4_do_fallback(struct skcipher_request *req, bool enc) +{ + struct starfive_cryp_ctx *ctx =3D + crypto_skcipher_ctx(crypto_skcipher_reqtfm(req)); + struct skcipher_request *subreq =3D skcipher_request_ctx(req); + + skcipher_request_set_tfm(subreq, ctx->skcipher_fbk); + skcipher_request_set_callback(subreq, req->base.flags, + req->base.complete, + req->base.data); + skcipher_request_set_crypt(subreq, req->src, req->dst, + req->cryptlen, req->iv); + + return enc ? crypto_skcipher_encrypt(subreq) : + crypto_skcipher_decrypt(subreq); +} + +static int starfive_sm4_crypt(struct skcipher_request *req, unsigned long = flags) +{ + struct crypto_skcipher *tfm =3D crypto_skcipher_reqtfm(req); + struct starfive_cryp_ctx *ctx =3D crypto_skcipher_ctx(tfm); + struct starfive_cryp_dev *cryp =3D ctx->cryp; + unsigned int blocksize_align =3D crypto_skcipher_blocksize(tfm) - 1; + + cryp->flags =3D flags; + + if ((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_ECB || + (cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_CBC) + if (req->cryptlen & blocksize_align) + return -EINVAL; + + if (starfive_sm4_check_unaligned(cryp, req->src, req->dst)) + return starfive_sm4_do_fallback(req, is_encrypt(cryp)); + + return crypto_transfer_skcipher_request_to_engine(cryp->engine, req); +} + +static int starfive_sm4_aead_do_fallback(struct aead_request *req, bool en= c) +{ + struct starfive_cryp_ctx *ctx =3D + crypto_aead_ctx(crypto_aead_reqtfm(req)); + struct aead_request *subreq =3D aead_request_ctx(req); + + aead_request_set_tfm(subreq, ctx->aead_fbk); + aead_request_set_callback(subreq, req->base.flags, + req->base.complete, + req->base.data); + aead_request_set_crypt(subreq, req->src, req->dst, + req->cryptlen, req->iv); + aead_request_set_ad(subreq, req->assoclen); + + return enc ? crypto_aead_encrypt(subreq) : + crypto_aead_decrypt(subreq); +} + +static int starfive_sm4_aead_crypt(struct aead_request *req, unsigned long= flags) +{ + struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(crypto_aead_reqtfm(req)= ); + struct starfive_cryp_dev *cryp =3D ctx->cryp; + struct scatterlist *src, *dst, _src[2], _dst[2]; + + cryp->flags =3D flags; + + /* sm4-ccm does not support tag verification for non-aligned text, + * use fallback for ccm decryption instead. + */ + if (((cryp->flags & FLG_MODE_MASK) =3D=3D STARFIVE_SM4_MODE_CCM) && + !is_encrypt(cryp)) + return starfive_sm4_aead_do_fallback(req, 0); + + src =3D scatterwalk_ffwd(_src, req->src, req->assoclen); + + if (req->src =3D=3D req->dst) + dst =3D src; + else + dst =3D scatterwalk_ffwd(_dst, req->dst, req->assoclen); + + if (starfive_sm4_check_unaligned(cryp, src, dst)) + return starfive_sm4_aead_do_fallback(req, is_encrypt(cryp)); + + return crypto_transfer_aead_request_to_engine(cryp->engine, req); +} + +static int starfive_sm4_setkey(struct crypto_skcipher *tfm, const u8 *key, + unsigned int keylen) +{ + struct starfive_cryp_ctx *ctx =3D crypto_skcipher_ctx(tfm); + + if (!key || !keylen) + return -EINVAL; + + if (keylen !=3D SM4_KEY_SIZE) + return -EINVAL; + + memcpy(ctx->key, key, keylen); + ctx->keylen =3D keylen; + + return crypto_skcipher_setkey(ctx->skcipher_fbk, key, keylen); +} + +static int starfive_sm4_aead_setkey(struct crypto_aead *tfm, const u8 *key, + unsigned int keylen) +{ + struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(tfm); + + if (!key || !keylen) + return -EINVAL; + + if (keylen !=3D SM4_KEY_SIZE) + return -EINVAL; + + memcpy(ctx->key, key, keylen); + ctx->keylen =3D keylen; + + return crypto_aead_setkey(ctx->aead_fbk, key, keylen); +} + +static int starfive_sm4_gcm_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(tfm); + int ret; + + ret =3D crypto_gcm_check_authsize(authsize); + if (ret) + return ret; + + return crypto_aead_setauthsize(ctx->aead_fbk, authsize); +} + +static int starfive_sm4_ccm_setauthsize(struct crypto_aead *tfm, + unsigned int authsize) +{ + struct starfive_cryp_ctx *ctx =3D crypto_aead_ctx(tfm); + + switch (authsize) { + case 4: + case 6: + case 8: + case 10: + case 12: + case 14: + case 16: + break; + default: + return -EINVAL; + } + + return crypto_aead_setauthsize(ctx->aead_fbk, authsize); +} + +static int starfive_sm4_ecb_encrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_ECB | FLG_ENCRYPT); +} + +static int starfive_sm4_ecb_decrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_ECB); +} + +static int starfive_sm4_cbc_encrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CBC | FLG_ENCRYPT); +} + +static int starfive_sm4_cbc_decrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CBC); +} + +static int starfive_sm4_ctr_encrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CTR | FLG_ENCRYPT); +} + +static int starfive_sm4_ctr_decrypt(struct skcipher_request *req) +{ + return starfive_sm4_crypt(req, STARFIVE_SM4_MODE_CTR); +} + +static int starfive_sm4_gcm_encrypt(struct aead_request *req) +{ + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_GCM | FLG_ENCRYPT); +} + +static int starfive_sm4_gcm_decrypt(struct aead_request *req) +{ + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_GCM); +} + +static int starfive_sm4_ccm_encrypt(struct aead_request *req) +{ + int ret; + + ret =3D starfive_sm4_ccm_check_iv(req->iv); + if (ret) + return ret; + + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_CCM | FLG_ENCRYPT); +} + +static int starfive_sm4_ccm_decrypt(struct aead_request *req) +{ + int ret; + + ret =3D starfive_sm4_ccm_check_iv(req->iv); + if (ret) + return ret; + + return starfive_sm4_aead_crypt(req, STARFIVE_SM4_MODE_CCM); +} + +static int starfive_sm4_ecb_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_sm4_init_tfm(tfm, "ecb(sm4-generic)"); +} + +static int starfive_sm4_cbc_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_sm4_init_tfm(tfm, "cbc(sm4-generic)"); +} + +static int starfive_sm4_ctr_init_tfm(struct crypto_skcipher *tfm) +{ + return starfive_sm4_init_tfm(tfm, "ctr(sm4-generic)"); +} + +static int starfive_sm4_ccm_aead_init_tfm(struct crypto_aead *tfm) +{ + return starfive_sm4_aead_init_tfm(tfm, "ccm_base(ctr(sm4-generic),cbcmac(= sm4-generic))"); +} + +static int starfive_sm4_gcm_aead_init_tfm(struct crypto_aead *tfm) +{ + return starfive_sm4_aead_init_tfm(tfm, "gcm_base(ctr(sm4-generic),ghash-g= eneric)"); +} + +static struct skcipher_engine_alg skcipher_sm4[] =3D { +{ + .base.init =3D starfive_sm4_ecb_init_tfm, + .base.exit =3D starfive_sm4_exit_tfm, + .base.setkey =3D starfive_sm4_setkey, + .base.encrypt =3D starfive_sm4_ecb_encrypt, + .base.decrypt =3D starfive_sm4_ecb_decrypt, + .base.min_keysize =3D SM4_KEY_SIZE, + .base.max_keysize =3D SM4_KEY_SIZE, + .base.base =3D { + .cra_name =3D "ecb(sm4)", + .cra_driver_name =3D "starfive-ecb-sm4", + .cra_priority =3D 200, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize =3D SM4_BLOCK_SIZE, + .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), + .cra_alignmask =3D 0xf, + .cra_module =3D THIS_MODULE, + }, + .op =3D { + .do_one_request =3D starfive_sm4_do_one_req, + }, +}, { + .base.init =3D starfive_sm4_ctr_init_tfm, + .base.exit =3D starfive_sm4_exit_tfm, + .base.setkey =3D starfive_sm4_setkey, + .base.encrypt =3D starfive_sm4_ctr_encrypt, + .base.decrypt =3D starfive_sm4_ctr_decrypt, + .base.min_keysize =3D SM4_KEY_SIZE, + .base.max_keysize =3D SM4_KEY_SIZE, + .base.ivsize =3D SM4_BLOCK_SIZE, + .base.base =3D { + .cra_name =3D "ctr(sm4)", + .cra_driver_name =3D "starfive-ctr-sm4", + .cra_priority =3D 200, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize =3D 1, + .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), + .cra_alignmask =3D 0xf, + .cra_module =3D THIS_MODULE, + }, + .op =3D { + .do_one_request =3D starfive_sm4_do_one_req, + }, +}, { + .base.init =3D starfive_sm4_cbc_init_tfm, + .base.exit =3D starfive_sm4_exit_tfm, + .base.setkey =3D starfive_sm4_setkey, + .base.encrypt =3D starfive_sm4_cbc_encrypt, + .base.decrypt =3D starfive_sm4_cbc_decrypt, + .base.min_keysize =3D SM4_KEY_SIZE, + .base.max_keysize =3D SM4_KEY_SIZE, + .base.ivsize =3D SM4_BLOCK_SIZE, + .base.base =3D { + .cra_name =3D "cbc(sm4)", + .cra_driver_name =3D "starfive-cbc-sm4", + .cra_priority =3D 200, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize =3D SM4_BLOCK_SIZE, + .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), + .cra_alignmask =3D 0xf, + .cra_module =3D THIS_MODULE, + }, + .op =3D { + .do_one_request =3D starfive_sm4_do_one_req, + }, +}, +}; + +static struct aead_engine_alg aead_sm4[] =3D { +{ + .base.setkey =3D starfive_sm4_aead_setkey, + .base.setauthsize =3D starfive_sm4_gcm_setauthsize, + .base.encrypt =3D starfive_sm4_gcm_encrypt, + .base.decrypt =3D starfive_sm4_gcm_decrypt, + .base.init =3D starfive_sm4_gcm_aead_init_tfm, + .base.exit =3D starfive_sm4_aead_exit_tfm, + .base.ivsize =3D GCM_AES_IV_SIZE, + .base.maxauthsize =3D SM4_BLOCK_SIZE, + .base.base =3D { + .cra_name =3D "gcm(sm4)", + .cra_driver_name =3D "starfive-gcm-sm4", + .cra_priority =3D 200, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize =3D 1, + .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), + .cra_alignmask =3D 0xf, + .cra_module =3D THIS_MODULE, + }, + .op =3D { + .do_one_request =3D starfive_sm4_aead_do_one_req, + }, +}, { + .base.setkey =3D starfive_sm4_aead_setkey, + .base.setauthsize =3D starfive_sm4_ccm_setauthsize, + .base.encrypt =3D starfive_sm4_ccm_encrypt, + .base.decrypt =3D starfive_sm4_ccm_decrypt, + .base.init =3D starfive_sm4_ccm_aead_init_tfm, + .base.exit =3D starfive_sm4_aead_exit_tfm, + .base.ivsize =3D SM4_BLOCK_SIZE, + .base.maxauthsize =3D SM4_BLOCK_SIZE, + .base.base =3D { + .cra_name =3D "ccm(sm4)", + .cra_driver_name =3D "starfive-ccm-sm4", + .cra_priority =3D 200, + .cra_flags =3D CRYPTO_ALG_ASYNC | + CRYPTO_ALG_NEED_FALLBACK, + .cra_blocksize =3D 1, + .cra_ctxsize =3D sizeof(struct starfive_cryp_ctx), + .cra_alignmask =3D 0xf, + .cra_module =3D THIS_MODULE, + }, + .op =3D { + .do_one_request =3D starfive_sm4_aead_do_one_req, + }, +}, +}; + +int starfive_sm4_register_algs(void) +{ + int ret; + + ret =3D crypto_engine_register_skciphers(skcipher_sm4, ARRAY_SIZE(skciphe= r_sm4)); + if (ret) + return ret; + + ret =3D crypto_engine_register_aeads(aead_sm4, ARRAY_SIZE(aead_sm4)); + if (ret) + crypto_engine_unregister_skciphers(skcipher_sm4, ARRAY_SIZE(skcipher_sm4= )); + + return ret; +} + +void starfive_sm4_unregister_algs(void) +{ + crypto_engine_unregister_aeads(aead_sm4, ARRAY_SIZE(aead_sm4)); + crypto_engine_unregister_skciphers(skcipher_sm4, ARRAY_SIZE(skcipher_sm4)= ); +} --=20 2.34.1