From nobody Fri May 15 09:18:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07E10C433EF for ; Thu, 21 Apr 2022 10:44:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1388510AbiDUKrF (ORCPT ); Thu, 21 Apr 2022 06:47:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1388497AbiDUKqw (ORCPT ); Thu, 21 Apr 2022 06:46:52 -0400 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0D8726ACE for ; Thu, 21 Apr 2022 03:44:02 -0700 (PDT) Received: by mail-pg1-x532.google.com with SMTP id q19so4307559pgm.6 for ; Thu, 21 Apr 2022 03:44:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4xRXz2QffjyjmDDlKiShW1CJlOS8mA0XsBK3pkYpP+I=; b=DqnSHvaeErWhOEkWVhlOnY41iEh06ev+a1qRDRozBjGZ9vgbd/X50nM13li+qw/m1J 0QmJKV2DlYLu0e9TfB8ytbB0agF69s/QsVOOEHJLLiZnFngrYnHgWG6MOdolpgGWXcyP E0XffZjEoSqavXUDTdLq0gZIeMsyctYDujFyIQjMt1ggJKe034PYoaB4DrbKzNT4NQRx cpgBZknOo/6+w/XqGdUDbzsFiTFr4pTlS26t4m98pe7SHsqZn0ftKJOG/IFU8TeKEIk2 fH4FX/Gfwz+s41KGyq479Ef6zgW5Bo5faBAuaGrs73DdHiZmpdRKoAGSxGmdnk9vMK05 ryWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4xRXz2QffjyjmDDlKiShW1CJlOS8mA0XsBK3pkYpP+I=; b=Le9NDTunbqNrZmLTilt6bVvXVmNCTmx7BoTCMhlYVJgN0LdMSqgmyObIO5xr55N83Z stv5wcgbp/CandMI9lzNyDRZ5mTQWmzkuSGkridhDsIkygiGX+At7UOwSlQyDHzrKU+1 7BExtSJbaZjDLYVUsojHbpbjcolFFdaYlM6w28igo5kNP0SKAFYAMXHBkZx9oktFKd6o BR8A5UE1pfrQanKZwdP4SbYtdjsaMGHJyprWuDH1XmRjP3SC86sHXfyYGQQ4orIH3KSX eheoRwKv+Fco8EiKoeVqF30XhfNP/Wugddt2S6oyZVUJJ46qEC3XFgZCu7KnptfiOg8n VYBg== X-Gm-Message-State: AOAM531yVHULRneIVVBgWZjA0l1lKD4dReUet7l6FxVNf6u+0XiAcAGi 30+d1TkGmja0hhIp6UrFW8tC3A== X-Google-Smtp-Source: ABdhPJx3J6EybljrSE0kR7wa6nYhtYJdkrzbcJvFpRDLrjMdArSb9/qGTQ4mMPNGgsPKW4k4AdQomQ== X-Received: by 2002:a63:e355:0:b0:39d:7956:6d3c with SMTP id o21-20020a63e355000000b0039d79566d3cmr23475265pgj.385.1650537842321; Thu, 21 Apr 2022 03:44:02 -0700 (PDT) Received: from always-x1.bytedance.net ([61.120.150.69]) by smtp.gmail.com with ESMTPSA id w7-20020aa79547000000b0050ad0e82e6dsm3772485pfq.215.2022.04.21.03.43.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Apr 2022 03:44:01 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, davem@davemloft.net, zhenwei pi Subject: [PATCH v3 1/5] virtio-crypto: use private buffer for control request Date: Thu, 21 Apr 2022 18:40:12 +0800 Message-Id: <20220421104016.453458-2-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220421104016.453458-1-pizhenwei@bytedance.com> References: <20220421104016.453458-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Originally, all of the control requests share a single buffer( ctrl & input & ctrl_status fields in struct virtio_crypto), this allows queue depth 1 only, the performance of control queue gets limited by this design. In this patch, each request allocates request buffer dynamically, and free buffer after request, it's possible to optimize control queue depth in the next step. A necessary comment is already in code, still describe it again: /* * Note: there are padding fields in request, clear them to zero before * sending to host, * Ex, virtio_crypto_ctrl_request::ctrl::u::destroy_session::padding[48] */ So use kzalloc to allocate buffer of struct virtio_crypto_ctrl_request. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Signed-off-by: zhenwei pi --- drivers/crypto/virtio/Makefile | 1 + .../virtio/virtio_crypto_akcipher_algs.c | 90 ++++++------ drivers/crypto/virtio/virtio_crypto_common.c | 39 +++++ drivers/crypto/virtio/virtio_crypto_common.h | 19 ++- .../virtio/virtio_crypto_skcipher_algs.c | 133 ++++++++---------- 5 files changed, 156 insertions(+), 126 deletions(-) create mode 100644 drivers/crypto/virtio/virtio_crypto_common.c diff --git a/drivers/crypto/virtio/Makefile b/drivers/crypto/virtio/Makefile index bfa6cbae342e..49c1fa80e465 100644 --- a/drivers/crypto/virtio/Makefile +++ b/drivers/crypto/virtio/Makefile @@ -3,5 +3,6 @@ obj-$(CONFIG_CRYPTO_DEV_VIRTIO) +=3D virtio_crypto.o virtio_crypto-objs :=3D \ virtio_crypto_skcipher_algs.o \ virtio_crypto_akcipher_algs.o \ + virtio_crypto_common.o \ virtio_crypto_mgr.o \ virtio_crypto_core.o diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/= crypto/virtio/virtio_crypto_akcipher_algs.c index f3ec9420215e..9561bc2df62b 100644 --- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c @@ -102,8 +102,8 @@ static int virtio_crypto_alg_akcipher_init_session(stru= ct virtio_crypto_akcipher { struct scatterlist outhdr_sg, key_sg, inhdr_sg, *sgs[3]; struct virtio_crypto *vcrypto =3D ctx->vcrypto; + struct virtio_crypto_ctrl_request *vc_ctrl_req; uint8_t *pkey; - unsigned int inlen; int err; unsigned int num_out =3D 0, num_in =3D 0; =20 @@ -111,98 +111,91 @@ static int virtio_crypto_alg_akcipher_init_session(st= ruct virtio_crypto_akcipher if (!pkey) return -ENOMEM; =20 - spin_lock(&vcrypto->ctrl_lock); - memcpy(&vcrypto->ctrl.header, header, sizeof(vcrypto->ctrl.header)); - memcpy(&vcrypto->ctrl.u, para, sizeof(vcrypto->ctrl.u)); - vcrypto->input.status =3D cpu_to_le32(VIRTIO_CRYPTO_ERR); + vc_ctrl_req =3D kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) { + err =3D -ENOMEM; + goto out; + } =20 - sg_init_one(&outhdr_sg, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + memcpy(&vc_ctrl_req->ctrl.header, header, sizeof(vc_ctrl_req->ctrl.header= )); + memcpy(&vc_ctrl_req->ctrl.u, para, sizeof(vc_ctrl_req->ctrl.u)); + sg_init_one(&outhdr_sg, &vc_ctrl_req->ctrl, sizeof(vc_ctrl_req->ctrl)); sgs[num_out++] =3D &outhdr_sg; =20 sg_init_one(&key_sg, pkey, keylen); sgs[num_out++] =3D &key_sg; =20 - sg_init_one(&inhdr_sg, &vcrypto->input, sizeof(vcrypto->input)); + vc_ctrl_req->input.status =3D cpu_to_le32(VIRTIO_CRYPTO_ERR); + sg_init_one(&inhdr_sg, &vc_ctrl_req->input, sizeof(vc_ctrl_req->input)); sgs[num_out + num_in++] =3D &inhdr_sg; =20 - err =3D virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto= , GFP_ATOMIC); + err =3D virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_c= trl_req); if (err < 0) goto out; =20 - virtqueue_kick(vcrypto->ctrl_vq); - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); - - if (le32_to_cpu(vcrypto->input.status) !=3D VIRTIO_CRYPTO_OK) { + if (le32_to_cpu(vc_ctrl_req->input.status) !=3D VIRTIO_CRYPTO_OK) { + pr_err("virtio_crypto: Create session failed status: %u\n", + le32_to_cpu(vc_ctrl_req->input.status)); err =3D -EINVAL; goto out; } =20 - ctx->session_id =3D le64_to_cpu(vcrypto->input.session_id); + ctx->session_id =3D le64_to_cpu(vc_ctrl_req->input.session_id); ctx->session_valid =3D true; err =3D 0; =20 out: - spin_unlock(&vcrypto->ctrl_lock); + kfree(vc_ctrl_req); kfree_sensitive(pkey); =20 - if (err < 0) - pr_err("virtio_crypto: Create session failed status: %u\n", - le32_to_cpu(vcrypto->input.status)); - return err; } =20 static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_a= kcipher_ctx *ctx) { struct scatterlist outhdr_sg, inhdr_sg, *sgs[2]; + struct virtio_crypto_ctrl_request *vc_ctrl_req; struct virtio_crypto_destroy_session_req *destroy_session; struct virtio_crypto *vcrypto =3D ctx->vcrypto; - unsigned int num_out =3D 0, num_in =3D 0, inlen; + unsigned int num_out =3D 0, num_in =3D 0; int err; =20 - spin_lock(&vcrypto->ctrl_lock); - if (!ctx->session_valid) { - err =3D 0; - goto out; - } - vcrypto->ctrl_status.status =3D VIRTIO_CRYPTO_ERR; - vcrypto->ctrl.header.opcode =3D cpu_to_le32(VIRTIO_CRYPTO_AKCIPHER_DESTRO= Y_SESSION); - vcrypto->ctrl.header.queue_id =3D 0; + if (!ctx->session_valid) + return 0; =20 - destroy_session =3D &vcrypto->ctrl.u.destroy_session; + vc_ctrl_req =3D kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) + return -ENOMEM; + + vc_ctrl_req->ctrl.header.opcode =3D cpu_to_le32(VIRTIO_CRYPTO_AKCIPHER_DE= STROY_SESSION); + vc_ctrl_req->ctrl.header.queue_id =3D 0; + + destroy_session =3D &vc_ctrl_req->ctrl.u.destroy_session; destroy_session->session_id =3D cpu_to_le64(ctx->session_id); =20 - sg_init_one(&outhdr_sg, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + sg_init_one(&outhdr_sg, &vc_ctrl_req->ctrl, sizeof(vc_ctrl_req->ctrl)); sgs[num_out++] =3D &outhdr_sg; =20 - sg_init_one(&inhdr_sg, &vcrypto->ctrl_status.status, sizeof(vcrypto->ctrl= _status.status)); + vc_ctrl_req->ctrl_status.status =3D VIRTIO_CRYPTO_ERR; + sg_init_one(&inhdr_sg, &vc_ctrl_req->ctrl_status.status, + sizeof(vc_ctrl_req->ctrl_status.status)); sgs[num_out + num_in++] =3D &inhdr_sg; =20 - err =3D virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, num_in, vcrypto= , GFP_ATOMIC); + err =3D virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_c= trl_req); if (err < 0) goto out; =20 - virtqueue_kick(vcrypto->ctrl_vq); - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); - - if (vcrypto->ctrl_status.status !=3D VIRTIO_CRYPTO_OK) { + if (vc_ctrl_req->ctrl_status.status !=3D VIRTIO_CRYPTO_OK) { err =3D -EINVAL; + pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%l= lx\n", + vc_ctrl_req->ctrl_status.status, destroy_session->session_id); goto out; } =20 err =3D 0; ctx->session_valid =3D false; - out: - spin_unlock(&vcrypto->ctrl_lock); - if (err < 0) { - pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%l= lx\n", - vcrypto->ctrl_status.status, destroy_session->session_id); - } + kfree(vc_ctrl_req); =20 return err; } @@ -210,14 +203,11 @@ static int virtio_crypto_alg_akcipher_close_session(s= truct virtio_crypto_akciphe static int __virtio_crypto_akcipher_do_req(struct virtio_crypto_akcipher_r= equest *vc_akcipher_req, struct akcipher_request *req, struct data_queue *data_vq) { - struct virtio_crypto_akcipher_ctx *ctx =3D vc_akcipher_req->akcipher_ctx; struct virtio_crypto_request *vc_req =3D &vc_akcipher_req->base; - struct virtio_crypto *vcrypto =3D ctx->vcrypto; struct virtio_crypto_op_data_req *req_data =3D vc_req->req_data; struct scatterlist *sgs[4], outhdr_sg, inhdr_sg, srcdata_sg, dstdata_sg; void *src_buf =3D NULL, *dst_buf =3D NULL; unsigned int num_out =3D 0, num_in =3D 0; - int node =3D dev_to_node(&vcrypto->vdev->dev); unsigned long flags; int ret =3D -ENOMEM; bool verify =3D vc_akcipher_req->opcode =3D=3D VIRTIO_CRYPTO_AKCIPHER_VER= IFY; @@ -228,7 +218,7 @@ static int __virtio_crypto_akcipher_do_req(struct virti= o_crypto_akcipher_request sgs[num_out++] =3D &outhdr_sg; =20 /* src data */ - src_buf =3D kcalloc_node(src_len, 1, GFP_KERNEL, node); + src_buf =3D kcalloc(src_len, 1, GFP_KERNEL); if (!src_buf) goto err; =20 @@ -243,7 +233,7 @@ static int __virtio_crypto_akcipher_do_req(struct virti= o_crypto_akcipher_request sgs[num_out++] =3D &srcdata_sg; =20 /* dst data */ - dst_buf =3D kcalloc_node(req->dst_len, 1, GFP_KERNEL, node); + dst_buf =3D kcalloc(req->dst_len, 1, GFP_KERNEL); if (!dst_buf) goto err; =20 diff --git a/drivers/crypto/virtio/virtio_crypto_common.c b/drivers/crypto/= virtio/virtio_crypto_common.c new file mode 100644 index 000000000000..e65125a74db2 --- /dev/null +++ b/drivers/crypto/virtio/virtio_crypto_common.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Common functions and helpers + * + * Authors: zhenwei pi + * + * Copyright 2022 Bytedance CO., LTD. + */ + +#include "virtio_crypto_common.h" + +int virtio_crypto_ctrl_vq_request(struct virtio_crypto *vcrypto, struct sc= atterlist *sgs[], + unsigned int out_sgs, unsigned int in_sgs, + struct virtio_crypto_ctrl_request *vc_ctrl_req) +{ + int err; + unsigned int inlen; + unsigned long flags; + + spin_lock_irqsave(&vcrypto->ctrl_lock, flags); + err =3D virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, out_sgs, in_sgs, vc_ctrl= _req, GFP_ATOMIC); + if (err < 0) { + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); + return err; + } + + virtqueue_kick(vcrypto->ctrl_vq); + + /* + * Trapping into the hypervisor, so the request should be + * handled immediately. + */ + while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && + !virtqueue_is_broken(vcrypto->ctrl_vq)) + cpu_relax(); + + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); + + return 0; +} diff --git a/drivers/crypto/virtio/virtio_crypto_common.h b/drivers/crypto/= virtio/virtio_crypto_common.h index e693d4ee83a6..d2a20fe6e13e 100644 --- a/drivers/crypto/virtio/virtio_crypto_common.h +++ b/drivers/crypto/virtio/virtio_crypto_common.h @@ -13,6 +13,7 @@ #include #include #include +#include =20 =20 /* Internal representation of a data virtqueue */ @@ -65,11 +66,6 @@ struct virtio_crypto { /* Maximum size of per request */ u64 max_size; =20 - /* Control VQ buffers: protected by the ctrl_lock */ - struct virtio_crypto_op_ctrl_req ctrl; - struct virtio_crypto_session_input input; - struct virtio_crypto_inhdr ctrl_status; - unsigned long status; atomic_t ref_count; struct list_head list; @@ -85,6 +81,16 @@ struct virtio_crypto_sym_session_info { __u64 session_id; }; =20 +/* + * Note: there are padding fields in request, clear them to zero before se= nding to host, + * Ex, virtio_crypto_ctrl_request::ctrl::u::destroy_session::padding[48] + */ +struct virtio_crypto_ctrl_request { + struct virtio_crypto_op_ctrl_req ctrl; + struct virtio_crypto_session_input input; + struct virtio_crypto_inhdr ctrl_status; +}; + struct virtio_crypto_request; typedef void (*virtio_crypto_data_callback) (struct virtio_crypto_request *vc_req, int len); @@ -135,4 +141,7 @@ void virtio_crypto_skcipher_algs_unregister(struct virt= io_crypto *vcrypto); int virtio_crypto_akcipher_algs_register(struct virtio_crypto *vcrypto); void virtio_crypto_akcipher_algs_unregister(struct virtio_crypto *vcrypto); =20 +int virtio_crypto_ctrl_vq_request(struct virtio_crypto *vcrypto, struct sc= atterlist *sgs[], + unsigned int out_sgs, unsigned int in_sgs, + struct virtio_crypto_ctrl_request *vc_ctrl_req); #endif /* _VIRTIO_CRYPTO_COMMON_H */ diff --git a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c b/drivers/= crypto/virtio/virtio_crypto_skcipher_algs.c index a618c46a52b8..fef355ff461c 100644 --- a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c @@ -118,11 +118,13 @@ static int virtio_crypto_alg_skcipher_init_session( int encrypt) { struct scatterlist outhdr, key_sg, inhdr, *sgs[3]; - unsigned int tmp; struct virtio_crypto *vcrypto =3D ctx->vcrypto; int op =3D encrypt ? VIRTIO_CRYPTO_OP_ENCRYPT : VIRTIO_CRYPTO_OP_DECRYPT; int err; unsigned int num_out =3D 0, num_in =3D 0; + struct virtio_crypto_ctrl_request *vc_ctrl_req; + struct virtio_crypto_ctrl_header *header; + struct virtio_crypto_sym_create_session_req *sym_create_session; =20 /* * Avoid to do DMA from the stack, switch to using @@ -133,26 +135,27 @@ static int virtio_crypto_alg_skcipher_init_session( if (!cipher_key) return -ENOMEM; =20 - spin_lock(&vcrypto->ctrl_lock); + vc_ctrl_req =3D kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) { + err =3D -ENOMEM; + goto out; + } + /* Pad ctrl header */ - vcrypto->ctrl.header.opcode =3D - cpu_to_le32(VIRTIO_CRYPTO_CIPHER_CREATE_SESSION); - vcrypto->ctrl.header.algo =3D cpu_to_le32(alg); + header =3D &vc_ctrl_req->ctrl.header; + header->opcode =3D cpu_to_le32(VIRTIO_CRYPTO_CIPHER_CREATE_SESSION); + header->algo =3D cpu_to_le32(alg); /* Set the default dataqueue id to 0 */ - vcrypto->ctrl.header.queue_id =3D 0; + header->queue_id =3D 0; =20 - vcrypto->input.status =3D cpu_to_le32(VIRTIO_CRYPTO_ERR); /* Pad cipher's parameters */ - vcrypto->ctrl.u.sym_create_session.op_type =3D - cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER); - vcrypto->ctrl.u.sym_create_session.u.cipher.para.algo =3D - vcrypto->ctrl.header.algo; - vcrypto->ctrl.u.sym_create_session.u.cipher.para.keylen =3D - cpu_to_le32(keylen); - vcrypto->ctrl.u.sym_create_session.u.cipher.para.op =3D - cpu_to_le32(op); - - sg_init_one(&outhdr, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + sym_create_session =3D &vc_ctrl_req->ctrl.u.sym_create_session; + sym_create_session->op_type =3D cpu_to_le32(VIRTIO_CRYPTO_SYM_OP_CIPHER); + sym_create_session->u.cipher.para.algo =3D header->algo; + sym_create_session->u.cipher.para.keylen =3D cpu_to_le32(keylen); + sym_create_session->u.cipher.para.op =3D cpu_to_le32(op); + + sg_init_one(&outhdr, &vc_ctrl_req->ctrl, sizeof(vc_ctrl_req->ctrl)); sgs[num_out++] =3D &outhdr; =20 /* Set key */ @@ -160,45 +163,34 @@ static int virtio_crypto_alg_skcipher_init_session( sgs[num_out++] =3D &key_sg; =20 /* Return status and session id back */ - sg_init_one(&inhdr, &vcrypto->input, sizeof(vcrypto->input)); + vc_ctrl_req->input.status =3D cpu_to_le32(VIRTIO_CRYPTO_ERR); + sg_init_one(&inhdr, &vc_ctrl_req->input, sizeof(vc_ctrl_req->input)); sgs[num_out + num_in++] =3D &inhdr; =20 - err =3D virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, - num_in, vcrypto, GFP_ATOMIC); - if (err < 0) { - spin_unlock(&vcrypto->ctrl_lock); - kfree_sensitive(cipher_key); - return err; - } - virtqueue_kick(vcrypto->ctrl_vq); - - /* - * Trapping into the hypervisor, so the request should be - * handled immediately. - */ - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); + err =3D virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_c= trl_req); + if (err < 0) + goto out; =20 - if (le32_to_cpu(vcrypto->input.status) !=3D VIRTIO_CRYPTO_OK) { - spin_unlock(&vcrypto->ctrl_lock); + if (le32_to_cpu(vc_ctrl_req->input.status) !=3D VIRTIO_CRYPTO_OK) { pr_err("virtio_crypto: Create session failed status: %u\n", - le32_to_cpu(vcrypto->input.status)); - kfree_sensitive(cipher_key); - return -EINVAL; + le32_to_cpu(vc_ctrl_req->input.status)); + err =3D -EINVAL; + goto out; } =20 if (encrypt) ctx->enc_sess_info.session_id =3D - le64_to_cpu(vcrypto->input.session_id); + le64_to_cpu(vc_ctrl_req->input.session_id); else ctx->dec_sess_info.session_id =3D - le64_to_cpu(vcrypto->input.session_id); - - spin_unlock(&vcrypto->ctrl_lock); + le64_to_cpu(vc_ctrl_req->input.session_id); =20 + err =3D 0; +out: + kfree(vc_ctrl_req); kfree_sensitive(cipher_key); - return 0; + + return err; } =20 static int virtio_crypto_alg_skcipher_close_session( @@ -206,21 +198,24 @@ static int virtio_crypto_alg_skcipher_close_session( int encrypt) { struct scatterlist outhdr, status_sg, *sgs[2]; - unsigned int tmp; struct virtio_crypto_destroy_session_req *destroy_session; struct virtio_crypto *vcrypto =3D ctx->vcrypto; int err; unsigned int num_out =3D 0, num_in =3D 0; + struct virtio_crypto_ctrl_request *vc_ctrl_req; + struct virtio_crypto_ctrl_header *header; + + vc_ctrl_req =3D kzalloc(sizeof(*vc_ctrl_req), GFP_KERNEL); + if (!vc_ctrl_req) + return -ENOMEM; =20 - spin_lock(&vcrypto->ctrl_lock); - vcrypto->ctrl_status.status =3D VIRTIO_CRYPTO_ERR; /* Pad ctrl header */ - vcrypto->ctrl.header.opcode =3D - cpu_to_le32(VIRTIO_CRYPTO_CIPHER_DESTROY_SESSION); + header =3D &vc_ctrl_req->ctrl.header; + header->opcode =3D cpu_to_le32(VIRTIO_CRYPTO_CIPHER_DESTROY_SESSION); /* Set the default virtqueue id to 0 */ - vcrypto->ctrl.header.queue_id =3D 0; + header->queue_id =3D 0; =20 - destroy_session =3D &vcrypto->ctrl.u.destroy_session; + destroy_session =3D &vc_ctrl_req->ctrl.u.destroy_session; =20 if (encrypt) destroy_session->session_id =3D @@ -229,37 +224,33 @@ static int virtio_crypto_alg_skcipher_close_session( destroy_session->session_id =3D cpu_to_le64(ctx->dec_sess_info.session_id); =20 - sg_init_one(&outhdr, &vcrypto->ctrl, sizeof(vcrypto->ctrl)); + sg_init_one(&outhdr, &vc_ctrl_req->ctrl, sizeof(vc_ctrl_req->ctrl)); sgs[num_out++] =3D &outhdr; =20 /* Return status and session id back */ - sg_init_one(&status_sg, &vcrypto->ctrl_status.status, - sizeof(vcrypto->ctrl_status.status)); + vc_ctrl_req->ctrl_status.status =3D VIRTIO_CRYPTO_ERR; + sg_init_one(&status_sg, &vc_ctrl_req->ctrl_status.status, + sizeof(vc_ctrl_req->ctrl_status.status)); sgs[num_out + num_in++] =3D &status_sg; =20 - err =3D virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, num_out, - num_in, vcrypto, GFP_ATOMIC); - if (err < 0) { - spin_unlock(&vcrypto->ctrl_lock); - return err; - } - virtqueue_kick(vcrypto->ctrl_vq); - - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &tmp) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); + err =3D virtio_crypto_ctrl_vq_request(vcrypto, sgs, num_out, num_in, vc_c= trl_req); + if (err < 0) + goto out; =20 - if (vcrypto->ctrl_status.status !=3D VIRTIO_CRYPTO_OK) { - spin_unlock(&vcrypto->ctrl_lock); + if (vc_ctrl_req->ctrl_status.status !=3D VIRTIO_CRYPTO_OK) { pr_err("virtio_crypto: Close session failed status: %u, session_id: 0x%l= lx\n", - vcrypto->ctrl_status.status, + vc_ctrl_req->ctrl_status.status, destroy_session->session_id); =20 - return -EINVAL; + err =3D -EINVAL; + goto out; } - spin_unlock(&vcrypto->ctrl_lock); =20 - return 0; + err =3D 0; +out: + kfree(vc_ctrl_req); + + return err; } =20 static int virtio_crypto_alg_skcipher_init_sessions( --=20 2.20.1 From nobody Fri May 15 09:18:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8828DC433EF for ; Thu, 21 Apr 2022 10:44:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1388526AbiDUKrW (ORCPT ); Thu, 21 Apr 2022 06:47:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1388505AbiDUKq7 (ORCPT ); Thu, 21 Apr 2022 06:46:59 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 462422C13C for ; Thu, 21 Apr 2022 03:44:10 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id u15so1076365ple.4 for ; Thu, 21 Apr 2022 03:44:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+wRkA/MgdpGcvHCj6EQvY7xr281zOQma1KCz2azqeQs=; b=jegRWHwuhvBJz/GaSZkEzZ6hoEymXfQqnPcmwCl69wJvDDeaoqtr1Cgv89p6+Zcojk nqtq+U/nuH3bT2ntZlFkEkovPtp+1wfbLu9/yCpKyD550W3fMB/CAVm3c2UQ8ahpqHBb SCHYt7KMoslj6+KDnJh+IQ4kas56WDt/nv4AWtcqZlzM5p78Nzxzw2qQwvAdOtqhvhrr O36N1xKrj/jUQU1FB0MSDuDfiz2tYYeK7p+y7d3jKZWsa2swIVBfYs3mTV4kAHgMRVJh QN+otgEZFGMeADO0yF8zoVJ9skQhWhoJsU/4ofHlv/e5JUgI8Iua5xrOPMsU3mtEMHwP 5EDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+wRkA/MgdpGcvHCj6EQvY7xr281zOQma1KCz2azqeQs=; b=zQBYuVJprY9obzrN4w78PJCk2QGV5LPRnSEwTY/5h9UtECy2yvs59wRX3BSnteRP8o gTTH84H+rdBYTzirH9dIfmnpjwwvQQcQaZDT6tnp4XxTHtuDJC2d/kKhxM2s0EbLylEy XkwW3AAXUAOIKYB/MmLGSUQ12Dxvw3T+nPOJTJYo3ymSg0pBen7yV75AQQC8WwQse8Pc 0wLrzkh+jBTRaLjnG0836AxaCj/nZMk9Uccxx6wHrya+DFFMIk9u2ANFZ8hLRpmk16lV pyPbNGUycBtDkiaDrW70D5KnyDAQO+bUKY23TYPN6csK0qC7u04cO5sknqx/84b+/OHO gp3Q== X-Gm-Message-State: AOAM533c7QYH3NdvEcB9Sm9xFF8YiDfUBJHsbDNuu4aCceDtnpasVs1j n2R3JgTjV84I0796csqhj5pPPA== X-Google-Smtp-Source: ABdhPJwIFNIZf4tx34yay+SImO1/KmEBVip2rmWSTb1FmjkUcJBqELt5K2q2XgxFhv5sfIC42zJwrQ== X-Received: by 2002:a17:902:e8c6:b0:158:f809:311f with SMTP id v6-20020a170902e8c600b00158f809311fmr20858174plg.4.1650537848321; Thu, 21 Apr 2022 03:44:08 -0700 (PDT) Received: from always-x1.bytedance.net ([61.120.150.69]) by smtp.gmail.com with ESMTPSA id w7-20020aa79547000000b0050ad0e82e6dsm3772485pfq.215.2022.04.21.03.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Apr 2022 03:44:07 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, davem@davemloft.net, zhenwei pi Subject: [PATCH v3 2/5] virtio-crypto: wait ctrl queue instead of busy polling Date: Thu, 21 Apr 2022 18:40:13 +0800 Message-Id: <20220421104016.453458-3-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220421104016.453458-1-pizhenwei@bytedance.com> References: <20220421104016.453458-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Originally, after submitting request into virtio crypto control queue, the guest side polls the result from the virt queue. This works like following: CPU0 CPU1 ... CPUx CPUy | | | | \ \ / / \--------spin_lock(&vcrypto->ctrl_lock)-------/ | virtqueue add & kick | busy poll virtqueue | spin_unlock(&vcrypto->ctrl_lock) ... There are two problems: 1, The queue depth is always 1, the performance of a virtio crypto device gets limited. Multi user processes share a single control queue, and hit spin lock race from control queue. Test on Intel Platinum 8260, a single worker gets ~35K/s create/close session operations, and 8 workers get ~40K/s operations with 800% CPU utilization. 2, The control request is supposed to get handled immediately, but in the current implementation of QEMU(v6.2), the vCPU thread kicks another thread to do this work, the latency also gets unstable. Tracking latency of virtio_crypto_alg_akcipher_close_session in 5s: usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 7 | | 4 -> 7 : 72 | | 8 -> 15 : 186485 |************************| 16 -> 31 : 687 | | 32 -> 63 : 5 | | 64 -> 127 : 3 | | 128 -> 255 : 1 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 0 | | 8192 -> 16383 : 2 | | This means that a CPU may hold vcrypto->ctrl_lock as long as 8192~16383u= s. To improve the performance of control queue, a request on control queue wai= ts completion instead of busy polling to reduce lock racing, and gets complete= d by control queue callback. CPU0 CPU1 ... CPUx CPUy | | | | \ \ / / \--------spin_lock(&vcrypto->ctrl_lock)-------/ | virtqueue add & kick | ---------spin_unlock(&vcrypto->ctrl_lock)------ / / \ \ | | | | wait wait wait wait Test this patch, the guest side get ~200K/s operations with 300% CPU utilization. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Signed-off-by: zhenwei pi --- drivers/crypto/virtio/virtio_crypto_common.c | 42 +++++++++++++++----- drivers/crypto/virtio/virtio_crypto_common.h | 8 ++++ drivers/crypto/virtio/virtio_crypto_core.c | 2 +- 3 files changed, 41 insertions(+), 11 deletions(-) diff --git a/drivers/crypto/virtio/virtio_crypto_common.c b/drivers/crypto/= virtio/virtio_crypto_common.c index e65125a74db2..93df73c40dd3 100644 --- a/drivers/crypto/virtio/virtio_crypto_common.c +++ b/drivers/crypto/virtio/virtio_crypto_common.c @@ -8,14 +8,21 @@ =20 #include "virtio_crypto_common.h" =20 +static void virtio_crypto_ctrlq_callback(struct virtio_crypto_ctrl_request= *vc_ctrl_req) +{ + complete(&vc_ctrl_req->compl); +} + int virtio_crypto_ctrl_vq_request(struct virtio_crypto *vcrypto, struct sc= atterlist *sgs[], unsigned int out_sgs, unsigned int in_sgs, struct virtio_crypto_ctrl_request *vc_ctrl_req) { int err; - unsigned int inlen; unsigned long flags; =20 + init_completion(&vc_ctrl_req->compl); + vc_ctrl_req->ctrl_cb =3D virtio_crypto_ctrlq_callback; + spin_lock_irqsave(&vcrypto->ctrl_lock, flags); err =3D virtqueue_add_sgs(vcrypto->ctrl_vq, sgs, out_sgs, in_sgs, vc_ctrl= _req, GFP_ATOMIC); if (err < 0) { @@ -24,16 +31,31 @@ int virtio_crypto_ctrl_vq_request(struct virtio_crypto = *vcrypto, struct scatterl } =20 virtqueue_kick(vcrypto->ctrl_vq); - - /* - * Trapping into the hypervisor, so the request should be - * handled immediately. - */ - while (!virtqueue_get_buf(vcrypto->ctrl_vq, &inlen) && - !virtqueue_is_broken(vcrypto->ctrl_vq)) - cpu_relax(); - spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); =20 + wait_for_completion(&vc_ctrl_req->compl); + return 0; } + +void virtcrypto_ctrlq_callback(struct virtqueue *vq) +{ + struct virtio_crypto *vcrypto =3D vq->vdev->priv; + struct virtio_crypto_ctrl_request *vc_ctrl_req; + unsigned long flags; + unsigned int len; + + spin_lock_irqsave(&vcrypto->ctrl_lock, flags); + do { + virtqueue_disable_cb(vq); + while ((vc_ctrl_req =3D virtqueue_get_buf(vq, &len)) !=3D NULL) { + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); + if (vc_ctrl_req->ctrl_cb) + vc_ctrl_req->ctrl_cb(vc_ctrl_req); + spin_lock_irqsave(&vcrypto->ctrl_lock, flags); + } + if (unlikely(virtqueue_is_broken(vq))) + break; + } while (!virtqueue_enable_cb(vq)); + spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); +} diff --git a/drivers/crypto/virtio/virtio_crypto_common.h b/drivers/crypto/= virtio/virtio_crypto_common.h index d2a20fe6e13e..25b4f22e8605 100644 --- a/drivers/crypto/virtio/virtio_crypto_common.h +++ b/drivers/crypto/virtio/virtio_crypto_common.h @@ -81,6 +81,10 @@ struct virtio_crypto_sym_session_info { __u64 session_id; }; =20 +struct virtio_crypto_ctrl_request; +typedef void (*virtio_crypto_ctrl_callback) + (struct virtio_crypto_ctrl_request *vc_ctrl_req); + /* * Note: there are padding fields in request, clear them to zero before se= nding to host, * Ex, virtio_crypto_ctrl_request::ctrl::u::destroy_session::padding[48] @@ -89,6 +93,8 @@ struct virtio_crypto_ctrl_request { struct virtio_crypto_op_ctrl_req ctrl; struct virtio_crypto_session_input input; struct virtio_crypto_inhdr ctrl_status; + virtio_crypto_ctrl_callback ctrl_cb; + struct completion compl; }; =20 struct virtio_crypto_request; @@ -141,7 +147,9 @@ void virtio_crypto_skcipher_algs_unregister(struct virt= io_crypto *vcrypto); int virtio_crypto_akcipher_algs_register(struct virtio_crypto *vcrypto); void virtio_crypto_akcipher_algs_unregister(struct virtio_crypto *vcrypto); =20 +void virtcrypto_ctrlq_callback(struct virtqueue *vq); int virtio_crypto_ctrl_vq_request(struct virtio_crypto *vcrypto, struct sc= atterlist *sgs[], unsigned int out_sgs, unsigned int in_sgs, struct virtio_crypto_ctrl_request *vc_ctrl_req); + #endif /* _VIRTIO_CRYPTO_COMMON_H */ diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/vi= rtio/virtio_crypto_core.c index c6f482db0bc0..e668d4b1bc6a 100644 --- a/drivers/crypto/virtio/virtio_crypto_core.c +++ b/drivers/crypto/virtio/virtio_crypto_core.c @@ -73,7 +73,7 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi) goto err_names; =20 /* Parameters for control virtqueue */ - callbacks[total_vqs - 1] =3D NULL; + callbacks[total_vqs - 1] =3D virtcrypto_ctrlq_callback; names[total_vqs - 1] =3D "controlq"; =20 /* Allocate/initialize parameters for data virtqueues */ --=20 2.20.1 From nobody Fri May 15 09:18:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1482C433F5 for ; Thu, 21 Apr 2022 10:44:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1388518AbiDUKrQ (ORCPT ); Thu, 21 Apr 2022 06:47:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1388512AbiDUKrM (ORCPT ); Thu, 21 Apr 2022 06:47:12 -0400 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 519152CCAF for ; Thu, 21 Apr 2022 03:44:13 -0700 (PDT) Received: by mail-pl1-x631.google.com with SMTP id t12so4462230pll.7 for ; Thu, 21 Apr 2022 03:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Soqf3DM2EhHRALM+l3+OvMMOLUa3IL3/z83R4lQAUME=; b=MFcgCbgJJ5SK6J0X7stUHJhXJ0mBu8nHN96Z4tvfqDrjvLucvnYZgOGT0QjT597Uby X1P1qxXXARFF8wit/aAxMKwfDXO0rZDn3F4cbxwmxkKokfzYB1z7LhuoaTYegIMgyLqv ly0HoDdhUXeNpgQ1mwtvXwvP/8FrtIQEvkfMEezAoF7dwPoFpG+VMQ6TQ//imHHxOyBL WBSWvxXIsH8lO/KABabvTqewYOam48h/9Eli0b7/wsbHJREAKCSreIl3TDjrsRmS7fZQ qwNH4ZZ/D49sflTZ4fn2bPHMvETRXj/qB6rZXBbEbHddeeXH8ZwfU3nfojguPxeKrs06 SBRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Soqf3DM2EhHRALM+l3+OvMMOLUa3IL3/z83R4lQAUME=; b=vh+ZIoFMC+DfauxwPJ36HZXjqIRpOeATMNy5Dm2WpW/3rw52eD3Dqc09GGNwcFja6s nQlmABUUwhBuT6Xz6DPn0Ewm7o29ZPlrQumvRmaNfKdpT7XHzOdOiPO+5smjMn/8KLRs OkS/x4VM6+DZgSB7BBBTQtLLsr+TgvJIypMnTF/ucDcGGXvnKqdW1EC9pD9poVmEf2I3 ZjKuyVoEeRYUI5LGQ9pAYlj+kEHg4e/eZc4rEgxr62CxHYo9KUNFUrQBZVZrKTWNzjqp ISUphUg4l9CL1OVndtxb6ZtQUH3dCijk8ThnntPmI88gPYh5x7wv3VyWvpOa6uuDIMNN peNw== X-Gm-Message-State: AOAM533/E/YjEeQvrH59rlHjJS93skitnPKkVer8ASmmjGWYNmCmGuVi wAWBZFs/cpzeerQ8kfr6kZSeRw== X-Google-Smtp-Source: ABdhPJwERO7VeCmPyZBwxIsQHqWASpqeFP8RVtxBICGxK+ZjXbAoVUBaCflO3+ba3KBsGrekR9xcJg== X-Received: by 2002:a17:902:bb92:b0:153:4eae:c77e with SMTP id m18-20020a170902bb9200b001534eaec77emr24831615pls.93.1650537852862; Thu, 21 Apr 2022 03:44:12 -0700 (PDT) Received: from always-x1.bytedance.net ([61.120.150.69]) by smtp.gmail.com with ESMTPSA id w7-20020aa79547000000b0050ad0e82e6dsm3772485pfq.215.2022.04.21.03.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Apr 2022 03:44:12 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, davem@davemloft.net, zhenwei pi Subject: [PATCH v3 3/5] virtio-crypto: move helpers into virtio_crypto_common.c Date: Thu, 21 Apr 2022 18:40:14 +0800 Message-Id: <20220421104016.453458-4-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220421104016.453458-1-pizhenwei@bytedance.com> References: <20220421104016.453458-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Move virtcrypto_clear_request and virtcrypto_dataq_callback into virtio_crypto_common.c to make code clear. Then the xx_core.c supports: - probe/remove/irq affinity seting for a virtio device - basic virtio related operations xx_common.c supports: - common helpers/functions for algos Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Signed-off-by: zhenwei pi --- drivers/crypto/virtio/virtio_crypto_common.c | 31 +++++++++++++++++++ drivers/crypto/virtio/virtio_crypto_common.h | 2 ++ drivers/crypto/virtio/virtio_crypto_core.c | 32 -------------------- 3 files changed, 33 insertions(+), 32 deletions(-) diff --git a/drivers/crypto/virtio/virtio_crypto_common.c b/drivers/crypto/= virtio/virtio_crypto_common.c index 93df73c40dd3..4a23524896fe 100644 --- a/drivers/crypto/virtio/virtio_crypto_common.c +++ b/drivers/crypto/virtio/virtio_crypto_common.c @@ -8,6 +8,14 @@ =20 #include "virtio_crypto_common.h" =20 +void virtcrypto_clear_request(struct virtio_crypto_request *vc_req) +{ + if (vc_req) { + kfree_sensitive(vc_req->req_data); + kfree(vc_req->sgs); + } +} + static void virtio_crypto_ctrlq_callback(struct virtio_crypto_ctrl_request= *vc_ctrl_req) { complete(&vc_ctrl_req->compl); @@ -59,3 +67,26 @@ void virtcrypto_ctrlq_callback(struct virtqueue *vq) } while (!virtqueue_enable_cb(vq)); spin_unlock_irqrestore(&vcrypto->ctrl_lock, flags); } + +void virtcrypto_dataq_callback(struct virtqueue *vq) +{ + struct virtio_crypto *vcrypto =3D vq->vdev->priv; + struct virtio_crypto_request *vc_req; + unsigned long flags; + unsigned int len; + unsigned int qid =3D vq->index; + + spin_lock_irqsave(&vcrypto->data_vq[qid].lock, flags); + do { + virtqueue_disable_cb(vq); + while ((vc_req =3D virtqueue_get_buf(vq, &len)) !=3D NULL) { + spin_unlock_irqrestore( + &vcrypto->data_vq[qid].lock, flags); + if (vc_req->alg_cb) + vc_req->alg_cb(vc_req, len); + spin_lock_irqsave( + &vcrypto->data_vq[qid].lock, flags); + } + } while (!virtqueue_enable_cb(vq)); + spin_unlock_irqrestore(&vcrypto->data_vq[qid].lock, flags); +} diff --git a/drivers/crypto/virtio/virtio_crypto_common.h b/drivers/crypto/= virtio/virtio_crypto_common.h index 25b4f22e8605..4d33ed5593d4 100644 --- a/drivers/crypto/virtio/virtio_crypto_common.h +++ b/drivers/crypto/virtio/virtio_crypto_common.h @@ -152,4 +152,6 @@ int virtio_crypto_ctrl_vq_request(struct virtio_crypto = *vcrypto, struct scatterl unsigned int out_sgs, unsigned int in_sgs, struct virtio_crypto_ctrl_request *vc_ctrl_req); =20 +void virtcrypto_dataq_callback(struct virtqueue *vq); + #endif /* _VIRTIO_CRYPTO_COMMON_H */ diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/vi= rtio/virtio_crypto_core.c index e668d4b1bc6a..d8edefcb966c 100644 --- a/drivers/crypto/virtio/virtio_crypto_core.c +++ b/drivers/crypto/virtio/virtio_crypto_core.c @@ -13,38 +13,6 @@ #include "virtio_crypto_common.h" =20 =20 -void -virtcrypto_clear_request(struct virtio_crypto_request *vc_req) -{ - if (vc_req) { - kfree_sensitive(vc_req->req_data); - kfree(vc_req->sgs); - } -} - -static void virtcrypto_dataq_callback(struct virtqueue *vq) -{ - struct virtio_crypto *vcrypto =3D vq->vdev->priv; - struct virtio_crypto_request *vc_req; - unsigned long flags; - unsigned int len; - unsigned int qid =3D vq->index; - - spin_lock_irqsave(&vcrypto->data_vq[qid].lock, flags); - do { - virtqueue_disable_cb(vq); - while ((vc_req =3D virtqueue_get_buf(vq, &len)) !=3D NULL) { - spin_unlock_irqrestore( - &vcrypto->data_vq[qid].lock, flags); - if (vc_req->alg_cb) - vc_req->alg_cb(vc_req, len); - spin_lock_irqsave( - &vcrypto->data_vq[qid].lock, flags); - } - } while (!virtqueue_enable_cb(vq)); - spin_unlock_irqrestore(&vcrypto->data_vq[qid].lock, flags); -} - static int virtcrypto_find_vqs(struct virtio_crypto *vi) { vq_callback_t **callbacks; --=20 2.20.1 From nobody Fri May 15 09:18:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA66DC433F5 for ; Thu, 21 Apr 2022 10:44:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1388558AbiDUKrc (ORCPT ); Thu, 21 Apr 2022 06:47:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1388520AbiDUKrN (ORCPT ); Thu, 21 Apr 2022 06:47:13 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FF952D1D0 for ; Thu, 21 Apr 2022 03:44:18 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id q1so3294736plx.13 for ; Thu, 21 Apr 2022 03:44:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6zOuRYVFWDGuC1PrwNUGb6FpSSDkw8m8NU3bhEpx4ck=; b=kCpy3srq/EiBanbPyFzUd0w0fdOrd3PL2QHCcbTp7TZh1XfM6gSXBFFrxDXiKzePET BwCk62X45XrHU1TRgo+rpF5cPK87lvBqe9Z5X9gX1XXa42IXLMh0wrxSbi/ZzoLCkWyg SG7njy+qJTfzPqsl+gQYTHqx+ULb086FwT7Ngo75IOdDDyyoxoL5pgLeUqjDxGNWgG9G 225aPoSR1vTY6k9fszQ/AZzu6oQaJEoYoAUR+o+wuYp5q30SnukeGjrXpM2hl3VEYIsH JgLtg/EZ6VTM4QfR2EGeiOBXfMD7vkr3EurEZ95KVGyPnNSwjYsuJ7l1lwN1LgNM1bFS TBag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6zOuRYVFWDGuC1PrwNUGb6FpSSDkw8m8NU3bhEpx4ck=; b=x1JNnUfO7APYEy9lGqFMcrI2P9I8UzvC03s1a0cPYx6pwpTz6QjCYlnODYrb5pwNyP vESLH/1LErwbM3QQyLpR+UTkNDt1fhd0MX1LC81eqfc3BAd3ui9XBZrNitJ1gcFWMbDM bpyqHg2WL/b05qQH0UwKGDFshzJAphmtrhzFjt+DP7v4KnqqaQSOhHjXzSxY0sjEOP/T 6z6KNd45986wvDVfda5j6jdhnRdN0EnD08RPkNSLHLdpk8V+fgTxB4rrUdu4Z1RVax+V SoYJ6PTSVVfA47n5cYR3+8hiyY+NxvIUF/WFlJgj752kFwOXhe9Gpl3j9qIn6oBgU6Dl AUvQ== X-Gm-Message-State: AOAM533RI3qkTjMSTxhCRLTso00345343oPGxxf012HA3w/XbinuG8rh FrbCyBtkxGgzDfFUfKYxkJ+ftw== X-Google-Smtp-Source: ABdhPJxu/20L6f4UIu3TjxYb9mGaNdxihBAZzHQCrTg/qmpKmPcYQWNqpuUUco8L1qBhWTdWiI8RRQ== X-Received: by 2002:a17:90b:15c9:b0:1d2:f2f0:e15e with SMTP id lh9-20020a17090b15c900b001d2f2f0e15emr9734310pjb.195.1650537857872; Thu, 21 Apr 2022 03:44:17 -0700 (PDT) Received: from always-x1.bytedance.net ([61.120.150.69]) by smtp.gmail.com with ESMTPSA id w7-20020aa79547000000b0050ad0e82e6dsm3772485pfq.215.2022.04.21.03.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Apr 2022 03:44:17 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, davem@davemloft.net, zhenwei pi Subject: [PATCH v3 4/5] virtio-crypto: adjust dst_len at ops callback Date: Thu, 21 Apr 2022 18:40:15 +0800 Message-Id: <20220421104016.453458-5-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220421104016.453458-1-pizhenwei@bytedance.com> References: <20220421104016.453458-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: lei he For some akcipher operations(eg, decryption of pkcs1pad(rsa)), the length of returned result maybe less than akcipher_req->dst_len, we need to recalculate the actual dst_len through the virt-queue protocol. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Signed-off-by: lei he Signed-off-by: zhenwei pi --- drivers/crypto/virtio/virtio_crypto_akcipher_algs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/= crypto/virtio/virtio_crypto_akcipher_algs.c index 9561bc2df62b..82db86e088c2 100644 --- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c +++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c @@ -90,9 +90,12 @@ static void virtio_crypto_dataq_akcipher_callback(struct= virtio_crypto_request * } =20 akcipher_req =3D vc_akcipher_req->akcipher_req; - if (vc_akcipher_req->opcode !=3D VIRTIO_CRYPTO_AKCIPHER_VERIFY) + if (vc_akcipher_req->opcode !=3D VIRTIO_CRYPTO_AKCIPHER_VERIFY) { + /* actuall length maybe less than dst buffer */ + akcipher_req->dst_len =3D len - sizeof(vc_req->status); sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst), vc_akcipher_req->dst_buf, akcipher_req->dst_len); + } virtio_crypto_akcipher_finalize_req(vc_akcipher_req, akcipher_req, error); } =20 --=20 2.20.1 From nobody Fri May 15 09:18:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E47EC433EF for ; Thu, 21 Apr 2022 10:44:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1388541AbiDUKr1 (ORCPT ); Thu, 21 Apr 2022 06:47:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1388523AbiDUKrN (ORCPT ); Thu, 21 Apr 2022 06:47:13 -0400 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCE892D1FC for ; Thu, 21 Apr 2022 03:44:23 -0700 (PDT) Received: by mail-pf1-x42a.google.com with SMTP id x1so4660889pfj.2 for ; Thu, 21 Apr 2022 03:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Vgyphshva4+1MoZbiUZHYBed15/dj2PxkqwfRwCNPpE=; b=EfTBFRclrg2OJBpUoWZLcqlhoTqhRPEqclktYCgbN7WckD+ipu67xVjpp/KgX7S3df 0IajjdCl6lQ4JqNYZ2giw2+aoayfss17DKxqHwh6fCOCKA+6vJXZux5rCgrI/FvsvgZW 6maMRybm/GRxdmiP7SY1PPpQYHJ3wabRoo8f+BHWM0ec9Z8SSDXgW5f4txOhg+0YNp9e igR+AHPtL0BXLeX/E03XpjTxJnzt+JoQtKDa/lJErvLuZ7Mnh2Vs9vvIAggOKq9U4eSY xmdDLBRPP7BdgUm0SkYvRhfiNAkFaZkxy2rIWHF7ahWI1j//YJPqwXC44BuF6XA7i7kQ FeHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Vgyphshva4+1MoZbiUZHYBed15/dj2PxkqwfRwCNPpE=; b=1ALtdC085fzrZadYARLGkqrmBMnQFNWvVyWQpVWSH3aBBWSBBswv+suiV8qXig+TQG UyzC67sNXJNhExPjo+5/5qVc8pqf6lu0UpiwB6/SZWypAYSNdQ/vJm79wtby9y8qUKLO C1T83GbySswWdfe3/bK9pd9EEL/lkG6trCzL2tTi1Rf0wY4rmUJ/jtNSRm91UkGOi/hK lBI0WKTzOJmgE7rlfbStplsIAEkM/Pc4G6lEgs8XTLXYZf3lQm8rjo/Gbi4ZYgFBoftx Oeqaj/b3yeMPqrL7goX1PYb5KHHiESsOxY2ioHVDrl7qd9r/arIcgXA9PE4ajhrXV68r e9Mw== X-Gm-Message-State: AOAM53370orcWDj7yWm09ntt0IacdAMuQpGceBSgw5B1nlnK7bfW63GN aEIWO5FLYg2rH/4umYp09YvNjg== X-Google-Smtp-Source: ABdhPJyJLS3mZ53Vu2R9I+Mt2sPcqwMG8V444JCZXnqM/k/gHuRh+50aLFK0VyGbedV642Rj9fvPKg== X-Received: by 2002:a65:6e9a:0:b0:382:1804:35c8 with SMTP id bm26-20020a656e9a000000b00382180435c8mr23664506pgb.584.1650537863265; Thu, 21 Apr 2022 03:44:23 -0700 (PDT) Received: from always-x1.bytedance.net ([61.120.150.69]) by smtp.gmail.com with ESMTPSA id w7-20020aa79547000000b0050ad0e82e6dsm3772485pfq.215.2022.04.21.03.44.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Apr 2022 03:44:22 -0700 (PDT) From: zhenwei pi To: arei.gonglei@huawei.com, mst@redhat.com Cc: jasowang@redhat.com, herbert@gondor.apana.org.au, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-crypto@vger.kernel.org, helei.sig11@bytedance.com, davem@davemloft.net, zhenwei pi Subject: [PATCH v3 5/5] virtio-crypto: enable retry for virtio-crypto-dev Date: Thu, 21 Apr 2022 18:40:16 +0800 Message-Id: <20220421104016.453458-6-pizhenwei@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220421104016.453458-1-pizhenwei@bytedance.com> References: <20220421104016.453458-1-pizhenwei@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: lei he Enable retry for virtio-crypto-dev, so that crypto-engine can process cipher-requests parallelly. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Gonglei Signed-off-by: lei he Signed-off-by: zhenwei pi --- drivers/crypto/virtio/virtio_crypto_core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/virtio/virtio_crypto_core.c b/drivers/crypto/vi= rtio/virtio_crypto_core.c index d8edefcb966c..5c0d68c9e894 100644 --- a/drivers/crypto/virtio/virtio_crypto_core.c +++ b/drivers/crypto/virtio/virtio_crypto_core.c @@ -62,7 +62,8 @@ static int virtcrypto_find_vqs(struct virtio_crypto *vi) spin_lock_init(&vi->data_vq[i].lock); vi->data_vq[i].vq =3D vqs[i]; /* Initialize crypto engine */ - vi->data_vq[i].engine =3D crypto_engine_alloc_init(dev, 1); + vi->data_vq[i].engine =3D crypto_engine_alloc_init_and_set(dev, true, NU= LL, 1, + virtqueue_get_vring_size(vqs[i])); if (!vi->data_vq[i].engine) { ret =3D -ENOMEM; goto err_engine; --=20 2.20.1