From nobody Fri Nov 7 11:47:42 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1547892669046725.3276814373546; Sat, 19 Jan 2019 02:11:09 -0800 (PST) Received: from localhost ([127.0.0.1]:53833 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gknal-0007TE-Rb for importer@patchew.org; Sat, 19 Jan 2019 05:11:07 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55333) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gknTb-0001N9-8D for qemu-devel@nongnu.org; Sat, 19 Jan 2019 05:03:45 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gknTZ-00083B-GK for qemu-devel@nongnu.org; Sat, 19 Jan 2019 05:03:43 -0500 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]:50644) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gknTZ-0007qx-5V for qemu-devel@nongnu.org; Sat, 19 Jan 2019 05:03:41 -0500 Received: by mail-wm1-x342.google.com with SMTP id n190so6683569wmd.0 for ; Sat, 19 Jan 2019 02:03:23 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id j8sm113868847wmd.0.2019.01.19.02.03.20 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 19 Jan 2019 02:03:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Rpvk67SOKANBS4Yvg060Izb+4UTq2jlqpeTx8nVQIjg=; b=MGzaIeD7wJKaKwvWm/Qqq1w+3SeAt+VR2vJGvl/1X9k2atgGaOWvK5B1GUPuGfjO5N uI4r/OiAVXoChCCBN5oEFKP51WEAkswgsl0mRijtyAs8MABrrfYCs75dcCnZ8I3zDnO3 689trkwS2EfIAmEXE6Jdpo2xmAv8zkfhc3EetaL+osna9UZPbyeJUALBmWKPGRRLeQ1S D3BmnKSk48fnTcppDRPIAyYb2IZLt2BRVBa+Zaf6T/M4Kvmg0hn60Ek45HuY8nIRkgYS 1faq+l4InhbmXuW4q98nPLNAh9DOhfjfbB1PVdab7bsX15fiyXJmM4FW/fabhrv9LN6+ oX6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Rpvk67SOKANBS4Yvg060Izb+4UTq2jlqpeTx8nVQIjg=; b=qQfMNKuoTf1BGmLQgd/nlcjn2xDrSrio+yZzzRN5UlmCtD6mzuOvjIMD+qrThfeJni dRZmZhFKmAMyy2+BEsGyxaPjeeJUeG50nUcuNXRhmxCC7g/mgaTSdVV7UeEphQfq3oR/ Y8utue7HE9eKDb8TC4NtqL45BpTPFB8rnvl5eM12BHc8ATgsIN75g32GjBQNQJP8IOeD YbGPYf+b/kIGUSYbTDWOzCiA2alT5Eas+gRch3T7MW63G/mN8F+3SCaMwRfnqfy+4NDG 2dCQfZRmfEYo8c4s9h3ozppYYyYSi1ggl48HbHxEKMScZ5M6Kcb0MiNop/FxgG+drnOk U5JQ== X-Gm-Message-State: AJcUukeCU3ITip2dsCmmulXBIbxZYOHcvj8O0T2KLoelA/StrEZAtHLe Wznbs7fesLjgIKfFJIm8eldj/31A X-Google-Smtp-Source: ALg8bN6E3gyJAiQmy0qub8dveGEH/Uhbzaenr4uqhTj2vSGkejet8b1Gg4U8jYtPqYA1OlX2izudRQ== X-Received: by 2002:a1c:dc86:: with SMTP id t128mr19417174wmg.42.1547892202105; Sat, 19 Jan 2019 02:03:22 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 19 Jan 2019 12:03:07 +0200 Message-Id: <20190119100315.23985-3-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190119100315.23985-1-marcel.apfelbaum@gmail.com> References: <20190119100315.23985-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::342 Subject: [Qemu-devel] [PATCH PULL 02/10] hw/pvrdma: Remove max-sge command-line param X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: cohuck@redhat.com, liq3ea@163.com, yuval.shaia@oracle.com, kamalheib1@gmail.com, philmd@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Yuval Shaia This parameter has no effect, fix it. The function init_dev_caps sets the front-end's max-sge to MAX_SGE. Then it checks backend's max-sge and adjust it accordingly (we can't send more than what the device supports). On send and recv we need to make sure the num_sge in the WQE does not exceeds the backend device capability. This check is done in pvrdma level so check on rdma level is deleted. Signed-off-by: Yuval Shaia Message-Id: <20190109194123.3468-1-yuval.shaia@oracle.com> Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- docs/pvrdma.txt | 1 - hw/rdma/rdma_backend.c | 23 ++--------------------- hw/rdma/rdma_backend.h | 11 +++++++++++ hw/rdma/vmw/pvrdma_main.c | 10 +++++----- hw/rdma/vmw/pvrdma_qp_ops.c | 24 ++++++++++++++++++++++++ 5 files changed, 42 insertions(+), 27 deletions(-) diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt index 81959378b1..0f0dd8a7e5 100644 --- a/docs/pvrdma.txt +++ b/docs/pvrdma.txt @@ -156,7 +156,6 @@ Ethernet function can be used for other Ethernet purpos= es such as IP. specify the port to use. If not set 1 will be used. - dev-caps-max-mr-size: The maximum size of MR. - dev-caps-max-qp: Maximum number of QPs. -- dev-caps-max-sge: Maximum number of SGE elements in WR. - dev-caps-max-cq: Maximum number of CQs. - dev-caps-max-mr: Maximum number of MRs. - dev-caps-max-pd: Maximum number of PDs. diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index c28bfbd44d..16dca69ee9 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -32,17 +32,6 @@ #include "rdma_rm.h" #include "rdma_backend.h" =20 -/* Vendor Errors */ -#define VENDOR_ERR_FAIL_BACKEND 0x201 -#define VENDOR_ERR_TOO_MANY_SGES 0x202 -#define VENDOR_ERR_NOMEM 0x203 -#define VENDOR_ERR_QP0 0x204 -#define VENDOR_ERR_INV_NUM_SGE 0x205 -#define VENDOR_ERR_MAD_SEND 0x206 -#define VENDOR_ERR_INVLKEY 0x207 -#define VENDOR_ERR_MR_SMALL 0x208 -#define VENDOR_ERR_INV_MAD_BUFF 0x209 - #define THR_NAME_LEN 16 #define THR_POLL_TO 5000 =20 @@ -475,11 +464,6 @@ void rdma_backend_post_send(RdmaBackendDev *backend_de= v, } =20 pr_dbg("num_sge=3D%d\n", num_sge); - if (!num_sge || num_sge > MAX_SGE) { - pr_dbg("invalid num_sge=3D%d\n", num_sge); - complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_NUM_SGE, ctx); - return; - } =20 bctx =3D g_malloc0(sizeof(*bctx)); bctx->up_ctx =3D ctx; @@ -602,11 +586,6 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_de= v, } =20 pr_dbg("num_sge=3D%d\n", num_sge); - if (!num_sge || num_sge > MAX_SGE) { - pr_dbg("invalid num_sge=3D%d\n", num_sge); - complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_NUM_SGE, ctx); - return; - } =20 bctx =3D g_malloc0(sizeof(*bctx)); bctx->up_ctx =3D ctx; @@ -942,6 +921,8 @@ static int init_device_caps(RdmaBackendDev *backend_dev, return -EIO; } =20 + dev_attr->max_sge =3D MAX_SGE; + CHK_ATTR(dev_attr, backend_dev->dev_attr, max_mr_size, "%" PRId64); CHK_ATTR(dev_attr, backend_dev->dev_attr, max_qp, "%d"); CHK_ATTR(dev_attr, backend_dev->dev_attr, max_sge, "%d"); diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 8cae40f827..a9ba40ae48 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -22,6 +22,17 @@ #include "rdma_rm_defs.h" #include "rdma_backend_defs.h" =20 +/* Vendor Errors */ +#define VENDOR_ERR_FAIL_BACKEND 0x201 +#define VENDOR_ERR_TOO_MANY_SGES 0x202 +#define VENDOR_ERR_NOMEM 0x203 +#define VENDOR_ERR_QP0 0x204 +#define VENDOR_ERR_INV_NUM_SGE 0x205 +#define VENDOR_ERR_MAD_SEND 0x206 +#define VENDOR_ERR_INVLKEY 0x207 +#define VENDOR_ERR_MR_SMALL 0x208 +#define VENDOR_ERR_INV_MAD_BUFF 0x209 + /* Add definition for QP0 and QP1 as there is no userspace enums for them = */ enum ibv_special_qp_type { IBV_QPT_SMI =3D 0, diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 838ad8a949..d2bdb5ba8c 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -43,7 +43,6 @@ static Property pvrdma_dev_properties[] =3D { DEFINE_PROP_UINT64("dev-caps-max-mr-size", PVRDMADev, dev_attr.max_mr_= size, MAX_MR_SIZE), DEFINE_PROP_INT32("dev-caps-max-qp", PVRDMADev, dev_attr.max_qp, MAX_Q= P), - DEFINE_PROP_INT32("dev-caps-max-sge", PVRDMADev, dev_attr.max_sge, MAX= _SGE), DEFINE_PROP_INT32("dev-caps-max-cq", PVRDMADev, dev_attr.max_cq, MAX_C= Q), DEFINE_PROP_INT32("dev-caps-max-mr", PVRDMADev, dev_attr.max_mr, MAX_M= R), DEFINE_PROP_INT32("dev-caps-max-pd", PVRDMADev, dev_attr.max_pd, MAX_P= D), @@ -549,8 +548,9 @@ static void init_dev_caps(PVRDMADev *dev) sizeof(struct pvrdma_rq_wqe_hdr)); =20 dev->dev_attr.max_qp_wr =3D pg_tbl_bytes / - (wr_sz + sizeof(struct pvrdma_sge) * MAX_SGE= ) - - TARGET_PAGE_SIZE; /* First page is ring stat= e */ + (wr_sz + sizeof(struct pvrdma_sge) * + dev->dev_attr.max_sge) - TARGET_PAGE_SIZE; + /* First page is ring state ^^^^ */ pr_dbg("max_qp_wr=3D%d\n", dev->dev_attr.max_qp_wr); =20 dev->dev_attr.max_cqe =3D pg_tbl_bytes / sizeof(struct pvrdma_cqe) - @@ -626,8 +626,6 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) =20 init_regs(pdev); =20 - init_dev_caps(dev); - rc =3D init_msix(pdev, errp); if (rc) { goto out; @@ -640,6 +638,8 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) goto out; } =20 + init_dev_caps(dev); + rc =3D rdma_rm_init(&dev->rdma_dev_res, &dev->dev_attr, errp); if (rc) { goto out; diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 300471a4c9..465bee8641 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -121,6 +121,16 @@ static void pvrdma_qp_ops_comp_handler(void *ctx, stru= ct ibv_wc *wc) g_free(ctx); } =20 +static void complete_with_error(uint32_t vendor_err, void *ctx) +{ + struct ibv_wc wc =3D {0}; + + wc.status =3D IBV_WC_GENERAL_ERR; + wc.vendor_err =3D vendor_err; + + pvrdma_qp_ops_comp_handler(ctx, &wc); +} + void pvrdma_qp_ops_fini(void) { rdma_backend_unregister_comp_handler(); @@ -182,6 +192,13 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) return -EIO; } =20 + if (wqe->hdr.num_sge > dev->dev_attr.max_sge) { + pr_dbg("Invalid num_sge=3D%d (max %d)\n", wqe->hdr.num_sge, + dev->dev_attr.max_sge); + complete_with_error(VENDOR_ERR_INV_NUM_SGE, comp_ctx); + continue; + } + rdma_backend_post_send(&dev->backend_dev, &qp->backend_qp, qp->qp_= type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.nu= m_sge, sgid_idx, sgid, @@ -227,6 +244,13 @@ int pvrdma_qp_recv(PVRDMADev *dev, uint32_t qp_handle) comp_ctx->cqe.qp =3D qp_handle; comp_ctx->cqe.opcode =3D IBV_WC_RECV; =20 + if (wqe->hdr.num_sge > dev->dev_attr.max_sge) { + pr_dbg("Invalid num_sge=3D%d (max %d)\n", wqe->hdr.num_sge, + dev->dev_attr.max_sge); + complete_with_error(VENDOR_ERR_INV_NUM_SGE, comp_ctx); + continue; + } + rdma_backend_post_recv(&dev->backend_dev, &dev->rdma_dev_res, &qp->backend_qp, qp->qp_type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.nu= m_sge, --=20 2.17.1