From nobody Sun Feb 8 12:38:32 2026 Delivered-To: importer@patchew.org Received-SPF: temperror (zoho.com: Error in retrieving data from DNS) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1556977660; cv=none; d=zoho.com; s=zohoarc; b=SYoQnCj2UNuP3eqV4LsaYz1Z7AsKbwACgeZdgaHZP42jT6H5nc1OL3G4MVv6rRMjUqpeDG61vjkIls76H9mJE2tmj+iGQjV0UHAZ/yJLEMUYMhoFpWWWX3SlwHAYVnpxvjaMdIXH1mle/YwAGuJBvrpZa+jRbWDT9GGN4B+aTbc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1556977660; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=2693G5zlF0KPjpGu8OLoWU8DoCSHGNg7d6xJXxwQddg=; b=DtkeLM8QHW+sWmOsUXbwrdssENTH9IWaBV6fM0BmCLJmkINE9jD4bsnZohNIQQSv9N8ftpnLorOQ7G6NiD1v+v4YkYMtfaSh8Y9vPwXDWXoGgDgxNUvDj5z/HQj2UTTOH+Q4QGjIsynEWB01pqwpIbsZya6BDdFTMF9Jt2laYO8= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1556977660919888.6299593876172; Sat, 4 May 2019 06:47:40 -0700 (PDT) Received: from localhost ([127.0.0.1]:56957 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMv0i-00031n-PW for importer@patchew.org; Sat, 04 May 2019 09:47:28 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43984) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMuyw-00021H-01 for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hMuyu-0002Ju-8w for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:37 -0400 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:35323) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hMuyu-0002J1-08 for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:36 -0400 Received: by mail-wr1-x442.google.com with SMTP id h15so5611098wrb.2 for ; Sat, 04 May 2019 06:45:35 -0700 (PDT) Received: from localhost.localdomain ([37.142.144.27]) by smtp.gmail.com with ESMTPSA id t1sm5095610wro.34.2019.05.04.06.45.33 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 04 May 2019 06:45:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2693G5zlF0KPjpGu8OLoWU8DoCSHGNg7d6xJXxwQddg=; b=CLg0hyefSf5BibpnWaV2iEZyDZ3xy+QTcLkrad2YF5Jc+U4lQldIPGqGBW8SyWGtbq WZe2qFmFPaYJpCDkgvwBe7MmhdrKpSmEO5Va/bCgSdJXVTETLlOZqoe/xM7Oh3eqKAnZ CLi68PJVI7ySJs+Y0UCAvyMII5FCqyoBGo9sclGeUWErdq6gjFOF5kQqF6ztVgVUqH6F V1kviBOPj4V1KDOeK8BVzQaip6QTchN8YojBOHM8QPwFEBhss2aciI6VkNBL7LgXn3o2 jzrKQsOlYcQNZ0gstFxs11z1mRvVHxnwC0LtGq/Q2N1eUtSVfwHaIQAJGTr1Q1n/xLER QDFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2693G5zlF0KPjpGu8OLoWU8DoCSHGNg7d6xJXxwQddg=; b=hjR57KjKJk+GB8m+ooJdr0AGxgiHl8NJBi3fV8ugqBZbr7ZrJf/Tj5/C+LhBxoBxzk 1bEcvX+lbUCYv5r3t52Qu2pquhlQXWv8CBJAB1GUQwW+pnoJ5Y6vrOYmnMjDNhu78tkt 3FFAeAeQk1OBIEPMgB8BijIy8gczihBKK/qKceCwqpCLKVUIqcQiGWKl3r+BrjafHfDQ 5rDLj5+lD/9+9euRpMhQ0SvfL7dPFKL7if7AO5o/kthBV/W70iMn3+DgPJ9sxzYltBBA 3S25gHkGnpOJ4PbGPLrmImC4Klde+yb1cqcv4wJoZtIU5NgDq4B4Xq1smN6Cp0gZ2mWC TiRA== X-Gm-Message-State: APjAAAXAZW5OhzaVAe4lPXdJxGyZVqGrHs1qC3U83hup9PmY3y/AMCv8 umlx5dsfvPskgdKZ0OAE2fJfo5ve X-Google-Smtp-Source: APXvYqzfkAuH0Rrz3oXOALc7e/ojtGsdLGh33U5Snu+7uDmm4p/pjPRUEVvpGbqbDx2/vmJZi1OHLw== X-Received: by 2002:adf:9cc7:: with SMTP id h7mr10556055wre.259.1556977534537; Sat, 04 May 2019 06:45:34 -0700 (PDT) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 4 May 2019 16:45:26 +0300 Message-Id: <20190504134529.4755-2-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> References: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PATCH PULL 1/4] hw/rdma: Add SRQ support to backend layer X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kamalheib1@gmail.com, yuval.shaia@oracle.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Kamal Heib Add the required functions and definitions to support shared receive queues (SRQs) in the backend layer. Signed-off-by: Kamal Heib Message-Id: <20190403113343.26384-2-kamalheib1@gmail.com> Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 116 +++++++++++++++++++++++++++++++++++- hw/rdma/rdma_backend.h | 12 ++++ hw/rdma/rdma_backend_defs.h | 5 ++ hw/rdma/rdma_rm.c | 2 + hw/rdma/rdma_rm_defs.h | 1 + 5 files changed, 134 insertions(+), 2 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index d1660b6474..04dfd63a57 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -40,6 +40,7 @@ typedef struct BackendCtx { void *up_ctx; struct ibv_sge sge; /* Used to save MAD recv buffer */ RdmaBackendQP *backend_qp; /* To maintain recv buffers */ + RdmaBackendSRQ *backend_srq; } BackendCtx; =20 struct backend_umad { @@ -99,6 +100,7 @@ static int rdma_poll_cq(RdmaDeviceResources *rdma_dev_re= s, struct ibv_cq *ibcq) int i, ne, total_ne =3D 0; BackendCtx *bctx; struct ibv_wc wc[2]; + RdmaProtectedGSList *cqe_ctx_list; =20 qemu_mutex_lock(&rdma_dev_res->lock); do { @@ -116,8 +118,13 @@ static int rdma_poll_cq(RdmaDeviceResources *rdma_dev_= res, struct ibv_cq *ibcq) =20 comp_handler(bctx->up_ctx, &wc[i]); =20 - rdma_protected_gslist_remove_int32(&bctx->backend_qp->cqe_ctx_= list, - wc[i].wr_id); + if (bctx->backend_qp) { + cqe_ctx_list =3D &bctx->backend_qp->cqe_ctx_list; + } else { + cqe_ctx_list =3D &bctx->backend_srq->cqe_ctx_list; + } + + rdma_protected_gslist_remove_int32(cqe_ctx_list, wc[i].wr_id); rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id); g_free(bctx); } @@ -662,6 +669,60 @@ err_free_bctx: g_free(bctx); } =20 +void rdma_backend_post_srq_recv(RdmaBackendDev *backend_dev, + RdmaBackendSRQ *srq, struct ibv_sge *sge, + uint32_t num_sge, void *ctx) +{ + BackendCtx *bctx; + struct ibv_sge new_sge[MAX_SGE]; + uint32_t bctx_id; + int rc; + struct ibv_recv_wr wr =3D {}, *bad_wr; + + bctx =3D g_malloc0(sizeof(*bctx)); + bctx->up_ctx =3D ctx; + bctx->backend_srq =3D srq; + + rc =3D rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx= ); + if (unlikely(rc)) { + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + goto err_free_bctx; + } + + rdma_protected_gslist_append_int32(&srq->cqe_ctx_list, bctx_id); + + rc =3D build_host_sge_array(backend_dev->rdma_dev_res, new_sge, sge, n= um_sge, + &backend_dev->rdma_dev_res->stats.rx_bufs_le= n); + if (rc) { + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); + goto err_dealloc_cqe_ctx; + } + + wr.num_sge =3D num_sge; + wr.sg_list =3D new_sge; + wr.wr_id =3D bctx_id; + rc =3D ibv_post_srq_recv(srq->ibsrq, &wr, &bad_wr); + if (rc) { + rdma_error_report("ibv_post_srq_recv fail, srqn=3D0x%x, rc=3D%d, e= rrno=3D%d", + srq->ibsrq->handle, rc, errno); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + goto err_dealloc_cqe_ctx; + } + + atomic_inc(&backend_dev->rdma_dev_res->stats.missing_cqe); + backend_dev->rdma_dev_res->stats.rx_bufs++; + backend_dev->rdma_dev_res->stats.rx_srq++; + + return; + +err_dealloc_cqe_ctx: + backend_dev->rdma_dev_res->stats.rx_bufs_err++; + rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, bctx_id); + +err_free_bctx: + g_free(bctx); +} + int rdma_backend_create_pd(RdmaBackendDev *backend_dev, RdmaBackendPD *pd) { pd->ibpd =3D ibv_alloc_pd(backend_dev->context); @@ -938,6 +999,55 @@ void rdma_backend_destroy_qp(RdmaBackendQP *qp, RdmaDe= viceResources *dev_res) rdma_protected_gslist_destroy(&qp->cqe_ctx_list); } =20 +int rdma_backend_create_srq(RdmaBackendSRQ *srq, RdmaBackendPD *pd, + uint32_t max_wr, uint32_t max_sge, + uint32_t srq_limit) +{ + struct ibv_srq_init_attr srq_init_attr =3D {}; + + srq_init_attr.attr.max_wr =3D max_wr; + srq_init_attr.attr.max_sge =3D max_sge; + srq_init_attr.attr.srq_limit =3D srq_limit; + + srq->ibsrq =3D ibv_create_srq(pd->ibpd, &srq_init_attr); + if (!srq->ibsrq) { + rdma_error_report("ibv_create_srq failed, errno=3D%d", errno); + return -EIO; + } + + rdma_protected_gslist_init(&srq->cqe_ctx_list); + + return 0; +} + +int rdma_backend_query_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_a= ttr) +{ + if (!srq->ibsrq) { + return -EINVAL; + } + + return ibv_query_srq(srq->ibsrq, srq_attr); +} + +int rdma_backend_modify_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_= attr, + int srq_attr_mask) +{ + if (!srq->ibsrq) { + return -EINVAL; + } + + return ibv_modify_srq(srq->ibsrq, srq_attr, srq_attr_mask); +} + +void rdma_backend_destroy_srq(RdmaBackendSRQ *srq, RdmaDeviceResources *de= v_res) +{ + if (srq->ibsrq) { + ibv_destroy_srq(srq->ibsrq); + } + g_slist_foreach(srq->cqe_ctx_list.list, free_cqe_ctx, dev_res); + rdma_protected_gslist_destroy(&srq->cqe_ctx_list); +} + #define CHK_ATTR(req, dev, member, fmt) ({ \ trace_rdma_check_dev_attr(#member, dev.member, req->member); \ if (req->member > dev.member) { \ @@ -960,6 +1070,7 @@ static int init_device_caps(RdmaBackendDev *backend_de= v, } =20 dev_attr->max_sge =3D MAX_SGE; + dev_attr->max_srq_sge =3D MAX_SGE; =20 CHK_ATTR(dev_attr, bk_dev_attr, max_mr_size, "%" PRId64); CHK_ATTR(dev_attr, bk_dev_attr, max_qp, "%d"); @@ -970,6 +1081,7 @@ static int init_device_caps(RdmaBackendDev *backend_de= v, CHK_ATTR(dev_attr, bk_dev_attr, max_qp_rd_atom, "%d"); CHK_ATTR(dev_attr, bk_dev_attr, max_qp_init_rd_atom, "%d"); CHK_ATTR(dev_attr, bk_dev_attr, max_ah, "%d"); + CHK_ATTR(dev_attr, bk_dev_attr, max_srq, "%d"); =20 return 0; } diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 38056d97c7..cad7956d98 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -114,4 +114,16 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_de= v, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, void *c= tx); =20 +int rdma_backend_create_srq(RdmaBackendSRQ *srq, RdmaBackendPD *pd, + uint32_t max_wr, uint32_t max_sge, + uint32_t srq_limit); +int rdma_backend_query_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_a= ttr); +int rdma_backend_modify_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_= attr, + int srq_attr_mask); +void rdma_backend_destroy_srq(RdmaBackendSRQ *srq, + RdmaDeviceResources *dev_res); +void rdma_backend_post_srq_recv(RdmaBackendDev *backend_dev, + RdmaBackendSRQ *srq, struct ibv_sge *sge, + uint32_t num_sge, void *ctx); + #endif diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 817153dc8c..0b55be3503 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -68,4 +68,9 @@ typedef struct RdmaBackendQP { RdmaProtectedGSList cqe_ctx_list; } RdmaBackendQP; =20 +typedef struct RdmaBackendSRQ { + struct ibv_srq *ibsrq; + RdmaProtectedGSList cqe_ctx_list; +} RdmaBackendSRQ; + #endif diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index bac3b2f4a6..b683506b86 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -37,6 +37,8 @@ void rdma_dump_device_counters(Monitor *mon, RdmaDeviceRe= sources *dev_res) dev_res->stats.tx_err); monitor_printf(mon, "\trx_bufs : %" PRId64 "\n", dev_res->stats.rx_bufs); + monitor_printf(mon, "\trx_srq : %" PRId64 "\n", + dev_res->stats.rx_srq); monitor_printf(mon, "\trx_bufs_len : %" PRId64 "\n", dev_res->stats.rx_bufs_len); monitor_printf(mon, "\trx_bufs_err : %" PRId64 "\n", diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index c200d311de..e774af5280 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -106,6 +106,7 @@ typedef struct RdmaRmStats { uint64_t rx_bufs; uint64_t rx_bufs_len; uint64_t rx_bufs_err; + uint64_t rx_srq; uint64_t completions; uint64_t mad_tx; uint64_t mad_tx_err; --=20 2.17.1