From nobody Thu May 2 13:00:53 2024 Delivered-To: importer@patchew.org Received-SPF: temperror (zoho.com: Error in retrieving data from DNS) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1556977660; cv=none; d=zoho.com; s=zohoarc; b=SYoQnCj2UNuP3eqV4LsaYz1Z7AsKbwACgeZdgaHZP42jT6H5nc1OL3G4MVv6rRMjUqpeDG61vjkIls76H9mJE2tmj+iGQjV0UHAZ/yJLEMUYMhoFpWWWX3SlwHAYVnpxvjaMdIXH1mle/YwAGuJBvrpZa+jRbWDT9GGN4B+aTbc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1556977660; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=2693G5zlF0KPjpGu8OLoWU8DoCSHGNg7d6xJXxwQddg=; b=DtkeLM8QHW+sWmOsUXbwrdssENTH9IWaBV6fM0BmCLJmkINE9jD4bsnZohNIQQSv9N8ftpnLorOQ7G6NiD1v+v4YkYMtfaSh8Y9vPwXDWXoGgDgxNUvDj5z/HQj2UTTOH+Q4QGjIsynEWB01pqwpIbsZya6BDdFTMF9Jt2laYO8= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1556977660919888.6299593876172; Sat, 4 May 2019 06:47:40 -0700 (PDT) Received: from localhost ([127.0.0.1]:56957 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMv0i-00031n-PW for importer@patchew.org; Sat, 04 May 2019 09:47:28 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43984) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMuyw-00021H-01 for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hMuyu-0002Ju-8w for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:37 -0400 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:35323) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hMuyu-0002J1-08 for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:36 -0400 Received: by mail-wr1-x442.google.com with SMTP id h15so5611098wrb.2 for ; Sat, 04 May 2019 06:45:35 -0700 (PDT) Received: from localhost.localdomain ([37.142.144.27]) by smtp.gmail.com with ESMTPSA id t1sm5095610wro.34.2019.05.04.06.45.33 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 04 May 2019 06:45:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2693G5zlF0KPjpGu8OLoWU8DoCSHGNg7d6xJXxwQddg=; b=CLg0hyefSf5BibpnWaV2iEZyDZ3xy+QTcLkrad2YF5Jc+U4lQldIPGqGBW8SyWGtbq WZe2qFmFPaYJpCDkgvwBe7MmhdrKpSmEO5Va/bCgSdJXVTETLlOZqoe/xM7Oh3eqKAnZ CLi68PJVI7ySJs+Y0UCAvyMII5FCqyoBGo9sclGeUWErdq6gjFOF5kQqF6ztVgVUqH6F V1kviBOPj4V1KDOeK8BVzQaip6QTchN8YojBOHM8QPwFEBhss2aciI6VkNBL7LgXn3o2 jzrKQsOlYcQNZ0gstFxs11z1mRvVHxnwC0LtGq/Q2N1eUtSVfwHaIQAJGTr1Q1n/xLER QDFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2693G5zlF0KPjpGu8OLoWU8DoCSHGNg7d6xJXxwQddg=; b=hjR57KjKJk+GB8m+ooJdr0AGxgiHl8NJBi3fV8ugqBZbr7ZrJf/Tj5/C+LhBxoBxzk 1bEcvX+lbUCYv5r3t52Qu2pquhlQXWv8CBJAB1GUQwW+pnoJ5Y6vrOYmnMjDNhu78tkt 3FFAeAeQk1OBIEPMgB8BijIy8gczihBKK/qKceCwqpCLKVUIqcQiGWKl3r+BrjafHfDQ 5rDLj5+lD/9+9euRpMhQ0SvfL7dPFKL7if7AO5o/kthBV/W70iMn3+DgPJ9sxzYltBBA 3S25gHkGnpOJ4PbGPLrmImC4Klde+yb1cqcv4wJoZtIU5NgDq4B4Xq1smN6Cp0gZ2mWC TiRA== X-Gm-Message-State: APjAAAXAZW5OhzaVAe4lPXdJxGyZVqGrHs1qC3U83hup9PmY3y/AMCv8 umlx5dsfvPskgdKZ0OAE2fJfo5ve X-Google-Smtp-Source: APXvYqzfkAuH0Rrz3oXOALc7e/ojtGsdLGh33U5Snu+7uDmm4p/pjPRUEVvpGbqbDx2/vmJZi1OHLw== X-Received: by 2002:adf:9cc7:: with SMTP id h7mr10556055wre.259.1556977534537; Sat, 04 May 2019 06:45:34 -0700 (PDT) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 4 May 2019 16:45:26 +0300 Message-Id: <20190504134529.4755-2-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> References: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PATCH PULL 1/4] hw/rdma: Add SRQ support to backend layer X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kamalheib1@gmail.com, yuval.shaia@oracle.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Kamal Heib Add the required functions and definitions to support shared receive queues (SRQs) in the backend layer. Signed-off-by: Kamal Heib Message-Id: <20190403113343.26384-2-kamalheib1@gmail.com> Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 116 +++++++++++++++++++++++++++++++++++- hw/rdma/rdma_backend.h | 12 ++++ hw/rdma/rdma_backend_defs.h | 5 ++ hw/rdma/rdma_rm.c | 2 + hw/rdma/rdma_rm_defs.h | 1 + 5 files changed, 134 insertions(+), 2 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index d1660b6474..04dfd63a57 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -40,6 +40,7 @@ typedef struct BackendCtx { void *up_ctx; struct ibv_sge sge; /* Used to save MAD recv buffer */ RdmaBackendQP *backend_qp; /* To maintain recv buffers */ + RdmaBackendSRQ *backend_srq; } BackendCtx; =20 struct backend_umad { @@ -99,6 +100,7 @@ static int rdma_poll_cq(RdmaDeviceResources *rdma_dev_re= s, struct ibv_cq *ibcq) int i, ne, total_ne =3D 0; BackendCtx *bctx; struct ibv_wc wc[2]; + RdmaProtectedGSList *cqe_ctx_list; =20 qemu_mutex_lock(&rdma_dev_res->lock); do { @@ -116,8 +118,13 @@ static int rdma_poll_cq(RdmaDeviceResources *rdma_dev_= res, struct ibv_cq *ibcq) =20 comp_handler(bctx->up_ctx, &wc[i]); =20 - rdma_protected_gslist_remove_int32(&bctx->backend_qp->cqe_ctx_= list, - wc[i].wr_id); + if (bctx->backend_qp) { + cqe_ctx_list =3D &bctx->backend_qp->cqe_ctx_list; + } else { + cqe_ctx_list =3D &bctx->backend_srq->cqe_ctx_list; + } + + rdma_protected_gslist_remove_int32(cqe_ctx_list, wc[i].wr_id); rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id); g_free(bctx); } @@ -662,6 +669,60 @@ err_free_bctx: g_free(bctx); } =20 +void rdma_backend_post_srq_recv(RdmaBackendDev *backend_dev, + RdmaBackendSRQ *srq, struct ibv_sge *sge, + uint32_t num_sge, void *ctx) +{ + BackendCtx *bctx; + struct ibv_sge new_sge[MAX_SGE]; + uint32_t bctx_id; + int rc; + struct ibv_recv_wr wr =3D {}, *bad_wr; + + bctx =3D g_malloc0(sizeof(*bctx)); + bctx->up_ctx =3D ctx; + bctx->backend_srq =3D srq; + + rc =3D rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx= ); + if (unlikely(rc)) { + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + goto err_free_bctx; + } + + rdma_protected_gslist_append_int32(&srq->cqe_ctx_list, bctx_id); + + rc =3D build_host_sge_array(backend_dev->rdma_dev_res, new_sge, sge, n= um_sge, + &backend_dev->rdma_dev_res->stats.rx_bufs_le= n); + if (rc) { + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); + goto err_dealloc_cqe_ctx; + } + + wr.num_sge =3D num_sge; + wr.sg_list =3D new_sge; + wr.wr_id =3D bctx_id; + rc =3D ibv_post_srq_recv(srq->ibsrq, &wr, &bad_wr); + if (rc) { + rdma_error_report("ibv_post_srq_recv fail, srqn=3D0x%x, rc=3D%d, e= rrno=3D%d", + srq->ibsrq->handle, rc, errno); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + goto err_dealloc_cqe_ctx; + } + + atomic_inc(&backend_dev->rdma_dev_res->stats.missing_cqe); + backend_dev->rdma_dev_res->stats.rx_bufs++; + backend_dev->rdma_dev_res->stats.rx_srq++; + + return; + +err_dealloc_cqe_ctx: + backend_dev->rdma_dev_res->stats.rx_bufs_err++; + rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, bctx_id); + +err_free_bctx: + g_free(bctx); +} + int rdma_backend_create_pd(RdmaBackendDev *backend_dev, RdmaBackendPD *pd) { pd->ibpd =3D ibv_alloc_pd(backend_dev->context); @@ -938,6 +999,55 @@ void rdma_backend_destroy_qp(RdmaBackendQP *qp, RdmaDe= viceResources *dev_res) rdma_protected_gslist_destroy(&qp->cqe_ctx_list); } =20 +int rdma_backend_create_srq(RdmaBackendSRQ *srq, RdmaBackendPD *pd, + uint32_t max_wr, uint32_t max_sge, + uint32_t srq_limit) +{ + struct ibv_srq_init_attr srq_init_attr =3D {}; + + srq_init_attr.attr.max_wr =3D max_wr; + srq_init_attr.attr.max_sge =3D max_sge; + srq_init_attr.attr.srq_limit =3D srq_limit; + + srq->ibsrq =3D ibv_create_srq(pd->ibpd, &srq_init_attr); + if (!srq->ibsrq) { + rdma_error_report("ibv_create_srq failed, errno=3D%d", errno); + return -EIO; + } + + rdma_protected_gslist_init(&srq->cqe_ctx_list); + + return 0; +} + +int rdma_backend_query_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_a= ttr) +{ + if (!srq->ibsrq) { + return -EINVAL; + } + + return ibv_query_srq(srq->ibsrq, srq_attr); +} + +int rdma_backend_modify_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_= attr, + int srq_attr_mask) +{ + if (!srq->ibsrq) { + return -EINVAL; + } + + return ibv_modify_srq(srq->ibsrq, srq_attr, srq_attr_mask); +} + +void rdma_backend_destroy_srq(RdmaBackendSRQ *srq, RdmaDeviceResources *de= v_res) +{ + if (srq->ibsrq) { + ibv_destroy_srq(srq->ibsrq); + } + g_slist_foreach(srq->cqe_ctx_list.list, free_cqe_ctx, dev_res); + rdma_protected_gslist_destroy(&srq->cqe_ctx_list); +} + #define CHK_ATTR(req, dev, member, fmt) ({ \ trace_rdma_check_dev_attr(#member, dev.member, req->member); \ if (req->member > dev.member) { \ @@ -960,6 +1070,7 @@ static int init_device_caps(RdmaBackendDev *backend_de= v, } =20 dev_attr->max_sge =3D MAX_SGE; + dev_attr->max_srq_sge =3D MAX_SGE; =20 CHK_ATTR(dev_attr, bk_dev_attr, max_mr_size, "%" PRId64); CHK_ATTR(dev_attr, bk_dev_attr, max_qp, "%d"); @@ -970,6 +1081,7 @@ static int init_device_caps(RdmaBackendDev *backend_de= v, CHK_ATTR(dev_attr, bk_dev_attr, max_qp_rd_atom, "%d"); CHK_ATTR(dev_attr, bk_dev_attr, max_qp_init_rd_atom, "%d"); CHK_ATTR(dev_attr, bk_dev_attr, max_ah, "%d"); + CHK_ATTR(dev_attr, bk_dev_attr, max_srq, "%d"); =20 return 0; } diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 38056d97c7..cad7956d98 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -114,4 +114,16 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_de= v, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, void *c= tx); =20 +int rdma_backend_create_srq(RdmaBackendSRQ *srq, RdmaBackendPD *pd, + uint32_t max_wr, uint32_t max_sge, + uint32_t srq_limit); +int rdma_backend_query_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_a= ttr); +int rdma_backend_modify_srq(RdmaBackendSRQ *srq, struct ibv_srq_attr *srq_= attr, + int srq_attr_mask); +void rdma_backend_destroy_srq(RdmaBackendSRQ *srq, + RdmaDeviceResources *dev_res); +void rdma_backend_post_srq_recv(RdmaBackendDev *backend_dev, + RdmaBackendSRQ *srq, struct ibv_sge *sge, + uint32_t num_sge, void *ctx); + #endif diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 817153dc8c..0b55be3503 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -68,4 +68,9 @@ typedef struct RdmaBackendQP { RdmaProtectedGSList cqe_ctx_list; } RdmaBackendQP; =20 +typedef struct RdmaBackendSRQ { + struct ibv_srq *ibsrq; + RdmaProtectedGSList cqe_ctx_list; +} RdmaBackendSRQ; + #endif diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index bac3b2f4a6..b683506b86 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -37,6 +37,8 @@ void rdma_dump_device_counters(Monitor *mon, RdmaDeviceRe= sources *dev_res) dev_res->stats.tx_err); monitor_printf(mon, "\trx_bufs : %" PRId64 "\n", dev_res->stats.rx_bufs); + monitor_printf(mon, "\trx_srq : %" PRId64 "\n", + dev_res->stats.rx_srq); monitor_printf(mon, "\trx_bufs_len : %" PRId64 "\n", dev_res->stats.rx_bufs_len); monitor_printf(mon, "\trx_bufs_err : %" PRId64 "\n", diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index c200d311de..e774af5280 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -106,6 +106,7 @@ typedef struct RdmaRmStats { uint64_t rx_bufs; uint64_t rx_bufs_len; uint64_t rx_bufs_err; + uint64_t rx_srq; uint64_t completions; uint64_t mad_tx; uint64_t mad_tx_err; --=20 2.17.1 From nobody Thu May 2 13:00:53 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1556977791; cv=none; d=zoho.com; s=zohoarc; b=AJqnVEjK1L6LVfjRsdyzv2Mde/6/jDunppkvDGz5TRYVk6P9lcWOGq9HvBTgXtFYQc/sOwkXhMovgyb0uWfsj6TTGmbgrsAxVK7YL+lq4vRkojeVxZyhhFUZMBTfg+mASyrFI+3CscgWiKSvX2EgNcHIIE/3+k09R1WDCmwftGE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1556977791; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=ZzPqjXcVVs2xit2QjD3jgD4GNNyHuMQK4jtT3UL3/9E=; b=U6pf+AtnoY8QdEr08vuQAgu7i9/HsyeERedIX/7nO/J9G+3mETjTXUDxWNe/R9fdE/fjT34yeE0r9CHCG46ml+wHAmJF1240nUdceWKq6AlAo4207+m8xK/QVRJ8H5LCpS8z5mE2X1NsFsYgQSUr022rXWSVhykFS1IbA0P3Gho= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (209.51.188.17 [209.51.188.17]) by mx.zohomail.com with SMTPS id 1556977791681615.0201736248813; Sat, 4 May 2019 06:49:51 -0700 (PDT) Received: from localhost ([127.0.0.1]:56976 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMv2v-0004X1-BG for importer@patchew.org; Sat, 04 May 2019 09:49:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43990) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMuyw-00021I-Ny for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hMuyv-0002LT-GO for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:38 -0400 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]:46369) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hMuyv-0002KT-9U for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:37 -0400 Received: by mail-wr1-x443.google.com with SMTP id r7so11264015wrr.13 for ; Sat, 04 May 2019 06:45:37 -0700 (PDT) Received: from localhost.localdomain ([37.142.144.27]) by smtp.gmail.com with ESMTPSA id t1sm5095610wro.34.2019.05.04.06.45.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 04 May 2019 06:45:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZzPqjXcVVs2xit2QjD3jgD4GNNyHuMQK4jtT3UL3/9E=; b=iaJ+iwBu4GLxmqzP7VrCAO5rwzW45wK/sqfbV6CmLIZOWTH6k/KcIk8Xli32JoxEew 2PuACDF4vr1I7prtwAwA9pmRbpF+SXrtvmyxOFK9sKvDpYvcm0KEQqjv5L3xKiUlzyWo 8F4brWk7J4yMS8RCgR7UKZ7YeIAuO3JdktyEl7S2KfPMTK/f+4BQ0vvC6upaGTS//0iS IzmaM5DfIVodDAf00+yWBWeGhM5MXc//WV+lcNDJ/pqobzM9qNXdbwl1cdF8m8PPvTc0 grFGe08WX8NteeKa96l2CPNgyvscv7klTq0Oqd4CiQVKAyG20jSecoIINKEPN8ZLPR+g kI6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZzPqjXcVVs2xit2QjD3jgD4GNNyHuMQK4jtT3UL3/9E=; b=nwO6m2GbmQbW7jz6aD0IueNQsyRW2h0en4PI4j3GWROKdFt/lmfULbW39ccw4eWSw1 wR4KVTO2q3qx1P79FI3eJMYZPezotrLA/IvqtkGkEHGXUIgjQy6yf0A+LB2qDTUvB7tO lQOe0J3i5KfKZuGml8iEB2f+E/cQtH1bnnv+sNTMHaTMeIhURMl2dgxw7JSQSFW4+SGf CibuF+Pp+lbNKpYZkSZePybD0eNFOkd0shVUXJrTY6xg4CrMc0Iy35/UWVbwE5idDXUM oR+I4j/cn2o/Mbw6STtE/HB73wfpxmg437cj/28525KOba+S23XffQw/iViv4eeA87+d PBPw== X-Gm-Message-State: APjAAAVwy+3WtNmf8rAMc7e6m3Asx3606Zbeg+l8gfMGSMlBMQjsRDe+ LPM4S+UP6TAkhHksIIWGyf4BRQ+S X-Google-Smtp-Source: APXvYqxgJf9rcG8LhrGs//UWa4I9mEtDZWSWi/vD/UlwIyRSHsro2Hsq34+dtv7/Nbh/+OTaTatbPw== X-Received: by 2002:a5d:67cb:: with SMTP id n11mr11032873wrw.3.1556977535854; Sat, 04 May 2019 06:45:35 -0700 (PDT) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 4 May 2019 16:45:27 +0300 Message-Id: <20190504134529.4755-3-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> References: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::443 Subject: [Qemu-devel] [PATCH PULL 2/4] hw/rdma: Add support for managing SRQ resource X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kamalheib1@gmail.com, yuval.shaia@oracle.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Kamal Heib Adding the required functions and definitions for support managing the shared receive queues (SRQs). Signed-off-by: Kamal Heib Message-Id: <20190403113343.26384-3-kamalheib1@gmail.com> Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 93 ++++++++++++++++++++++++++++++++++++++++++ hw/rdma/rdma_rm.h | 10 +++++ hw/rdma/rdma_rm_defs.h | 8 ++++ 3 files changed, 111 insertions(+) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index b683506b86..c4fb140dcd 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -544,6 +544,96 @@ void rdma_rm_dealloc_qp(RdmaDeviceResources *dev_res, = uint32_t qp_handle) rdma_res_tbl_dealloc(&dev_res->qp_tbl, qp->qpn); } =20 +RdmaRmSRQ *rdma_rm_get_srq(RdmaDeviceResources *dev_res, uint32_t srq_hand= le) +{ + return rdma_res_tbl_get(&dev_res->srq_tbl, srq_handle); +} + +int rdma_rm_alloc_srq(RdmaDeviceResources *dev_res, uint32_t pd_handle, + uint32_t max_wr, uint32_t max_sge, uint32_t srq_limi= t, + uint32_t *srq_handle, void *opaque) +{ + RdmaRmSRQ *srq; + RdmaRmPD *pd; + int rc; + + pd =3D rdma_rm_get_pd(dev_res, pd_handle); + if (!pd) { + return -EINVAL; + } + + srq =3D rdma_res_tbl_alloc(&dev_res->srq_tbl, srq_handle); + if (!srq) { + return -ENOMEM; + } + + rc =3D rdma_backend_create_srq(&srq->backend_srq, &pd->backend_pd, + max_wr, max_sge, srq_limit); + if (rc) { + rc =3D -EIO; + goto out_dealloc_srq; + } + + srq->opaque =3D opaque; + + return 0; + +out_dealloc_srq: + rdma_res_tbl_dealloc(&dev_res->srq_tbl, *srq_handle); + + return rc; +} + +int rdma_rm_query_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr) +{ + RdmaRmSRQ *srq; + + srq =3D rdma_rm_get_srq(dev_res, srq_handle); + if (!srq) { + return -EINVAL; + } + + return rdma_backend_query_srq(&srq->backend_srq, srq_attr); +} + +int rdma_rm_modify_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr, int srq_attr_mask) +{ + RdmaRmSRQ *srq; + + srq =3D rdma_rm_get_srq(dev_res, srq_handle); + if (!srq) { + return -EINVAL; + } + + if ((srq_attr_mask & IBV_SRQ_LIMIT) && + (srq_attr->srq_limit =3D=3D 0)) { + return -EINVAL; + } + + if ((srq_attr_mask & IBV_SRQ_MAX_WR) && + (srq_attr->max_wr =3D=3D 0)) { + return -EINVAL; + } + + return rdma_backend_modify_srq(&srq->backend_srq, srq_attr, + srq_attr_mask); +} + +void rdma_rm_dealloc_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle) +{ + RdmaRmSRQ *srq; + + srq =3D rdma_rm_get_srq(dev_res, srq_handle); + if (!srq) { + return; + } + + rdma_backend_destroy_srq(&srq->backend_srq, dev_res); + rdma_res_tbl_dealloc(&dev_res->srq_tbl, srq_handle); +} + void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_i= d) { void **cqe_ctx; @@ -673,6 +763,8 @@ int rdma_rm_init(RdmaDeviceResources *dev_res, struct i= bv_device_attr *dev_attr) res_tbl_init("CQE_CTX", &dev_res->cqe_ctx_tbl, dev_attr->max_qp * dev_attr->max_qp_wr, sizeof(void *)); res_tbl_init("UC", &dev_res->uc_tbl, MAX_UCS, sizeof(RdmaRmUC)); + res_tbl_init("SRQ", &dev_res->srq_tbl, dev_attr->max_srq, + sizeof(RdmaRmSRQ)); =20 init_ports(dev_res); =20 @@ -691,6 +783,7 @@ void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBac= kendDev *backend_dev, =20 fini_ports(dev_res, backend_dev, ifname); =20 + res_tbl_free(&dev_res->srq_tbl); res_tbl_free(&dev_res->uc_tbl); res_tbl_free(&dev_res->cqe_ctx_tbl); res_tbl_free(&dev_res->qp_tbl); diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index 4f03f9b8c5..e88ab95e26 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -65,6 +65,16 @@ int rdma_rm_query_qp(RdmaDeviceResources *dev_res, RdmaB= ackendDev *backend_dev, int attr_mask, struct ibv_qp_init_attr *init_attr); void rdma_rm_dealloc_qp(RdmaDeviceResources *dev_res, uint32_t qp_handle); =20 +RdmaRmSRQ *rdma_rm_get_srq(RdmaDeviceResources *dev_res, uint32_t srq_hand= le); +int rdma_rm_alloc_srq(RdmaDeviceResources *dev_res, uint32_t pd_handle, + uint32_t max_wr, uint32_t max_sge, uint32_t srq_limi= t, + uint32_t *srq_handle, void *opaque); +int rdma_rm_query_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr); +int rdma_rm_modify_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle, + struct ibv_srq_attr *srq_attr, int srq_attr_mask); +void rdma_rm_dealloc_srq(RdmaDeviceResources *dev_res, uint32_t srq_handle= ); + int rdma_rm_alloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t *cqe_ctx_= id, void *ctx); void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_i= d); diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index e774af5280..7bdd9f291f 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -33,6 +33,7 @@ #define MAX_QP_RD_ATOM 16 #define MAX_QP_INIT_RD_ATOM 16 #define MAX_AH 64 +#define MAX_SRQ 512 =20 #define MAX_RM_TBL_NAME 16 #define MAX_CONSEQ_EMPTY_POLL_CQ 4096 /* considered as error above this= */ @@ -89,6 +90,12 @@ typedef struct RdmaRmQP { enum ibv_qp_state qp_state; } RdmaRmQP; =20 +typedef struct RdmaRmSRQ { + RdmaBackendSRQ backend_srq; + uint32_t recv_cq_handle; + void *opaque; +} RdmaRmSRQ; + typedef struct RdmaRmGid { union ibv_gid gid; int backend_gid_index; @@ -129,6 +136,7 @@ struct RdmaDeviceResources { RdmaRmResTbl qp_tbl; RdmaRmResTbl cq_tbl; RdmaRmResTbl cqe_ctx_tbl; + RdmaRmResTbl srq_tbl; GHashTable *qp_hash; /* Keeps mapping between real and emulated */ QemuMutex lock; RdmaRmStats stats; --=20 2.17.1 From nobody Thu May 2 13:00:53 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1556977664; cv=none; d=zoho.com; s=zohoarc; b=WFT9MHlo4PbPBR1CywDcrFPIiK3/VZgAlPrYzGEMcehqHDfHO8VS00135Egf0st5AnzjZu5J+LzkS+UDYTh9GgiwPxnpEmDziA1kxDxPkoxP8gcyKTEq3sASn5Qn9eNwnfB9fs4fPlG6hTBtVfe47CD9y8BDgx+qlAKTN881Ggo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1556977664; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=/QV3bKWmdEB50/rH15uuq3cqW0QRXb1M7/j8cW2cPSY=; b=WCfqdaA3FsS5nMJ5QzNoklbPVTm/5pgBxg8dLaOXU2XHAltUBXPhIsk376+iWmkMa4sB+yrAdH8nOytaoJKm0pDqblRRdqAJK0d+aLyEHhSe6qSAK5Ss990qEwkw7bwiUhhC5n2lnk3uYhdO8XlWdrpkjJuazU32YSD0DT8/iwU= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1556977664794768.3770384404236; Sat, 4 May 2019 06:47:44 -0700 (PDT) Received: from localhost ([127.0.0.1]:56959 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMv0p-00033k-VM for importer@patchew.org; Sat, 04 May 2019 09:47:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44007) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMuyy-000226-M3 for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hMuyx-0002M7-2f for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:40 -0400 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:35887) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hMuyw-0002Lk-QF for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:39 -0400 Received: by mail-wr1-x441.google.com with SMTP id o4so11331130wra.3 for ; Sat, 04 May 2019 06:45:38 -0700 (PDT) Received: from localhost.localdomain ([37.142.144.27]) by smtp.gmail.com with ESMTPSA id t1sm5095610wro.34.2019.05.04.06.45.35 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 04 May 2019 06:45:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/QV3bKWmdEB50/rH15uuq3cqW0QRXb1M7/j8cW2cPSY=; b=G6Oss63nBX+LYBKv3FPrWJt6wEduifcSZ4eMFELCnY8lqGt/+/YPn1dalm9HhZRXyV Rq19N30kF3YzchXId/43q1EFCfcM+UCj19FqVBI4DZRplMVsSXh97nb7IL6woJ8ZD4Bt SdpWKpNJtZ2iOSj5hxH2kdJWO/qvmCUyphajf45ngpBvOoBjOWI7lGDG4W+w7A8B/q65 q0OojlZoktBhePcuRM2tgZlxe8gC4ry6uEeqfwBfi4Tbr62//NcE9jU/VDXn1bnJSyeJ uq+6FgwXnokgZUqIeuAvFrSG/WBfEJ6rNV9yb8Ca79KLK00iDU0tKIrUXT41GnnxPGp1 vxWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/QV3bKWmdEB50/rH15uuq3cqW0QRXb1M7/j8cW2cPSY=; b=D2UBaYiGW0SHv0gwVq1v7EQ+T/5vXhep4v9BNWtJDbtc83ZFvd+/TUE04NQZIZxGgG Sk4bHzSpWLc/o/UjW13ZQx8lYuwrsUdMqmxPbp+4wKEaAuWgEa02vKp+2ikr/qqdngoT hZFre4OuQeyTLVIobNVtmDnap0mMHCB7liFIC1gnEh1p5VC+7roU7HBeeMUkkEIlXGEY BDZwYxmtG/YaMmxIdS6H3G2RgOd889mGUhB7WKqMhPmZEgafB+JDeJn2xb59LvXviAGp 6N8hC3+LxZX7ahPmDOmcvrK93V2kab+r1Ws/ls9U/blS6IOE2m3ApkI8q8Bh2hITWO1d 7VgQ== X-Gm-Message-State: APjAAAUYbpV0bRv1+Q0FU2PmWDNEXpryv97h559d00sQ0NNT64JKIMnE 2N0ddwWtpphl7ARsoP2okXsN/jT4 X-Google-Smtp-Source: APXvYqzOiA1xzgfpOuAR6MM5/N3rDlNcz5EC7drdPZa6rtgID9p7XSCfZQjRPnbxYy6yk4rqfNJDSw== X-Received: by 2002:a05:6000:ca:: with SMTP id q10mr3550948wrx.148.1556977537461; Sat, 04 May 2019 06:45:37 -0700 (PDT) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 4 May 2019 16:45:28 +0300 Message-Id: <20190504134529.4755-4-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> References: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 3/4] hw/rdma: Modify create/destroy QP to support SRQ X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kamalheib1@gmail.com, yuval.shaia@oracle.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Kamal Heib Modify create/destroy QP to support shared receive queue and rearrange the destroy_qp() code to avoid touching the QP after calling rdma_rm_dealloc_qp(). Signed-off-by: Kamal Heib Message-Id: <20190403113343.26384-4-kamalheib1@gmail.com> Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 9 ++++-- hw/rdma/rdma_backend.h | 6 ++-- hw/rdma/rdma_rm.c | 22 +++++++++++++-- hw/rdma/rdma_rm.h | 3 +- hw/rdma/rdma_rm_defs.h | 1 + hw/rdma/vmw/pvrdma_cmd.c | 59 ++++++++++++++++++++++++---------------- 6 files changed, 67 insertions(+), 33 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 04dfd63a57..cf34874e9d 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -794,9 +794,9 @@ void rdma_backend_destroy_cq(RdmaBackendCQ *cq) =20 int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_type, RdmaBackendPD *pd, RdmaBackendCQ *scq, - RdmaBackendCQ *rcq, uint32_t max_send_wr, - uint32_t max_recv_wr, uint32_t max_send_sge, - uint32_t max_recv_sge) + RdmaBackendCQ *rcq, RdmaBackendSRQ *srq, + uint32_t max_send_wr, uint32_t max_recv_wr, + uint32_t max_send_sge, uint32_t max_recv_sge) { struct ibv_qp_init_attr attr =3D {}; =20 @@ -824,6 +824,9 @@ int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t q= p_type, attr.cap.max_recv_wr =3D max_recv_wr; attr.cap.max_send_sge =3D max_send_sge; attr.cap.max_recv_sge =3D max_recv_sge; + if (srq) { + attr.srq =3D srq->ibsrq; + } =20 qp->ibqp =3D ibv_create_qp(pd->ibpd, &attr); if (!qp->ibqp) { diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index cad7956d98..7c1a19a2b5 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -89,9 +89,9 @@ void rdma_backend_poll_cq(RdmaDeviceResources *rdma_dev_r= es, RdmaBackendCQ *cq); =20 int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_type, RdmaBackendPD *pd, RdmaBackendCQ *scq, - RdmaBackendCQ *rcq, uint32_t max_send_wr, - uint32_t max_recv_wr, uint32_t max_send_sge, - uint32_t max_recv_sge); + RdmaBackendCQ *rcq, RdmaBackendSRQ *srq, + uint32_t max_send_wr, uint32_t max_recv_wr, + uint32_t max_send_sge, uint32_t max_recv_sge); int rdma_backend_qp_state_init(RdmaBackendDev *backend_dev, RdmaBackendQP = *qp, uint8_t qp_type, uint32_t qkey); int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *= qp, diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index c4fb140dcd..1927f85472 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -386,12 +386,14 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, ui= nt32_t pd_handle, uint8_t qp_type, uint32_t max_send_wr, uint32_t max_send_sge, uint32_t send_cq_handle, uint32_t max_recv_wr, uint32_t max_recv_sge, - uint32_t recv_cq_handle, void *opaque, uint32_t *qpn) + uint32_t recv_cq_handle, void *opaque, uint32_t *qpn, + uint8_t is_srq, uint32_t srq_handle) { int rc; RdmaRmQP *qp; RdmaRmCQ *scq, *rcq; RdmaRmPD *pd; + RdmaRmSRQ *srq =3D NULL; uint32_t rm_qpn; =20 pd =3D rdma_rm_get_pd(dev_res, pd_handle); @@ -408,6 +410,16 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uin= t32_t pd_handle, return -EINVAL; } =20 + if (is_srq) { + srq =3D rdma_rm_get_srq(dev_res, srq_handle); + if (!srq) { + rdma_error_report("Invalid srqn %d", srq_handle); + return -EINVAL; + } + + srq->recv_cq_handle =3D recv_cq_handle; + } + if (qp_type =3D=3D IBV_QPT_GSI) { scq->notify =3D CNT_SET; rcq->notify =3D CNT_SET; @@ -424,10 +436,14 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, ui= nt32_t pd_handle, qp->send_cq_handle =3D send_cq_handle; qp->recv_cq_handle =3D recv_cq_handle; qp->opaque =3D opaque; + qp->is_srq =3D is_srq; =20 rc =3D rdma_backend_create_qp(&qp->backend_qp, qp_type, &pd->backend_p= d, - &scq->backend_cq, &rcq->backend_cq, max_se= nd_wr, - max_recv_wr, max_send_sge, max_recv_sge); + &scq->backend_cq, &rcq->backend_cq, + is_srq ? &srq->backend_srq : NULL, + max_send_wr, max_recv_wr, max_send_sge, + max_recv_sge); + if (rc) { rc =3D -EIO; goto out_dealloc_qp; diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index e88ab95e26..e8639909cd 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -53,7 +53,8 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uint32= _t pd_handle, uint8_t qp_type, uint32_t max_send_wr, uint32_t max_send_sge, uint32_t send_cq_handle, uint32_t max_recv_wr, uint32_t max_recv_sge, - uint32_t recv_cq_handle, void *opaque, uint32_t *qpn); + uint32_t recv_cq_handle, void *opaque, uint32_t *qpn, + uint8_t is_srq, uint32_t srq_handle); RdmaRmQP *rdma_rm_get_qp(RdmaDeviceResources *dev_res, uint32_t qpn); int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backen= d_dev, uint32_t qp_handle, uint32_t attr_mask, uint8_t sgid= _idx, diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 7bdd9f291f..534f2f74d3 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -88,6 +88,7 @@ typedef struct RdmaRmQP { uint32_t send_cq_handle; uint32_t recv_cq_handle; enum ibv_qp_state qp_state; + uint8_t is_srq; } RdmaRmQP; =20 typedef struct RdmaRmSRQ { diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 4afcd2037d..b931bb6dc9 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -357,7 +357,7 @@ static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd_= req *req, static int create_qp_rings(PCIDevice *pci_dev, uint64_t pdir_dma, PvrdmaRing **rings, uint32_t scqe, uint32_t sma= x_sge, uint32_t spages, uint32_t rcqe, uint32_t rmax_s= ge, - uint32_t rpages) + uint32_t rpages, uint8_t is_srq) { uint64_t *dir =3D NULL, *tbl =3D NULL; PvrdmaRing *sr, *rr; @@ -365,9 +365,14 @@ static int create_qp_rings(PCIDevice *pci_dev, uint64_= t pdir_dma, char ring_name[MAX_RING_NAME_SZ]; uint32_t wqe_sz; =20 - if (!spages || spages > PVRDMA_MAX_FAST_REG_PAGES - || !rpages || rpages > PVRDMA_MAX_FAST_REG_PAGES) { - rdma_error_report("Got invalid page count for QP ring: %d, %d", sp= ages, + if (!spages || spages > PVRDMA_MAX_FAST_REG_PAGES) { + rdma_error_report("Got invalid send page count for QP ring: %d", + spages); + return rc; + } + + if (!is_srq && (!rpages || rpages > PVRDMA_MAX_FAST_REG_PAGES)) { + rdma_error_report("Got invalid recv page count for QP ring: %d", rpages); return rc; } @@ -384,8 +389,12 @@ static int create_qp_rings(PCIDevice *pci_dev, uint64_= t pdir_dma, goto out; } =20 - sr =3D g_malloc(2 * sizeof(*rr)); - rr =3D &sr[1]; + if (!is_srq) { + sr =3D g_malloc(2 * sizeof(*rr)); + rr =3D &sr[1]; + } else { + sr =3D g_malloc(sizeof(*sr)); + } =20 *rings =3D sr; =20 @@ -407,15 +416,18 @@ static int create_qp_rings(PCIDevice *pci_dev, uint64= _t pdir_dma, goto out_unmap_ring_state; } =20 - /* Create recv ring */ - rr->ring_state =3D &sr->ring_state[1]; - wqe_sz =3D pow2ceil(sizeof(struct pvrdma_rq_wqe_hdr) + - sizeof(struct pvrdma_sge) * rmax_sge - 1); - sprintf(ring_name, "qp_rring_%" PRIx64, pdir_dma); - rc =3D pvrdma_ring_init(rr, ring_name, pci_dev, rr->ring_state, - rcqe, wqe_sz, (dma_addr_t *)&tbl[1 + spages], rp= ages); - if (rc) { - goto out_free_sr; + if (!is_srq) { + /* Create recv ring */ + rr->ring_state =3D &sr->ring_state[1]; + wqe_sz =3D pow2ceil(sizeof(struct pvrdma_rq_wqe_hdr) + + sizeof(struct pvrdma_sge) * rmax_sge - 1); + sprintf(ring_name, "qp_rring_%" PRIx64, pdir_dma); + rc =3D pvrdma_ring_init(rr, ring_name, pci_dev, rr->ring_state, + rcqe, wqe_sz, (dma_addr_t *)&tbl[1 + spages], + rpages); + if (rc) { + goto out_free_sr; + } } =20 goto out; @@ -436,10 +448,12 @@ out: return rc; } =20 -static void destroy_qp_rings(PvrdmaRing *ring) +static void destroy_qp_rings(PvrdmaRing *ring, uint8_t is_srq) { pvrdma_ring_free(&ring[0]); - pvrdma_ring_free(&ring[1]); + if (!is_srq) { + pvrdma_ring_free(&ring[1]); + } =20 rdma_pci_dma_unmap(ring->dev, ring->ring_state, TARGET_PAGE_SIZE); g_free(ring); @@ -458,7 +472,7 @@ static int create_qp(PVRDMADev *dev, union pvrdma_cmd_r= eq *req, rc =3D create_qp_rings(PCI_DEVICE(dev), cmd->pdir_dma, &rings, cmd->max_send_wr, cmd->max_send_sge, cmd->send_ch= unks, cmd->max_recv_wr, cmd->max_recv_sge, - cmd->total_chunks - cmd->send_chunks - 1); + cmd->total_chunks - cmd->send_chunks - 1, cmd->is= _srq); if (rc) { return rc; } @@ -467,9 +481,9 @@ static int create_qp(PVRDMADev *dev, union pvrdma_cmd_r= eq *req, cmd->max_send_wr, cmd->max_send_sge, cmd->send_cq_handle, cmd->max_recv_wr, cmd->max_recv_sge, cmd->recv_cq_handle, rings, - &resp->qpn); + &resp->qpn, cmd->is_srq, cmd->srq_handle); if (rc) { - destroy_qp_rings(rings); + destroy_qp_rings(rings, cmd->is_srq); return rc; } =20 @@ -531,10 +545,9 @@ static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd= _req *req, return -EINVAL; } =20 - rdma_rm_dealloc_qp(&dev->rdma_dev_res, cmd->qp_handle); - ring =3D (PvrdmaRing *)qp->opaque; - destroy_qp_rings(ring); + destroy_qp_rings(ring, qp->is_srq); + rdma_rm_dealloc_qp(&dev->rdma_dev_res, cmd->qp_handle); =20 return 0; } --=20 2.17.1 From nobody Thu May 2 13:00:53 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1556977831; cv=none; d=zoho.com; s=zohoarc; b=n+s+qlPW8juHPP+CDvGKturZ4/0Wt9GwrLyrOzjQWdeR8C+StVI1Uv5uixXuNpGtEK76pQ44jvwufHEorGsMQlLzRwAV26y+27D1rG3moB/7jfcnmFFsbHdmeYok3bL6Bi1DKAfo7rkpLClhHE4rI3Gr54jWYfACjehNVhpOHnI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1556977831; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=yBDZEMFabFIVF/5cxjG+KE5nHuYNnufiwwMl9tL4nuc=; b=h6he1nge37VKyEoRt+HOAXRkrXx183Qo0U3630p5f6Lx5ceFUc0U+A3ezkAvqenxaxtH7V07GhQIKVXN+P2dtwJdXYB1CRpUJKXfK1rll8/PvAyhPRpuzewo6wfWBoYy1oeoXBdoOWQCJXPQ78Cjx5zQ1usJkCNO/cD08FwlDhw= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 155697783150279.15509470691643; Sat, 4 May 2019 06:50:31 -0700 (PDT) Received: from localhost ([127.0.0.1]:56979 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMv3b-0004wP-El for importer@patchew.org; Sat, 04 May 2019 09:50:27 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44018) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hMuz0-00023S-4L for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hMuyy-0002Mf-Jz for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:42 -0400 Received: from mail-wr1-x42f.google.com ([2a00:1450:4864:20::42f]:35896) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hMuyy-0002ML-BY for qemu-devel@nongnu.org; Sat, 04 May 2019 09:45:40 -0400 Received: by mail-wr1-x42f.google.com with SMTP id o4so11331166wra.3 for ; Sat, 04 May 2019 06:45:40 -0700 (PDT) Received: from localhost.localdomain ([37.142.144.27]) by smtp.gmail.com with ESMTPSA id t1sm5095610wro.34.2019.05.04.06.45.37 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 04 May 2019 06:45:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yBDZEMFabFIVF/5cxjG+KE5nHuYNnufiwwMl9tL4nuc=; b=ULQ/FiHmLGKDfuOw/93eaxHFp2yoD9qTT2XphfaU1t/zNY+sJ98TzDTKUVNJoLSp/p dvFiPZPoM+2cnKbq6mr4RcgFM2us024gGnWwNpu2L8sLZ0xku/4vQeXIqqBhAOfXBzXZ vR+ffMCUQ41S85N7fwxeIa6CvaceXpBPHNgFO/ERzis4hWYw1xDFxLbEO7Tl1qPgdnOh EqxGKimslFuaiDKX0GimRpLdPHvly57sbtiAayOhfe8vXHj43iOZwJevb8tPR0rPkqT1 gIfbf1hjYizo6d4fk1g5BBthuzYDAFNtkgSRnW04zk+cpWrdUPQpPVKWhu0ZnosMPtcp 3bkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yBDZEMFabFIVF/5cxjG+KE5nHuYNnufiwwMl9tL4nuc=; b=Mw7ZS8VgMy7UKm3umhvlRUKHiYFPAjjCYo/x3RF/W8bty7kNQ/5WrYEQKu9MgW33Lt 2G24i7DBgGclQe2mB6npR07if0o9JrWg9g5N5dxAhsC1TufKXaWF9gHO6W3Z8Bib9LYu 48CWo4nS9WSZvu8IvyCypfqFNgsR2/w6ySnXpBwLX1FxIY13Wf4LN3FS0GL/PoIIFpp3 o85szLuZ1oKIdiGPI1XJaMVzafLlGuGDo/8aUg+JRYMGzTH2Pi2zPWKb5RoS3XWliRAE AbY0R8QAndokqDHEOlVYr6G/RHztmtQxMNzWvMb5zoneoNLYglKl6uICjU8zZNopegFw joAw== X-Gm-Message-State: APjAAAVjUS6WMJ2mHVqECrkwZEXI001SVW9xXYaoKL/7HlaTBm1272kp GYjX7RcUTcWpOe/ChGmwa5+CIPe3 X-Google-Smtp-Source: APXvYqybTnw6c81EfqctPbqETtLwS98kY0Nx7D9RJVvdL+UVClMiYkkHAmTCUA3bWGHWzrakm4IPQQ== X-Received: by 2002:adf:f2c9:: with SMTP id d9mr11419829wrp.36.1556977538936; Sat, 04 May 2019 06:45:38 -0700 (PDT) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 4 May 2019 16:45:29 +0300 Message-Id: <20190504134529.4755-5-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> References: <20190504134529.4755-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::42f Subject: [Qemu-devel] [PATCH PULL 4/4] hw/pvrdma: Add support for SRQ X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kamalheib1@gmail.com, yuval.shaia@oracle.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Kamal Heib Implement the pvrdma device commands for supporting SRQ Signed-off-by: Kamal Heib Message-Id: <20190403113343.26384-5-kamalheib1@gmail.com> Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_cmd.c | 147 ++++++++++++++++++++++++++++++++++++ hw/rdma/vmw/pvrdma_main.c | 16 ++++ hw/rdma/vmw/pvrdma_qp_ops.c | 46 ++++++++++- hw/rdma/vmw/pvrdma_qp_ops.h | 1 + 4 files changed, 209 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index b931bb6dc9..8d70c0d23d 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -609,6 +609,149 @@ static int destroy_uc(PVRDMADev *dev, union pvrdma_cm= d_req *req, return 0; } =20 +static int create_srq_ring(PCIDevice *pci_dev, PvrdmaRing **ring, + uint64_t pdir_dma, uint32_t max_wr, + uint32_t max_sge, uint32_t nchunks) +{ + uint64_t *dir =3D NULL, *tbl =3D NULL; + PvrdmaRing *r; + int rc =3D -EINVAL; + char ring_name[MAX_RING_NAME_SZ]; + uint32_t wqe_sz; + + if (!nchunks || nchunks > PVRDMA_MAX_FAST_REG_PAGES) { + rdma_error_report("Got invalid page count for SRQ ring: %d", + nchunks); + return rc; + } + + dir =3D rdma_pci_dma_map(pci_dev, pdir_dma, TARGET_PAGE_SIZE); + if (!dir) { + rdma_error_report("Failed to map to SRQ page directory"); + goto out; + } + + tbl =3D rdma_pci_dma_map(pci_dev, dir[0], TARGET_PAGE_SIZE); + if (!tbl) { + rdma_error_report("Failed to map to SRQ page table"); + goto out; + } + + r =3D g_malloc(sizeof(*r)); + *ring =3D r; + + r->ring_state =3D (struct pvrdma_ring *) + rdma_pci_dma_map(pci_dev, tbl[0], TARGET_PAGE_SIZE); + if (!r->ring_state) { + rdma_error_report("Failed to map tp SRQ ring state"); + goto out_free_ring_mem; + } + + wqe_sz =3D pow2ceil(sizeof(struct pvrdma_rq_wqe_hdr) + + sizeof(struct pvrdma_sge) * max_sge - 1); + sprintf(ring_name, "srq_ring_%" PRIx64, pdir_dma); + rc =3D pvrdma_ring_init(r, ring_name, pci_dev, &r->ring_state[1], max_= wr, + wqe_sz, (dma_addr_t *)&tbl[1], nchunks - 1); + if (rc) { + goto out_unmap_ring_state; + } + + goto out; + +out_unmap_ring_state: + rdma_pci_dma_unmap(pci_dev, r->ring_state, TARGET_PAGE_SIZE); + +out_free_ring_mem: + g_free(r); + +out: + rdma_pci_dma_unmap(pci_dev, tbl, TARGET_PAGE_SIZE); + rdma_pci_dma_unmap(pci_dev, dir, TARGET_PAGE_SIZE); + + return rc; +} + +static void destroy_srq_ring(PvrdmaRing *ring) +{ + pvrdma_ring_free(ring); + rdma_pci_dma_unmap(ring->dev, ring->ring_state, TARGET_PAGE_SIZE); + g_free(ring); +} + +static int create_srq(PVRDMADev *dev, union pvrdma_cmd_req *req, + union pvrdma_cmd_resp *rsp) +{ + struct pvrdma_cmd_create_srq *cmd =3D &req->create_srq; + struct pvrdma_cmd_create_srq_resp *resp =3D &rsp->create_srq_resp; + PvrdmaRing *ring =3D NULL; + int rc; + + memset(resp, 0, sizeof(*resp)); + + rc =3D create_srq_ring(PCI_DEVICE(dev), &ring, cmd->pdir_dma, + cmd->attrs.max_wr, cmd->attrs.max_sge, + cmd->nchunks); + if (rc) { + return rc; + } + + rc =3D rdma_rm_alloc_srq(&dev->rdma_dev_res, cmd->pd_handle, + cmd->attrs.max_wr, cmd->attrs.max_sge, + cmd->attrs.srq_limit, &resp->srqn, ring); + if (rc) { + destroy_srq_ring(ring); + return rc; + } + + return 0; +} + +static int query_srq(PVRDMADev *dev, union pvrdma_cmd_req *req, + union pvrdma_cmd_resp *rsp) +{ + struct pvrdma_cmd_query_srq *cmd =3D &req->query_srq; + struct pvrdma_cmd_query_srq_resp *resp =3D &rsp->query_srq_resp; + + memset(resp, 0, sizeof(*resp)); + + return rdma_rm_query_srq(&dev->rdma_dev_res, cmd->srq_handle, + (struct ibv_srq_attr *)&resp->attrs); +} + +static int modify_srq(PVRDMADev *dev, union pvrdma_cmd_req *req, + union pvrdma_cmd_resp *rsp) +{ + struct pvrdma_cmd_modify_srq *cmd =3D &req->modify_srq; + + /* Only support SRQ limit */ + if (!(cmd->attr_mask & IBV_SRQ_LIMIT) || + (cmd->attr_mask & IBV_SRQ_MAX_WR)) + return -EINVAL; + + return rdma_rm_modify_srq(&dev->rdma_dev_res, cmd->srq_handle, + (struct ibv_srq_attr *)&cmd->attrs, + cmd->attr_mask); +} + +static int destroy_srq(PVRDMADev *dev, union pvrdma_cmd_req *req, + union pvrdma_cmd_resp *rsp) +{ + struct pvrdma_cmd_destroy_srq *cmd =3D &req->destroy_srq; + RdmaRmSRQ *srq; + PvrdmaRing *ring; + + srq =3D rdma_rm_get_srq(&dev->rdma_dev_res, cmd->srq_handle); + if (!srq) { + return -EINVAL; + } + + ring =3D (PvrdmaRing *)srq->opaque; + destroy_srq_ring(ring); + rdma_rm_dealloc_srq(&dev->rdma_dev_res, cmd->srq_handle); + + return 0; +} + struct cmd_handler { uint32_t cmd; uint32_t ack; @@ -634,6 +777,10 @@ static struct cmd_handler cmd_handlers[] =3D { {PVRDMA_CMD_DESTROY_UC, PVRDMA_CMD_DESTROY_UC_RESP_NOOP, destroy_u= c}, {PVRDMA_CMD_CREATE_BIND, PVRDMA_CMD_CREATE_BIND_RESP_NOOP, create_bi= nd}, {PVRDMA_CMD_DESTROY_BIND, PVRDMA_CMD_DESTROY_BIND_RESP_NOOP, destroy_b= ind}, + {PVRDMA_CMD_CREATE_SRQ, PVRDMA_CMD_CREATE_SRQ_RESP, create_sr= q}, + {PVRDMA_CMD_QUERY_SRQ, PVRDMA_CMD_QUERY_SRQ_RESP, query_srq= }, + {PVRDMA_CMD_MODIFY_SRQ, PVRDMA_CMD_MODIFY_SRQ_RESP, modify_sr= q}, + {PVRDMA_CMD_DESTROY_SRQ, PVRDMA_CMD_DESTROY_SRQ_RESP, destroy_s= rq}, }; =20 int pvrdma_exec_cmd(PVRDMADev *dev) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 0b46561bad..769f7990f8 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -53,6 +53,7 @@ static Property pvrdma_dev_properties[] =3D { DEFINE_PROP_INT32("dev-caps-max-qp-init-rd-atom", PVRDMADev, dev_attr.max_qp_init_rd_atom, MAX_QP_INIT_RD_ATOM), DEFINE_PROP_INT32("dev-caps-max-ah", PVRDMADev, dev_attr.max_ah, MAX_A= H), + DEFINE_PROP_INT32("dev-caps-max-srq", PVRDMADev, dev_attr.max_srq, MAX= _SRQ), DEFINE_PROP_CHR("mad-chardev", PVRDMADev, mad_chr), DEFINE_PROP_END_OF_LIST(), }; @@ -261,6 +262,9 @@ static void init_dsr_dev_caps(PVRDMADev *dev) dsr->caps.max_mr =3D dev->dev_attr.max_mr; dsr->caps.max_pd =3D dev->dev_attr.max_pd; dsr->caps.max_ah =3D dev->dev_attr.max_ah; + dsr->caps.max_srq =3D dev->dev_attr.max_srq; + dsr->caps.max_srq_wr =3D dev->dev_attr.max_srq_wr; + dsr->caps.max_srq_sge =3D dev->dev_attr.max_srq_sge; dsr->caps.gid_tbl_len =3D MAX_GIDS; dsr->caps.sys_image_guid =3D 0; dsr->caps.node_guid =3D dev->node_guid; @@ -485,6 +489,13 @@ static void pvrdma_uar_write(void *opaque, hwaddr addr= , uint64_t val, pvrdma_cq_poll(&dev->rdma_dev_res, val & PVRDMA_UAR_HANDLE_MAS= K); } break; + case PVRDMA_UAR_SRQ_OFFSET: + if (val & PVRDMA_UAR_SRQ_RECV) { + trace_pvrdma_uar_write(addr, val, "QP", "SRQ", + val & PVRDMA_UAR_HANDLE_MASK, 0); + pvrdma_srq_recv(dev, val & PVRDMA_UAR_HANDLE_MASK); + } + break; default: rdma_error_report("Unsupported command, addr=3D0x%"PRIx64", val=3D= 0x%"PRIx64, addr, val); @@ -554,6 +565,11 @@ static void init_dev_caps(PVRDMADev *dev) =20 dev->dev_attr.max_cqe =3D pg_tbl_bytes / sizeof(struct pvrdma_cqe) - TARGET_PAGE_SIZE; /* First page is ring state = */ + + dev->dev_attr.max_srq_wr =3D pg_tbl_bytes / + ((sizeof(struct pvrdma_rq_wqe_hdr) + + sizeof(struct pvrdma_sge)) * + dev->dev_attr.max_sge) - TARGET_PAGE_SIZE; } =20 static int pvrdma_check_ram_shared(Object *obj, void *opaque) diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 5b9786efbe..bd6db858de 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -70,7 +70,7 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_ha= ndle, =20 memset(cqe1, 0, sizeof(*cqe1)); cqe1->wr_id =3D cqe->wr_id; - cqe1->qp =3D cqe->qp; + cqe1->qp =3D cqe->qp ? cqe->qp : wc->qp_num; cqe1->opcode =3D cqe->opcode; cqe1->status =3D wc->status; cqe1->byte_len =3D wc->byte_len; @@ -241,6 +241,50 @@ void pvrdma_qp_recv(PVRDMADev *dev, uint32_t qp_handle) } } =20 +void pvrdma_srq_recv(PVRDMADev *dev, uint32_t srq_handle) +{ + RdmaRmSRQ *srq; + PvrdmaRqWqe *wqe; + PvrdmaRing *ring; + + srq =3D rdma_rm_get_srq(&dev->rdma_dev_res, srq_handle); + if (unlikely(!srq)) { + return; + } + + ring =3D (PvrdmaRing *)srq->opaque; + + wqe =3D (struct PvrdmaRqWqe *)pvrdma_ring_next_elem_read(ring); + while (wqe) { + CompHandlerCtx *comp_ctx; + + /* Prepare CQE */ + comp_ctx =3D g_malloc(sizeof(CompHandlerCtx)); + comp_ctx->dev =3D dev; + comp_ctx->cq_handle =3D srq->recv_cq_handle; + comp_ctx->cqe.wr_id =3D wqe->hdr.wr_id; + comp_ctx->cqe.qp =3D 0; + comp_ctx->cqe.opcode =3D IBV_WC_RECV; + + if (wqe->hdr.num_sge > dev->dev_attr.max_sge) { + rdma_error_report("Invalid num_sge=3D%d (max %d)", wqe->hdr.nu= m_sge, + dev->dev_attr.max_sge); + complete_with_error(VENDOR_ERR_INV_NUM_SGE, comp_ctx); + continue; + } + + rdma_backend_post_srq_recv(&dev->backend_dev, &srq->backend_srq, + (struct ibv_sge *)&wqe->sge[0], + wqe->hdr.num_sge, + comp_ctx); + + pvrdma_ring_read_inc(ring); + + wqe =3D pvrdma_ring_next_elem_read(ring); + } + +} + void pvrdma_cq_poll(RdmaDeviceResources *dev_res, uint32_t cq_handle) { RdmaRmCQ *cq; diff --git a/hw/rdma/vmw/pvrdma_qp_ops.h b/hw/rdma/vmw/pvrdma_qp_ops.h index 31cb48ba29..82e720a76f 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.h +++ b/hw/rdma/vmw/pvrdma_qp_ops.h @@ -22,6 +22,7 @@ int pvrdma_qp_ops_init(void); void pvrdma_qp_ops_fini(void); void pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle); void pvrdma_qp_recv(PVRDMADev *dev, uint32_t qp_handle); +void pvrdma_srq_recv(PVRDMADev *dev, uint32_t srq_handle); void pvrdma_cq_poll(RdmaDeviceResources *dev_res, uint32_t cq_handle); =20 #endif --=20 2.17.1