From nobody Fri Mar 29 06:09:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1573051976; cv=none; d=zoho.com; s=zohoarc; b=cPN0fWiIm5DaFelK0LOheykNyZvscOI3bF450jCUmWX/Cj1eAkB8RBhzcEh2YoqG7ie3Tf9CEuwzC0y8iXP0qEAmUMSTtb2Lfxj4vALsnNDtIB60pIhmklraUaFvvGk9uzVczrdhtX1xw2VjtFAOTlmpad8RepZu9oTSck6/rpk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1573051976; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=ovURCSP5JXCPyC2mT33Og+1FsLn1SM62g4JS3pgQdkY=; b=LFChTSAstb2bN9IjQq4WUtN2Z66MBnTRc9hAuFMQIVoDHxkrpmmXiL+lb+eSN3294hnjtGFXDAx4Zf7lzeQxy7+nF4X0tNQhu401N5LoqNBfXtdTJygz0widIoewb+HvDk2N1zA3jWQDvFeSMqi/kTFl1egzGA4PSERi4x4S7bE= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1573051976542537.6277794118428; Wed, 6 Nov 2019 06:52:56 -0800 (PST) Received: from localhost ([::1]:59704 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iSMg2-0008J2-94 for importer@patchew.org; Wed, 06 Nov 2019 09:52:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35322) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iSMek-00073k-8J for qemu-devel@nongnu.org; Wed, 06 Nov 2019 09:51:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iSMej-00041T-6k for qemu-devel@nongnu.org; Wed, 06 Nov 2019 09:51:34 -0500 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]:55988) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iSMej-00040G-0x for qemu-devel@nongnu.org; Wed, 06 Nov 2019 09:51:33 -0500 Received: by mail-wm1-x342.google.com with SMTP id b11so860539wmb.5 for ; Wed, 06 Nov 2019 06:51:32 -0800 (PST) Received: from localhost.localdomain ([37.142.144.12]) by smtp.gmail.com with ESMTPSA id t133sm3116744wmb.1.2019.11.06.06.51.30 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 06 Nov 2019 06:51:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ovURCSP5JXCPyC2mT33Og+1FsLn1SM62g4JS3pgQdkY=; b=d5FgI29HrDPIUPIV9T3sXMvOh4keDlResSCx/mqGCZmTB6Gh+93PH4UsB9CRy2H8G8 wxPpWuaUXaabi4sbuxCaqllmsQosF4FlnK8CU9E+iWjEj9Bssfzcl8yl9hC49z759IWa ebJd0lUuH7Q//G7vxE3FSkdW0uJJGfxT9eYRcWH063sPRTzTcTSDs9iK3TtTmMKVNTlt MaK6vze8qei6TFd7pYh7/1f5BXve2u1UkyAOeTY7L68T+h3ADw00RsKUN50QYvaVfVuF KhSj3W9zAV7vw24jaJ3kYoiicg8+Ym1BM00gaLbyPFUvFmX3o94U7+c1Z6Ye7nDQ60nb qblA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ovURCSP5JXCPyC2mT33Og+1FsLn1SM62g4JS3pgQdkY=; b=RI4PfvGdSqkTtKzuFQdMmMmAfopvcxYrEgV6TtP30rEbzDxg0kZ7+jLrHdQePFyeUH OBEAL2LBNYG93Bqa3qcRWNMpiI843DEYiZH5p1BhTniqvsxyQ3vFUOVYszHmy+qHJJmi lhU3eAql35skayUYJczhp5wJI+roNQy6P9pPq8hJQgdn/IDzUlcEXYl7lmFPuJ5tt81+ Y8JedGOkSacAraBcRgR5xZRHqW81V3Uqh6rdkP2MIS8klbFtVXn/l76CjWfjVQW+3/K2 ufmmTocm0YsB4YaQk2y64nUKwWaY+4MAYxbrKVaglGE+JExPSIY7mpQtpGlZuCewWSzY 1exA== X-Gm-Message-State: APjAAAVqz1xqDfJWJ6oSy9+3n0FDf0EA0K3Ks3NSvNHLfQAwydEKcWqj j5YRCFDBROnwk2S8MULRFx6zVq8u X-Google-Smtp-Source: APXvYqx/1IhOwpMfK1PDQUwVEPQQSbc6tRYcQL6bYLAsW8WJ5BJ063vfgc7ulx2mi9fNCTzKAe3sGQ== X-Received: by 2002:a1c:6854:: with SMTP id d81mr3058894wmc.75.1573051891727; Wed, 06 Nov 2019 06:51:31 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Subject: [Qemu-devel] [PATCH PULL 1/2] configure: Check if we can use ibv_reg_mr_iova Date: Wed, 6 Nov 2019 16:51:26 +0200 Message-Id: <20191106145127.23700-2-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191106145127.23700-1-marcel.apfelbaum@gmail.com> References: <20191106145127.23700-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::342 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Yuval Shaia The function reg_mr_iova is an enhanced version of ibv_reg_mr function that can help to easly register and use guest's MRs. Add check in 'configure' phase to detect if we have libibverbs with this support. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Message-Id: <20190818132107.18181-2-yuval.shaia@oracle.com> Signed-off-by: Marcel Apfelbaum --- configure | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/configure b/configure index 72553f98ea..efe165edf9 100755 --- a/configure +++ b/configure @@ -3217,6 +3217,34 @@ else pvrdma=3D"no" fi =20 +# Let's see if enhanced reg_mr is supported +if test "$pvrdma" =3D "yes" ; then + +cat > $TMPC < + +int +main(void) +{ + struct ibv_mr *mr; + struct ibv_pd *pd =3D NULL; + size_t length =3D 10; + uint64_t iova =3D 0; + int access =3D 0; + void *addr =3D NULL; + + mr =3D ibv_reg_mr_iova(pd, addr, length, iova, access); + + ibv_dereg_mr(mr); + + return 0; +} +EOF + if ! compile_prog "" "-libverbs"; then + QEMU_CFLAGS=3D"$QEMU_CFLAGS -DLEGACY_RDMA_REG_MR" + fi +fi + ########################################## # VNC SASL detection if test "$vnc" =3D "yes" && test "$vnc_sasl" !=3D "no" ; then --=20 2.17.2 From nobody Fri Mar 29 06:09:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1573051978; cv=none; d=zoho.com; s=zohoarc; b=O98CEFP/MQbuHt+yNLDQRvdrksRpEWyM7q0Q9VmlYpZr8TU8s/5APU28oSVrf/fiLKha8JW9LK1v3EZvYCUIaMdqpVxGt97DIYQM/noCa1KBDKNTojhQhwoqxv94UjHXf9Tr90bGpcBI4/9x7Ac4JJfGL6TSBcsL6w9OwjH/tFQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1573051978; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=VoLJFtEIkSYy1clLkMnJOf/MQB43jYk2R2juAnJ5n0w=; b=Yo6EbfiYsOnCrA6u3grofuJtSnYDeTMZqb/2E/343KSaOOtatfx5wbyNpA1MgVQB0IeO1aRZvfLVuAUvyqSUZEmFA9IB/VZlqjHN7LEZv1FYPc5KXBB44nDEb4YZsNOapkQM0RaJkjnub2WB81Ft3N3+BItleuIWA51UkXhRYpE= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 157305197826359.291720486682266; Wed, 6 Nov 2019 06:52:58 -0800 (PST) Received: from localhost ([::1]:59706 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iSMg4-0008NG-2H for importer@patchew.org; Wed, 06 Nov 2019 09:52:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35337) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iSMel-00073x-V5 for qemu-devel@nongnu.org; Wed, 06 Nov 2019 09:51:37 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iSMek-000422-N3 for qemu-devel@nongnu.org; Wed, 06 Nov 2019 09:51:35 -0500 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]:55989) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iSMek-00041b-H8 for qemu-devel@nongnu.org; Wed, 06 Nov 2019 09:51:34 -0500 Received: by mail-wm1-x343.google.com with SMTP id b11so860643wmb.5 for ; Wed, 06 Nov 2019 06:51:34 -0800 (PST) Received: from localhost.localdomain ([37.142.144.12]) by smtp.gmail.com with ESMTPSA id t133sm3116744wmb.1.2019.11.06.06.51.31 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 06 Nov 2019 06:51:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=VoLJFtEIkSYy1clLkMnJOf/MQB43jYk2R2juAnJ5n0w=; b=G9QxXKomPNS7jxfHk2I1SAdPxK9GQNCkXwJ3rk8FXbQiQrd7a0XnDqN6+Rp0Kt6iO1 JPf2pVXpJJg1eQ67b0RovLOGBWpJKMhrXzAzRLI+9RL5KfSPQSSMwir0WlAYQ4MMtQLD k0BGGjtB4R0XAk8WA9I0Jay3Yt+mhZJd94smkrqD7oZXR+IbXn7qYm3wQMgkwP3fp4rO +PG+PGRzszw4KzFVsffkqcVTPboZiqHKl5PfKf1NBvHVqhYTAFyso1vkbMimyQbBMe1l ph3hNkPf4RkrHj1vWLNkMwhc7GHdx1xEGhm0L4Jn6oxp6A19gw29Qau1VnG9oHGdSGlq FOyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=VoLJFtEIkSYy1clLkMnJOf/MQB43jYk2R2juAnJ5n0w=; b=K0gSnP/gg/knzFLdRyXtGD8hohzWJAok2nmaamMF157ut7zt7g2Kc4C4UwBU11JURc 3I3D40GR7tWU49IPgacBznRMi5VVcYwOEFz+5pqr40m9MrtQGZC02SzXiyFl85ZxoAyq 1Z1GrDGPdA7TUpLs4bU0he9PeHql0H/8Wviyx4ylffyPHXaYLo/mTLybaFAKEvzAy9ds Kyx7emNFc5hn8tiHlXi6Umuc0hbFF/+OX31+m0q74JSbQ6ZGXPvA3SbIrY6ZopvKVX0r qzxN506+CTlLXSnymmWP2I8dTaZxBFzOQagyIvj8IZcQqujyDo+sz+dH5+RlG4ZlYrRt x68w== X-Gm-Message-State: APjAAAXcQunalaDtPw6kMMPQovxpW+y3Be6AdPrtBfS9bmLdRf0fzM05 hjK9HlMiCQiVNStN4anBQ9NttErh X-Google-Smtp-Source: APXvYqy2JezJHv7u6TmYsgez7AFeOhqSL6obEDXoCk3ulDNBdzjZUFxxh2/taAnKO540E+pmFHdBbA== X-Received: by 2002:a1c:cc16:: with SMTP id h22mr3067861wmb.51.1573051892958; Wed, 06 Nov 2019 06:51:32 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Subject: [Qemu-devel] [PATCH PULL 2/2] hw/rdma: Utilize ibv_reg_mr_iova for memory registration Date: Wed, 6 Nov 2019 16:51:27 +0200 Message-Id: <20191106145127.23700-3-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20191106145127.23700-1-marcel.apfelbaum@gmail.com> References: <20191106145127.23700-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::343 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Yuval Shaia The virtual address that is provided by the guest in post_send and post_recv operations is related to the guest address space. This address space is unknown to the HCA resides on host so extra step in these operations is needed to adjust the address to host virtual address. This step, which is done in data-path affects performances. An enhanced verion of MR registration introduced here https://patchwork.kernel.org/patch/11044467/ can be used so that the guest virtual address space for this MR is known to the HCA in host. This will save the data-path adjustment. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Message-Id: <20190818132107.18181-3-yuval.shaia@oracle.com> Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 13 +++++++++++++ hw/rdma/rdma_backend.h | 5 +++++ hw/rdma/rdma_rm.c | 5 +++++ hw/rdma/vmw/pvrdma_main.c | 6 ++++++ 4 files changed, 29 insertions(+) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index c39051068d..c346407cd3 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -391,7 +391,11 @@ static int build_host_sge_array(RdmaDeviceResources *r= dma_dev_res, return VENDOR_ERR_INVLKEY | ssge[ssge_idx].lkey; } =20 +#ifdef LEGACY_RDMA_REG_MR dsge->addr =3D (uintptr_t)mr->virt + ssge[ssge_idx].addr - mr->sta= rt; +#else + dsge->addr =3D ssge[ssge_idx].addr; +#endif dsge->length =3D ssge[ssge_idx].length; dsge->lkey =3D rdma_backend_mr_lkey(&mr->backend_mr); =20 @@ -735,10 +739,19 @@ void rdma_backend_destroy_pd(RdmaBackendPD *pd) } } =20 +#ifdef LEGACY_RDMA_REG_MR int rdma_backend_create_mr(RdmaBackendMR *mr, RdmaBackendPD *pd, void *add= r, size_t length, int access) +#else +int rdma_backend_create_mr(RdmaBackendMR *mr, RdmaBackendPD *pd, void *add= r, + size_t length, uint64_t guest_start, int access) +#endif { +#ifdef LEGACY_RDMA_REG_MR mr->ibmr =3D ibv_reg_mr(pd->ibpd, addr, length, access); +#else + mr->ibmr =3D ibv_reg_mr_iova(pd->ibpd, addr, length, guest_start, acce= ss); +#endif if (!mr->ibmr) { rdma_error_report("ibv_reg_mr fail, errno=3D%d", errno); return -EIO; diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 7c1a19a2b5..127f96e2d5 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -78,8 +78,13 @@ int rdma_backend_query_port(RdmaBackendDev *backend_dev, int rdma_backend_create_pd(RdmaBackendDev *backend_dev, RdmaBackendPD *pd); void rdma_backend_destroy_pd(RdmaBackendPD *pd); =20 +#ifdef LEGACY_RDMA_REG_MR int rdma_backend_create_mr(RdmaBackendMR *mr, RdmaBackendPD *pd, void *add= r, size_t length, int access); +#else +int rdma_backend_create_mr(RdmaBackendMR *mr, RdmaBackendPD *pd, void *add= r, + size_t length, uint64_t guest_start, int access= ); +#endif void rdma_backend_destroy_mr(RdmaBackendMR *mr); =20 int rdma_backend_create_cq(RdmaBackendDev *backend_dev, RdmaBackendCQ *cq, diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 1927f85472..1524dfaeaa 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -227,8 +227,13 @@ int rdma_rm_alloc_mr(RdmaDeviceResources *dev_res, uin= t32_t pd_handle, mr->length =3D guest_length; mr->virt +=3D (mr->start & (TARGET_PAGE_SIZE - 1)); =20 +#ifdef LEGACY_RDMA_REG_MR ret =3D rdma_backend_create_mr(&mr->backend_mr, &pd->backend_pd, m= r->virt, mr->length, access_flags); +#else + ret =3D rdma_backend_create_mr(&mr->backend_mr, &pd->backend_pd, m= r->virt, + mr->length, guest_start, access_flags= ); +#endif if (ret) { ret =3D -EIO; goto out_dealloc_mr; diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 3722d9e772..6f0fc405c7 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -664,6 +664,12 @@ static void pvrdma_realize(PCIDevice *pdev, Error **er= rp) dev->shutdown_notifier.notify =3D pvrdma_shutdown_notifier; qemu_register_shutdown_notifier(&dev->shutdown_notifier); =20 +#ifdef LEGACY_RDMA_REG_MR + rdma_info_report("Using legacy reg_mr"); +#else + rdma_info_report("Using iova reg_mr"); +#endif + out: if (rc) { pvrdma_fini(pdev); --=20 2.17.2