From nobody Fri Nov 7 03:53:14 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 154547387764928.85836991314875; Sat, 22 Dec 2018 02:17:57 -0800 (PST) Received: from localhost ([::1]:42760 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeLu-0001XL-WF for importer@patchew.org; Sat, 22 Dec 2018 05:17:51 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41888) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwO-0006ay-TW for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwN-0006la-9g for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:38456) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwM-0006cB-Vr for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: by mail-wr1-x442.google.com with SMTP id v13so7515556wrw.5 for ; Sat, 22 Dec 2018 01:51:15 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hay2rzA2FV36Ph6SD4BcCUe/zYRBRkL25GycwDCZ3nU=; b=CW5wXPDXSanRlqTfc7LMluvT/ZhTFBHj+tJAjAz4EDqwZKME0QM/kAlmiyTWM8AgZj RF2XP3n0qt2SWGBPGGlW7MRewfnz0ZpSw5z2vnySsuYWsAPxEURgcW668nMRBiZ58Mb4 WqCvisM1cB6VL8rUPOV8mkuI60Wjg0LE7mu/GBX9v6yfIhhYA4CKN1ynhfdihzYSsFv6 SVcEG1OTJMWeQ3TMOHEAR8YdvJWnMH3wdIXLS5TVtHSBEFiTLDfdfxOKyr5GfLJ6GlUm tNEXdybOcSU3M51UyX73zWUZ6uUIFffDjZcCnPnmu/wITa7gGJnLURr9WZjMu+YKrqWW cwfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hay2rzA2FV36Ph6SD4BcCUe/zYRBRkL25GycwDCZ3nU=; b=gojZExPjCpkGy+yhf44RqRDcyuBR6wP0NwRdlA6Nw0OR5Sxc3LCk3rRuTDCSk2gL9u /8oeroJZKDkf65iBvIUqMJf7vHbItqPvSX6s9UmhHMdNrLcZez8YFN1LKGw45FVTeroO fPzHbK4r3Y8m+oHkejautCK6PHRy1mL4xC/qAXGFf71BgjhbrOoCOBr/bCK8GEIehvGO G7oEF/eiYnhqFEBtXmBhlJgxtYGORbkGpvw5WtuAAeoh00xZ0lfsZQSb/L8KTjtk9O1w CMWTtXUPQ6hUDBdBnsmGSvuGsz9fKNv6fJA3Wi+pIbSrxgwlOYCg9ptuNjztvCSNwVe3 UHSQ== X-Gm-Message-State: AJcUukdKM3qyGGYBVppmjKHOP7ajzWetPUrcTxqRRKBRmkVb2ia48fz8 tZcM51+PrZu7QddfHdmm3NJ75zKf X-Google-Smtp-Source: ALg8bN4UtI8a1LwvFbzHjjJkKXGqghr+AL/Fvln3KpAdTLfhmW7OfFKN28DoPqkmw43P/csCmQA56Q== X-Received: by 2002:adf:eec9:: with SMTP id a9mr5539320wrp.242.1545472274010; Sat, 22 Dec 2018 01:51:14 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:29 +0200 Message-Id: <20181222095036.29743-25-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PATCH PULL 24/31] docs: Update pvrdma device documentation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Yuval Shaia Interface with the device is changed with the addition of support for MAD packets. Adjust documentation accordingly. While there fix a minor mistake which may lead to think that there is a relation between using RXE on host and the compatibility with bare-metal peers. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- docs/pvrdma.txt | 126 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 107 insertions(+), 19 deletions(-) diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt index 5599318159..5175251b47 100644 --- a/docs/pvrdma.txt +++ b/docs/pvrdma.txt @@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for an= y special guest modifications. =20 While it complies with the VMware device, it can also communicate with bare -metal RDMA-enabled machines and does not require an RDMA HCA in the host, = it -can work with Soft-RoCE (rxe). +metal RDMA-enabled machines as peers. + +It does not require an RDMA HCA in the host, it can work with Soft-RoCE (r= xe). =20 It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be @@ -78,29 +79,116 @@ the required RDMA libraries. =20 3. Usage =3D=3D=3D=3D=3D=3D=3D=3D + + +3.1 VM Memory settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Currently the device is working only with memory backed RAM and it must be mark as "shared": -m 1G \ -object memory-backend-ram,id=3Dmb1,size=3D1G,share \ -numa node,memdev=3Dmb1 \ =20 -The pvrdma device is composed of two functions: - - Function 0 is a vmxnet Ethernet Device which is redundant in Guest - but is required to pass the ibdevice GID using its MAC. - Examples: - For an rxe backend using eth0 interface it will use its mac: - -device vmxnet3,addr=3D.0,multifunction=3Don,mac=3D - For an SRIOV VF, we take the Ethernet Interface exposed by it: - -device vmxnet3,multifunction=3Don,mac=3D - - Function 1 is the actual device: - -device pvrdma,addr=3D.1,backend-dev=3D,backend-gid= -idx=3D,backend-port=3D - where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) - Note: Pay special attention that the GID at backend-gid-idx matches vmxne= t's MAC. - The rules of conversion are part of the RoCE spec, but since manual conve= rsion - is not required, spotting problems is not hard: - Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a - MAC: 7c:fe:90:cb:74:3a - Note the difference between the first byte of the MAC and the GID. + +3.2 MAD Multiplexer +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +MAD Multiplexer is a service that exposes MAD-like interface for VMs in +order to overcome the limitation where only single entity can register with +MAD layer to send and receive RDMA-CM MAD packets. + +To build rdmacm-mux run +# make rdmacm-mux + +The application accepts 3 command line arguments and exposes a UNIX socket +to pass control and data to it. +-d rdma-device-name Name of RDMA device to register with +-s unix-socket-path Path to unix socket to listen (default /var/run/rdmac= m-mux) +-p rdma-device-port Port number of RDMA device to register with (default = 1) +The final UNIX socket file name is a concatenation of the 3 arguments so +for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2 +will be created. + +pvrdma requires this service. + +Please refer to contrib/rdmacm-mux for more details. + + +3.3 Service exposed by libvirt daemon +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The control over the RDMA device's GID table is done by updating the +device's Ethernet function addresses. +Usually the first GID entry is determined by the MAC address, the second by +the first IPv6 address and the third by the IPv4 address. Other entries can +be added by adding more IP addresses. The opposite is the same, i.e. +whenever an address is removed, the corresponding GID entry is removed. +The process is done by the network and RDMA stacks. Whenever an address is +added the ib_core driver is notified and calls the device driver add_gid +function which in turn update the device. +To support this in pvrdma device the device hooks into the create_bind and +destroy_bind HW commands triggered by pvrdma driver in guest. + +Whenever changed is made to the pvrdma port's GID table a special QMP +messages is sent to be processed by libvirt to update the address of the +backend Ethernet device. + +pvrdma requires that libvirt service will be up. + + +3.4 PCI devices settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +RoCE device exposes two functions - an Ethernet and RDMA. +To support it, pvrdma device is composed of two PCI functions, an Ethernet +device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The +Ethernet function can be used for other Ethernet purposes such as IP. + + +3.5 Device parameters +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +- netdev: Specifies the Ethernet device function name on the host for + example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet + device used to create it. +- ibdev: The IB device name on host for example rxe0, mlx5_0 etc. +- mad-chardev: The name of the MAD multiplexer char device. +- ibport: In case of multi-port device (such as Mellanox's HCA) this + specify the port to use. If not set 1 will be used. +- dev-caps-max-mr-size: The maximum size of MR. +- dev-caps-max-qp: Maximum number of QPs. +- dev-caps-max-sge: Maximum number of SGE elements in WR. +- dev-caps-max-cq: Maximum number of CQs. +- dev-caps-max-mr: Maximum number of MRs. +- dev-caps-max-pd: Maximum number of PDs. +- dev-caps-max-ah: Maximum number of AHs. + +Notes: +- The first 3 parameters are mandatory settings, the rest have their + defaults. +- The last 8 parameters (the ones that prefixed by dev-caps) defines the t= op + limits but the final values is adjusted by the backend device limitation= s. +- netdev can be extracted from ibdev's sysfs + (/sys/class/infiniband//device/net/) + + +3.6 Example +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Define bridge device with vmxnet3 network backend: + + + + +
+ + +Define pvrdma device: + + + + + + + + + + =20 =20 =20 --=20 2.17.1