From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093398614943.94708824398; Mon, 12 Nov 2018 23:16:38 -0800 (PST) Received: from localhost ([::1]:52320 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSw8-00051Z-Up for importer@patchew.org; Tue, 13 Nov 2018 02:16:36 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51576) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMStp-0003he-Rq for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMStn-0002VZ-6i for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:13 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:59288) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMStm-0002V2-SB for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:11 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7E9IO046110; Tue, 13 Nov 2018 07:14:09 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2nnw6egpmq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:08 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7E3xa019503 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:03 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7E2U6017419; Tue, 13 Nov 2018 07:14:02 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:01 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=W1rJf68BGm5BgYz41g+2SB0TaNcyLb74gAqt4g60BaI=; b=I7h3pEs2/Zs+5KXuB0fk/mwnlIA2zTKIpNldG04LpCuGAdANC//rDSvN8AoT3k/o2Hif 7V3Mj7krBEq6LdHvtcCvAWWrWxGdZIt9LQEcNQxTQKxYf2rn1ADMx8QMhPyvCeAm3YVg 2gDRvWnsfZahXXDtih8BB13kKr+Dsi9lOmvIMnBdplWWthL+1vwvBse5vBrnCZ8j7QYN q8ViyQk2FcY8lJQKNBzcR+2+vccLX2fvGxhQO2K1R2VMpeHX0QHylNibMFMmzomIkbXA ywNA+lXC+gkIF0/2vUX3Kp8qJgriQsreUFV/FZGPqF4q/Iy/quCq9bcUBWhvOF+F2JXa Zw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:50 +0200 Message-Id: <20181113071336.6242-2-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 01/23] contrib/rdmacm-mux: Add implementation of RDMA User MAD multiplexer X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" RDMA MAD kernel module (ibcm) disallow more than one MAD-agent for a given MAD class. This does not go hand-by-hand with qemu pvrdma device's requirements where each VM is MAD agent. Fix it by adding implementation of RDMA MAD multiplexer service which on one hand register as a sole MAD agent with the kernel module and on the other hand gives service to more than one VM. Design Overview: ---------------- A server process is registered to UMAD framework (for this to work the rdma_cm kernel module needs to be unloaded) and creates a unix socket to listen to incoming request from clients. A client process (such as QEMU) connects to this unix socket and registers with its own GID. TX: --- When client needs to send rdma_cm MAD message it construct it the same way as without this multiplexer, i.e. creates a umad packet but this time it writes its content to the socket instead of calling umad_send(). The server, upon receiving such a message fetch local_comm_id from it so a context for this session can be maintain and relay the message to UMAD layer by calling umad_send(). RX: --- The server creates a worker thread to process incoming rdma_cm MAD messages. When an incoming message arrived (umad_recv()) the server, depending on the message type (attr_id) looks for target client by either searching in gid->fd table or in local_comm_id->fd table. With the extracted fd the server relays to incoming message to the client. Signed-off-by: Yuval Shaia --- MAINTAINERS | 1 + Makefile | 3 + Makefile.objs | 1 + contrib/rdmacm-mux/Makefile.objs | 4 + contrib/rdmacm-mux/main.c | 771 +++++++++++++++++++++++++++++++ contrib/rdmacm-mux/rdmacm-mux.h | 56 +++ 6 files changed, 836 insertions(+) create mode 100644 contrib/rdmacm-mux/Makefile.objs create mode 100644 contrib/rdmacm-mux/main.c create mode 100644 contrib/rdmacm-mux/rdmacm-mux.h diff --git a/MAINTAINERS b/MAINTAINERS index 98a1856afc..e087d58ac6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2231,6 +2231,7 @@ S: Maintained F: hw/rdma/* F: hw/rdma/vmw/* F: docs/pvrdma.txt +F: contrib/rdmacm-mux/* =20 Build and test automation ------------------------- diff --git a/Makefile b/Makefile index f2947186a4..94072776ff 100644 --- a/Makefile +++ b/Makefile @@ -418,6 +418,7 @@ dummy :=3D $(call unnest-vars,, \ elf2dmp-obj-y \ ivshmem-client-obj-y \ ivshmem-server-obj-y \ + rdmacm-mux-obj-y \ libvhost-user-obj-y \ vhost-user-scsi-obj-y \ vhost-user-blk-obj-y \ @@ -725,6 +726,8 @@ vhost-user-scsi$(EXESUF): $(vhost-user-scsi-obj-y) libv= host-user.a $(call LINK, $^) vhost-user-blk$(EXESUF): $(vhost-user-blk-obj-y) libvhost-user.a $(call LINK, $^) +rdmacm-mux$(EXESUF): $(rdmacm-mux-obj-y) $(COMMON_LDADDS) + $(call LINK, $^) =20 module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak $(call quiet-command,$(PYTHON) $< $@ \ diff --git a/Makefile.objs b/Makefile.objs index 1e1ff387d7..cc7df3ad80 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -194,6 +194,7 @@ vhost-user-scsi.o-cflags :=3D $(LIBISCSI_CFLAGS) vhost-user-scsi.o-libs :=3D $(LIBISCSI_LIBS) vhost-user-scsi-obj-y =3D contrib/vhost-user-scsi/ vhost-user-blk-obj-y =3D contrib/vhost-user-blk/ +rdmacm-mux-obj-y =3D contrib/rdmacm-mux/ =20 ###################################################################### trace-events-subdirs =3D diff --git a/contrib/rdmacm-mux/Makefile.objs b/contrib/rdmacm-mux/Makefile= .objs new file mode 100644 index 0000000000..be3eacb6f7 --- /dev/null +++ b/contrib/rdmacm-mux/Makefile.objs @@ -0,0 +1,4 @@ +ifdef CONFIG_PVRDMA +CFLAGS +=3D -libumad -Wno-format-truncation +rdmacm-mux-obj-y =3D main.o +endif diff --git a/contrib/rdmacm-mux/main.c b/contrib/rdmacm-mux/main.c new file mode 100644 index 0000000000..47cf0ac7bc --- /dev/null +++ b/contrib/rdmacm-mux/main.c @@ -0,0 +1,771 @@ +/* + * QEMU paravirtual RDMA - rdmacm-mux implementation + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "sys/poll.h" +#include "sys/ioctl.h" +#include "pthread.h" +#include "syslog.h" + +#include "infiniband/verbs.h" +#include "infiniband/umad.h" +#include "infiniband/umad_types.h" +#include "infiniband/umad_sa.h" +#include "infiniband/umad_cm.h" + +#include "rdmacm-mux.h" + +#define SCALE_US 1000 +#define COMMID_TTL 2 /* How many SCALE_US a context of MAD session is save= d */ +#define SLEEP_SECS 5 /* This is used both in poll() and thread */ +#define SERVER_LISTEN_BACKLOG 10 +#define MAX_CLIENTS 4096 +#define MAD_RMPP_VERSION 0 +#define MAD_METHOD_MASK0 0x8 + +#define IB_USER_MAD_LONGS_PER_METHOD_MASK (128 / (8 * sizeof(long))) + +#define CM_REQ_DGID_POS 80 +#define CM_SIDR_REQ_DGID_POS 44 + +/* The below can be override by command line parameter */ +#define UNIX_SOCKET_PATH "/var/run/rdmacm-mux" +#define RDMA_DEVICE "rxe0" +#define RDMA_PORT_NUM 1 + +typedef struct RdmaCmServerArgs { + char unix_socket_path[PATH_MAX]; + char rdma_dev_name[NAME_MAX]; + int rdma_port_num; +} RdmaCMServerArgs; + +typedef struct CommId2FdEntry { + int fd; + int ttl; /* Initialized to 2, decrement each timeout, entry delete whe= n 0 */ + __be64 gid_ifid; +} CommId2FdEntry; + +typedef struct RdmaCmUMadAgent { + int port_id; + int agent_id; + GHashTable *gid2fd; /* Used to find fd of a given gid */ + GHashTable *commid2fd; /* Used to find fd on of a given comm_id */ +} RdmaCmUMadAgent; + +typedef struct RdmaCmServer { + bool run; + RdmaCMServerArgs args; + struct pollfd fds[MAX_CLIENTS]; + int nfds; + RdmaCmUMadAgent umad_agent; + pthread_t umad_recv_thread; + pthread_rwlock_t lock; +} RdmaCMServer; + +static RdmaCMServer server =3D {0}; + +static void usage(const char *progname) +{ + printf("Usage: %s [OPTION]...\n" + "Start a RDMA-CM multiplexer\n" + "\n" + "\t-h Show this help\n" + "\t-s unix-socket-path Path to unix socket to listen on (defa= ult %s)\n" + "\t-d rdma-device-name Name of RDMA device to register with (= default %s)\n" + "\t-p rdma-device-port Port number of RDMA device to register= with (default %d)\n", + progname, UNIX_SOCKET_PATH, RDMA_DEVICE, RDMA_PORT_NUM); +} + +static void help(const char *progname) +{ + fprintf(stderr, "Try '%s -h' for more information.\n", progname); +} + +static void parse_args(int argc, char *argv[]) +{ + int c; + char unix_socket_path[PATH_MAX]; + + strcpy(unix_socket_path, UNIX_SOCKET_PATH); + strncpy(server.args.rdma_dev_name, RDMA_DEVICE, NAME_MAX - 1); + server.args.rdma_port_num =3D RDMA_PORT_NUM; + + while ((c =3D getopt(argc, argv, "hs:d:p:")) !=3D -1) { + switch (c) { + case 'h': + usage(argv[0]); + exit(0); + + case 's': + /* This is temporary, final name will build below */ + strncpy(unix_socket_path, optarg, PATH_MAX); + break; + + case 'd': + strncpy(server.args.rdma_dev_name, optarg, NAME_MAX - 1); + break; + + case 'p': + server.args.rdma_port_num =3D atoi(optarg); + break; + + default: + help(argv[0]); + exit(1); + } + } + + /* Build unique unix-socket file name */ + snprintf(server.args.unix_socket_path, PATH_MAX, "%s-%s-%d", + unix_socket_path, server.args.rdma_dev_name, + server.args.rdma_port_num); + + syslog(LOG_INFO, "unix_socket_path=3D%s", server.args.unix_socket_path= ); + syslog(LOG_INFO, "rdma-device-name=3D%s", server.args.rdma_dev_name); + syslog(LOG_INFO, "rdma-device-port=3D%d", server.args.rdma_port_num); +} + +static void hash_tbl_alloc(void) +{ + + server.umad_agent.gid2fd =3D g_hash_table_new_full(g_int64_hash, + g_int64_equal, + g_free, g_free); + server.umad_agent.commid2fd =3D g_hash_table_new_full(g_int_hash, + g_int_equal, + g_free, g_free); +} + +static void hash_tbl_free(void) +{ + if (server.umad_agent.commid2fd) { + g_hash_table_destroy(server.umad_agent.commid2fd); + } + if (server.umad_agent.gid2fd) { + g_hash_table_destroy(server.umad_agent.gid2fd); + } +} + + +static int _hash_tbl_search_fd_by_ifid(__be64 *gid_ifid) +{ + int *fd; + + fd =3D g_hash_table_lookup(server.umad_agent.gid2fd, gid_ifid); + if (!fd) { + /* Let's try IPv4 */ + *gid_ifid |=3D 0x00000000ffff0000; + fd =3D g_hash_table_lookup(server.umad_agent.gid2fd, gid_ifid); + } + + return fd ? *fd : 0; +} + +static int hash_tbl_search_fd_by_ifid(int *fd, __be64 *gid_ifid) +{ + pthread_rwlock_rdlock(&server.lock); + *fd =3D _hash_tbl_search_fd_by_ifid(gid_ifid); + pthread_rwlock_unlock(&server.lock); + + if (!fd) { + syslog(LOG_WARNING, "Can't find matching for ifid 0x%llx\n", *gid_= ifid); + return -ENOENT; + } + + return 0; +} + +static int hash_tbl_search_fd_by_comm_id(uint32_t comm_id, int *fd, + __be64 *gid_idid) +{ + CommId2FdEntry *fde; + + pthread_rwlock_rdlock(&server.lock); + fde =3D g_hash_table_lookup(server.umad_agent.commid2fd, &comm_id); + pthread_rwlock_unlock(&server.lock); + + if (!fde) { + syslog(LOG_WARNING, "Can't find matching for comm_id 0x%x\n", comm= _id); + return -ENOENT; + } + + *fd =3D fde->fd; + *gid_idid =3D fde->gid_ifid; + + return 0; +} + +static RdmaCmMuxErrCode add_fd_ifid_pair(int fd, __be64 gid_ifid) +{ + int fd1; + + pthread_rwlock_wrlock(&server.lock); + + fd1 =3D _hash_tbl_search_fd_by_ifid(&gid_ifid); + if (fd1) { /* record already exist - an error */ + pthread_rwlock_unlock(&server.lock); + return fd =3D=3D fd1 ? RDMACM_MUX_ERR_CODE_EEXIST : + RDMACM_MUX_ERR_CODE_EACCES; + } + + g_hash_table_insert(server.umad_agent.gid2fd, g_memdup(&gid_ifid, + sizeof(gid_ifid)), g_memdup(&fd, sizeof(fd))); + + pthread_rwlock_unlock(&server.lock); + + syslog(LOG_INFO, "0x%lx registered on socket %d", (uint64_t)gid_ifid, = fd); + + return RDMACM_MUX_ERR_CODE_OK; +} + +static RdmaCmMuxErrCode delete_fd_ifid_pair(int fd, __be64 gid_ifid) +{ + int fd1; + + pthread_rwlock_wrlock(&server.lock); + + fd1 =3D _hash_tbl_search_fd_by_ifid(&gid_ifid); + if (!fd1) { /* record not exist - an error */ + pthread_rwlock_unlock(&server.lock); + return RDMACM_MUX_ERR_CODE_ENOTFOUND; + } + + g_hash_table_remove(server.umad_agent.gid2fd, g_memdup(&gid_ifid, + sizeof(gid_ifid))); + pthread_rwlock_unlock(&server.lock); + + syslog(LOG_INFO, "0x%lx unregistered on socket %d", (uint64_t)gid_ifid= , fd); + + return RDMACM_MUX_ERR_CODE_OK; +} + +static void hash_tbl_save_fd_comm_id_pair(int fd, uint32_t comm_id, + uint64_t gid_ifid) +{ + CommId2FdEntry fde =3D {fd, COMMID_TTL, gid_ifid}; + + pthread_rwlock_wrlock(&server.lock); + g_hash_table_insert(server.umad_agent.commid2fd, + g_memdup(&comm_id, sizeof(comm_id)), + g_memdup(&fde, sizeof(fde))); + pthread_rwlock_unlock(&server.lock); +} + +static gboolean remove_old_comm_ids(gpointer key, gpointer value, + gpointer user_data) +{ + CommId2FdEntry *fde =3D (CommId2FdEntry *)value; + + return !fde->ttl--; +} + +static gboolean remove_entry_from_gid2fd(gpointer key, gpointer value, + gpointer user_data) +{ + if (*(int *)value =3D=3D *(int *)user_data) { + syslog(LOG_INFO, "0x%lx unregistered on socket %d", *(uint64_t *)k= ey, + *(int *)value); + return true; + } + + return false; +} + +static void hash_tbl_remove_fd_ifid_pair(int fd) +{ + pthread_rwlock_wrlock(&server.lock); + g_hash_table_foreach_remove(server.umad_agent.gid2fd, + remove_entry_from_gid2fd, (gpointer)&fd); + pthread_rwlock_unlock(&server.lock); +} + +static int get_fd(const char *mad, int *fd, __be64 *gid_ifid) +{ + struct umad_hdr *hdr =3D (struct umad_hdr *)mad; + char *data =3D (char *)hdr + sizeof(*hdr); + int32_t comm_id; + uint16_t attr_id =3D be16toh(hdr->attr_id); + int rc =3D 0; + + switch (attr_id) { + case UMAD_CM_ATTR_REQ: + memcpy(gid_ifid, data + CM_REQ_DGID_POS, sizeof(*gid_ifid)); + rc =3D hash_tbl_search_fd_by_ifid(fd, gid_ifid); + break; + + case UMAD_CM_ATTR_SIDR_REQ: + memcpy(gid_ifid, data + CM_SIDR_REQ_DGID_POS, sizeof(*gid_ifid)); + rc =3D hash_tbl_search_fd_by_ifid(fd, gid_ifid); + break; + + case UMAD_CM_ATTR_REP: + /* Fall through */ + case UMAD_CM_ATTR_REJ: + /* Fall through */ + case UMAD_CM_ATTR_DREQ: + /* Fall through */ + case UMAD_CM_ATTR_DREP: + /* Fall through */ + case UMAD_CM_ATTR_RTU: + data +=3D sizeof(comm_id); + /* Fall through */ + case UMAD_CM_ATTR_SIDR_REP: + memcpy(&comm_id, data, sizeof(comm_id)); + if (comm_id) { + rc =3D hash_tbl_search_fd_by_comm_id(comm_id, fd, gid_ifid); + } + break; + + default: + rc =3D -EINVAL; + syslog(LOG_WARNING, "Unsupported attr_id 0x%x\n", attr_id); + } + + return rc; +} + +static void *umad_recv_thread_func(void *args) +{ + int rc; + RdmaCmMuxMsg msg =3D {0}; + int fd =3D -2; + + while (server.run) { + do { + msg.umad_len =3D sizeof(msg.umad.mad); + rc =3D umad_recv(server.umad_agent.port_id, &msg.umad, &msg.um= ad_len, + SLEEP_SECS * SCALE_US); + if ((rc =3D=3D -EIO) || (rc =3D=3D -EINVAL)) { + syslog(LOG_CRIT, "Fatal error while trying to read MAD"); + } + + if (rc =3D=3D -ETIMEDOUT) { + g_hash_table_foreach_remove(server.umad_agent.commid2fd, + remove_old_comm_ids, NULL); + } + } while (rc && server.run); + + if (server.run) { + rc =3D get_fd(msg.umad.mad, &fd, &msg.hdr.sgid.global.interfac= e_id); + if (rc) { + continue; + } + + send(fd, &msg, sizeof(msg), 0); + } + } + + return NULL; +} + +static int read_and_process(int fd) +{ + int rc; + RdmaCmMuxMsg msg =3D {0}; + struct umad_hdr *hdr; + uint32_t *comm_id; + uint16_t attr_id; + + rc =3D recv(fd, &msg, sizeof(msg), 0); + + if (rc < 0 && errno !=3D EWOULDBLOCK) { + return -EIO; + } + + if (!rc) { + return -EPIPE; + } + + switch (msg.hdr.msg_type) { + case RDMACM_MUX_MSG_TYPE_REG: + rc =3D add_fd_ifid_pair(fd, msg.hdr.sgid.global.interface_id); + break; + + case RDMACM_MUX_MSG_TYPE_UNREG: + rc =3D delete_fd_ifid_pair(fd, msg.hdr.sgid.global.interface_id); + break; + + case RDMACM_MUX_MSG_TYPE_MAD: + /* If this is REQ or REP then store the pair comm_id,fd to be later + * used for other messages where gid is unknown */ + hdr =3D (struct umad_hdr *)msg.umad.mad; + attr_id =3D be16toh(hdr->attr_id); + if ((attr_id =3D=3D UMAD_CM_ATTR_REQ) || (attr_id =3D=3D UMAD_CM_A= TTR_DREQ) || + (attr_id =3D=3D UMAD_CM_ATTR_SIDR_REQ) || + (attr_id =3D=3D UMAD_CM_ATTR_REP) || (attr_id =3D=3D UMAD_CM_A= TTR_DREP)) { + comm_id =3D (uint32_t *)(msg.umad.mad + sizeof(*hdr)); + hash_tbl_save_fd_comm_id_pair(fd, *comm_id, + msg.hdr.sgid.global.interface_id= ); + } + + rc =3D umad_send(server.umad_agent.port_id, server.umad_agent.agen= t_id, + &msg.umad, msg.umad_len, 1, 0); + if (rc) { + syslog(LOG_WARNING, "Fail to send MAD message, err=3D%d", rc); + } + break; + + default: + syslog(LOG_WARNING, "Got invalid message (%d) from %d", + msg.hdr.msg_type, fd); + rc =3D RDMACM_MUX_ERR_CODE_EINVAL; + } + + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_RESP; + msg.hdr.err_code =3D rc; + rc =3D send(fd, &msg, sizeof(msg), 0); + + return rc =3D=3D sizeof(msg) ? 0 : -EPIPE; +} + +static int accept_all(void) +{ + int fd, rc =3D 0;; + + pthread_rwlock_wrlock(&server.lock); + + do { + if ((server.nfds + 1) > MAX_CLIENTS) { + syslog(LOG_WARNING, "Too many clients (%d)", server.nfds); + rc =3D -EIO; + goto out; + } + + fd =3D accept(server.fds[0].fd, NULL, NULL); + if (fd < 0) { + if (errno !=3D EWOULDBLOCK) { + syslog(LOG_WARNING, "accept() failed"); + rc =3D -EIO; + goto out; + } + break; + } + + syslog(LOG_INFO, "Client connected on socket %d\n", fd); + server.fds[server.nfds].fd =3D fd; + server.fds[server.nfds].events =3D POLLIN; + server.nfds++; + } while (fd !=3D -1); + +out: + pthread_rwlock_unlock(&server.lock); + return rc; +} + +static void compress_fds(void) +{ + int i, j; + int closed =3D 0; + + pthread_rwlock_wrlock(&server.lock); + + for (i =3D 1; i < server.nfds; i++) { + if (!server.fds[i].fd) { + closed++; + for (j =3D i; j < server.nfds; j++) { + server.fds[j].fd =3D server.fds[j + 1].fd; + } + } + } + + server.nfds -=3D closed; + + pthread_rwlock_unlock(&server.lock); +} + +static void close_fd(int idx) +{ + close(server.fds[idx].fd); + syslog(LOG_INFO, "Socket %d closed\n", server.fds[idx].fd); + hash_tbl_remove_fd_ifid_pair(server.fds[idx].fd); + server.fds[idx].fd =3D 0; +} + +static void run(void) +{ + int rc, nfds, i; + bool compress =3D false; + + syslog(LOG_INFO, "Service started"); + + while (server.run) { + rc =3D poll(server.fds, server.nfds, SLEEP_SECS * SCALE_US); + if (rc < 0) { + if (errno !=3D EINTR) { + syslog(LOG_WARNING, "poll() failed"); + } + continue; + } + + if (rc =3D=3D 0) { + continue; + } + + nfds =3D server.nfds; + for (i =3D 0; i < nfds; i++) { + if (server.fds[i].revents =3D=3D 0) { + continue; + } + + if (server.fds[i].revents !=3D POLLIN) { + if (i =3D=3D 0) { + syslog(LOG_NOTICE, "Unexpected poll() event (0x%x)\n", + server.fds[i].revents); + } else { + close_fd(i); + compress =3D true; + } + continue; + } + + if (i =3D=3D 0) { + rc =3D accept_all(); + if (rc) { + continue; + } + } else { + rc =3D read_and_process(server.fds[i].fd); + if (rc) { + close_fd(i); + compress =3D true; + } + } + } + + if (compress) { + compress =3D false; + compress_fds(); + } + } +} + +static void fini_listener(void) +{ + int i; + + if (server.fds[0].fd <=3D 0) { + return; + } + + for (i =3D server.nfds - 1; i >=3D 0; i--) { + if (server.fds[i].fd) { + close(server.fds[i].fd); + } + } + + unlink(server.args.unix_socket_path); +} + +static void fini_umad(void) +{ + if (server.umad_agent.agent_id) { + umad_unregister(server.umad_agent.port_id, server.umad_agent.agent= _id); + } + + if (server.umad_agent.port_id) { + umad_close_port(server.umad_agent.port_id); + } + + hash_tbl_free(); +} + +static void fini(void) +{ + if (server.umad_recv_thread) { + pthread_join(server.umad_recv_thread, NULL); + server.umad_recv_thread =3D 0; + } + fini_umad(); + fini_listener(); + pthread_rwlock_destroy(&server.lock); + + syslog(LOG_INFO, "Service going down"); +} + +static int init_listener(void) +{ + struct sockaddr_un sun; + int rc, on =3D 1; + + server.fds[0].fd =3D socket(AF_UNIX, SOCK_STREAM, 0); + if (server.fds[0].fd < 0) { + syslog(LOG_ALERT, "socket() failed"); + return -EIO; + } + + rc =3D setsockopt(server.fds[0].fd, SOL_SOCKET, SO_REUSEADDR, (char *)= &on, + sizeof(on)); + if (rc < 0) { + syslog(LOG_ALERT, "setsockopt() failed"); + rc =3D -EIO; + goto err; + } + + rc =3D ioctl(server.fds[0].fd, FIONBIO, (char *)&on); + if (rc < 0) { + syslog(LOG_ALERT, "ioctl() failed"); + rc =3D -EIO; + goto err; + } + + if (strlen(server.args.unix_socket_path) >=3D sizeof(sun.sun_path)) { + syslog(LOG_ALERT, + "Invalid unix_socket_path, size must be less than %ld\n", + sizeof(sun.sun_path)); + rc =3D -EINVAL; + goto err; + } + + sun.sun_family =3D AF_UNIX; + rc =3D snprintf(sun.sun_path, sizeof(sun.sun_path), "%s", + server.args.unix_socket_path); + if (rc < 0 || rc >=3D sizeof(sun.sun_path)) { + syslog(LOG_ALERT, "Could not copy unix socket path\n"); + rc =3D -EINVAL; + goto err; + } + + rc =3D bind(server.fds[0].fd, (struct sockaddr *)&sun, sizeof(sun)); + if (rc < 0) { + syslog(LOG_ALERT, "bind() failed"); + rc =3D -EIO; + goto err; + } + + rc =3D listen(server.fds[0].fd, SERVER_LISTEN_BACKLOG); + if (rc < 0) { + syslog(LOG_ALERT, "listen() failed"); + rc =3D -EIO; + goto err; + } + + server.fds[0].events =3D POLLIN; + server.nfds =3D 1; + server.run =3D true; + + return 0; + +err: + close(server.fds[0].fd); + return rc; +} + +static int init_umad(void) +{ + long method_mask[IB_USER_MAD_LONGS_PER_METHOD_MASK]; + + server.umad_agent.port_id =3D umad_open_port(server.args.rdma_dev_name, + server.args.rdma_port_num); + + if (server.umad_agent.port_id < 0) { + syslog(LOG_WARNING, "umad_open_port() failed"); + return -EIO; + } + + memset(&method_mask, 0, sizeof(method_mask)); + method_mask[0] =3D MAD_METHOD_MASK0; + server.umad_agent.agent_id =3D umad_register(server.umad_agent.port_id, + UMAD_CLASS_CM, + UMAD_SA_CLASS_VERSION, + MAD_RMPP_VERSION, method_ma= sk); + if (server.umad_agent.agent_id < 0) { + syslog(LOG_WARNING, "umad_register() failed"); + return -EIO; + } + + hash_tbl_alloc(); + + return 0; +} + +static void signal_handler(int sig, siginfo_t *siginfo, void *context) +{ + static bool warned; + + /* Prevent stop if clients are connected */ + if (server.nfds !=3D 1) { + if (!warned) { + syslog(LOG_WARNING, + "Can't stop while active client exist, resend SIGINT to= overid"); + warned =3D true; + return; + } + } + + if (sig =3D=3D SIGINT) { + server.run =3D false; + fini(); + } + + exit(0); +} + +static int init(void) +{ + int rc; + struct sigaction sig =3D {0}; + + rc =3D init_listener(); + if (rc) { + return rc; + } + + rc =3D init_umad(); + if (rc) { + return rc; + } + + pthread_rwlock_init(&server.lock, 0); + + rc =3D pthread_create(&server.umad_recv_thread, NULL, umad_recv_thread= _func, + NULL); + if (rc) { + syslog(LOG_ERR, "Fail to create UMAD receiver thread (%d)\n", rc); + return rc; + } + + sig.sa_sigaction =3D &signal_handler; + sig.sa_flags =3D SA_SIGINFO; + rc =3D sigaction(SIGINT, &sig, NULL); + if (rc < 0) { + syslog(LOG_ERR, "Fail to install SIGINT handler (%d)\n", errno); + return rc; + } + + return 0; +} + +int main(int argc, char *argv[]) +{ + int rc; + + memset(&server, 0, sizeof(server)); + + parse_args(argc, argv); + + rc =3D init(); + if (rc) { + syslog(LOG_ERR, "Fail to initialize server (%d)\n", rc); + rc =3D -EAGAIN; + goto out; + } + + run(); + +out: + fini(); + + return rc; +} diff --git a/contrib/rdmacm-mux/rdmacm-mux.h b/contrib/rdmacm-mux/rdmacm-mu= x.h new file mode 100644 index 0000000000..03508d52b2 --- /dev/null +++ b/contrib/rdmacm-mux/rdmacm-mux.h @@ -0,0 +1,56 @@ +/* + * QEMU paravirtual RDMA - rdmacm-mux declarations + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef RDMACM_MUX_H +#define RDMACM_MUX_H + +#include "linux/if.h" +#include "infiniband/verbs.h" +#include "infiniband/umad.h" +#include "rdma/rdma_user_cm.h" + +typedef enum RdmaCmMuxMsgType { + RDMACM_MUX_MSG_TYPE_REG =3D 0, + RDMACM_MUX_MSG_TYPE_UNREG =3D 1, + RDMACM_MUX_MSG_TYPE_MAD =3D 2, + RDMACM_MUX_MSG_TYPE_RESP =3D 3, +} RdmaCmMuxMsgType; + +typedef enum RdmaCmMuxErrCode { + RDMACM_MUX_ERR_CODE_OK =3D 0, + RDMACM_MUX_ERR_CODE_EINVAL =3D 1, + RDMACM_MUX_ERR_CODE_EEXIST =3D 2, + RDMACM_MUX_ERR_CODE_EACCES =3D 3, + RDMACM_MUX_ERR_CODE_ENOTFOUND =3D 4, +} RdmaCmMuxErrCode; + +typedef struct RdmaCmMuxHdr { + RdmaCmMuxMsgType msg_type; + union ibv_gid sgid; + RdmaCmMuxErrCode err_code; +} RdmaCmUHdr; + +typedef struct RdmaCmUMad { + struct ib_user_mad hdr; + char mad[RDMA_MAX_PRIVATE_DATA]; +} RdmaCmUMad; + +typedef struct RdmaCmMuxMsg { + RdmaCmUHdr hdr; + int umad_len; + RdmaCmUMad umad; +} RdmaCmMuxMsg; + +#endif --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095020637871.9146113456686; Mon, 12 Nov 2018 23:43:40 -0800 (PST) Received: from localhost ([::1]:52457 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTMJ-0001p2-FQ for importer@patchew.org; Tue, 13 Nov 2018 02:43:39 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52110) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvG-0004vl-L8 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvC-0004KF-5h for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:41 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45484) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvA-0003vk-Fb for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:37 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EwZA037874; Tue, 13 Nov 2018 07:15:17 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2nnwc0gn3g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:17 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FFt1012727 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:16 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FFv1012155; Tue, 13 Nov 2018 07:15:15 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:14 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=W1rJf68BGm5BgYz41g+2SB0TaNcyLb74gAqt4g60BaI=; b=sSojoYb2g3Fr0lc3WFyjCxgxlaIAY0icKrHp/mzTnfjHj28LghT9eBcYHgN1YE6nCSzE q0jnlxVftr8S26zK1hOAE/4S+/BHMv6y46V6+7pknUYhkHOmlUZ/sUM5UoW+Y/Bqqz6n 8wDICO6EztpYcUrz6Ja6Ky315012L93YqP3J2X0qEckRQPW7WzQEXZVSsfH6aOXI5ECi tQwr7lQKWtDzh7yJjasrjABcVXHGVp6MuHF2s4Wz4Z5Gn7wgxV0VG8gqGwREHPSNrqXR oXXYK89iUTWIrQ87OBzvw4DFTDBpi3wkhmvQfc0z2m5W7TI4zSz9RmjESQLAsNfBDpIM Lw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:14 +0200 Message-Id: <20181113071336.6242-26-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 01/23] contrib/rdmacm-mux: Add implementation of RDMA User MAD multiplexer X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" RDMA MAD kernel module (ibcm) disallow more than one MAD-agent for a given MAD class. This does not go hand-by-hand with qemu pvrdma device's requirements where each VM is MAD agent. Fix it by adding implementation of RDMA MAD multiplexer service which on one hand register as a sole MAD agent with the kernel module and on the other hand gives service to more than one VM. Design Overview: Reviewed-by: Shamir Rabinovitch ---------------- A server process is registered to UMAD framework (for this to work the rdma_cm kernel module needs to be unloaded) and creates a unix socket to listen to incoming request from clients. A client process (such as QEMU) connects to this unix socket and registers with its own GID. TX: --- When client needs to send rdma_cm MAD message it construct it the same way as without this multiplexer, i.e. creates a umad packet but this time it writes its content to the socket instead of calling umad_send(). The server, upon receiving such a message fetch local_comm_id from it so a context for this session can be maintain and relay the message to UMAD layer by calling umad_send(). RX: --- The server creates a worker thread to process incoming rdma_cm MAD messages. When an incoming message arrived (umad_recv()) the server, depending on the message type (attr_id) looks for target client by either searching in gid->fd table or in local_comm_id->fd table. With the extracted fd the server relays to incoming message to the client. Signed-off-by: Yuval Shaia --- MAINTAINERS | 1 + Makefile | 3 + Makefile.objs | 1 + contrib/rdmacm-mux/Makefile.objs | 4 + contrib/rdmacm-mux/main.c | 771 +++++++++++++++++++++++++++++++ contrib/rdmacm-mux/rdmacm-mux.h | 56 +++ 6 files changed, 836 insertions(+) create mode 100644 contrib/rdmacm-mux/Makefile.objs create mode 100644 contrib/rdmacm-mux/main.c create mode 100644 contrib/rdmacm-mux/rdmacm-mux.h diff --git a/MAINTAINERS b/MAINTAINERS index 98a1856afc..e087d58ac6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2231,6 +2231,7 @@ S: Maintained F: hw/rdma/* F: hw/rdma/vmw/* F: docs/pvrdma.txt +F: contrib/rdmacm-mux/* =20 Build and test automation ------------------------- diff --git a/Makefile b/Makefile index f2947186a4..94072776ff 100644 --- a/Makefile +++ b/Makefile @@ -418,6 +418,7 @@ dummy :=3D $(call unnest-vars,, \ elf2dmp-obj-y \ ivshmem-client-obj-y \ ivshmem-server-obj-y \ + rdmacm-mux-obj-y \ libvhost-user-obj-y \ vhost-user-scsi-obj-y \ vhost-user-blk-obj-y \ @@ -725,6 +726,8 @@ vhost-user-scsi$(EXESUF): $(vhost-user-scsi-obj-y) libv= host-user.a $(call LINK, $^) vhost-user-blk$(EXESUF): $(vhost-user-blk-obj-y) libvhost-user.a $(call LINK, $^) +rdmacm-mux$(EXESUF): $(rdmacm-mux-obj-y) $(COMMON_LDADDS) + $(call LINK, $^) =20 module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak $(call quiet-command,$(PYTHON) $< $@ \ diff --git a/Makefile.objs b/Makefile.objs index 1e1ff387d7..cc7df3ad80 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -194,6 +194,7 @@ vhost-user-scsi.o-cflags :=3D $(LIBISCSI_CFLAGS) vhost-user-scsi.o-libs :=3D $(LIBISCSI_LIBS) vhost-user-scsi-obj-y =3D contrib/vhost-user-scsi/ vhost-user-blk-obj-y =3D contrib/vhost-user-blk/ +rdmacm-mux-obj-y =3D contrib/rdmacm-mux/ =20 ###################################################################### trace-events-subdirs =3D diff --git a/contrib/rdmacm-mux/Makefile.objs b/contrib/rdmacm-mux/Makefile= .objs new file mode 100644 index 0000000000..be3eacb6f7 --- /dev/null +++ b/contrib/rdmacm-mux/Makefile.objs @@ -0,0 +1,4 @@ +ifdef CONFIG_PVRDMA +CFLAGS +=3D -libumad -Wno-format-truncation +rdmacm-mux-obj-y =3D main.o +endif diff --git a/contrib/rdmacm-mux/main.c b/contrib/rdmacm-mux/main.c new file mode 100644 index 0000000000..47cf0ac7bc --- /dev/null +++ b/contrib/rdmacm-mux/main.c @@ -0,0 +1,771 @@ +/* + * QEMU paravirtual RDMA - rdmacm-mux implementation + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "sys/poll.h" +#include "sys/ioctl.h" +#include "pthread.h" +#include "syslog.h" + +#include "infiniband/verbs.h" +#include "infiniband/umad.h" +#include "infiniband/umad_types.h" +#include "infiniband/umad_sa.h" +#include "infiniband/umad_cm.h" + +#include "rdmacm-mux.h" + +#define SCALE_US 1000 +#define COMMID_TTL 2 /* How many SCALE_US a context of MAD session is save= d */ +#define SLEEP_SECS 5 /* This is used both in poll() and thread */ +#define SERVER_LISTEN_BACKLOG 10 +#define MAX_CLIENTS 4096 +#define MAD_RMPP_VERSION 0 +#define MAD_METHOD_MASK0 0x8 + +#define IB_USER_MAD_LONGS_PER_METHOD_MASK (128 / (8 * sizeof(long))) + +#define CM_REQ_DGID_POS 80 +#define CM_SIDR_REQ_DGID_POS 44 + +/* The below can be override by command line parameter */ +#define UNIX_SOCKET_PATH "/var/run/rdmacm-mux" +#define RDMA_DEVICE "rxe0" +#define RDMA_PORT_NUM 1 + +typedef struct RdmaCmServerArgs { + char unix_socket_path[PATH_MAX]; + char rdma_dev_name[NAME_MAX]; + int rdma_port_num; +} RdmaCMServerArgs; + +typedef struct CommId2FdEntry { + int fd; + int ttl; /* Initialized to 2, decrement each timeout, entry delete whe= n 0 */ + __be64 gid_ifid; +} CommId2FdEntry; + +typedef struct RdmaCmUMadAgent { + int port_id; + int agent_id; + GHashTable *gid2fd; /* Used to find fd of a given gid */ + GHashTable *commid2fd; /* Used to find fd on of a given comm_id */ +} RdmaCmUMadAgent; + +typedef struct RdmaCmServer { + bool run; + RdmaCMServerArgs args; + struct pollfd fds[MAX_CLIENTS]; + int nfds; + RdmaCmUMadAgent umad_agent; + pthread_t umad_recv_thread; + pthread_rwlock_t lock; +} RdmaCMServer; + +static RdmaCMServer server =3D {0}; + +static void usage(const char *progname) +{ + printf("Usage: %s [OPTION]...\n" + "Start a RDMA-CM multiplexer\n" + "\n" + "\t-h Show this help\n" + "\t-s unix-socket-path Path to unix socket to listen on (defa= ult %s)\n" + "\t-d rdma-device-name Name of RDMA device to register with (= default %s)\n" + "\t-p rdma-device-port Port number of RDMA device to register= with (default %d)\n", + progname, UNIX_SOCKET_PATH, RDMA_DEVICE, RDMA_PORT_NUM); +} + +static void help(const char *progname) +{ + fprintf(stderr, "Try '%s -h' for more information.\n", progname); +} + +static void parse_args(int argc, char *argv[]) +{ + int c; + char unix_socket_path[PATH_MAX]; + + strcpy(unix_socket_path, UNIX_SOCKET_PATH); + strncpy(server.args.rdma_dev_name, RDMA_DEVICE, NAME_MAX - 1); + server.args.rdma_port_num =3D RDMA_PORT_NUM; + + while ((c =3D getopt(argc, argv, "hs:d:p:")) !=3D -1) { + switch (c) { + case 'h': + usage(argv[0]); + exit(0); + + case 's': + /* This is temporary, final name will build below */ + strncpy(unix_socket_path, optarg, PATH_MAX); + break; + + case 'd': + strncpy(server.args.rdma_dev_name, optarg, NAME_MAX - 1); + break; + + case 'p': + server.args.rdma_port_num =3D atoi(optarg); + break; + + default: + help(argv[0]); + exit(1); + } + } + + /* Build unique unix-socket file name */ + snprintf(server.args.unix_socket_path, PATH_MAX, "%s-%s-%d", + unix_socket_path, server.args.rdma_dev_name, + server.args.rdma_port_num); + + syslog(LOG_INFO, "unix_socket_path=3D%s", server.args.unix_socket_path= ); + syslog(LOG_INFO, "rdma-device-name=3D%s", server.args.rdma_dev_name); + syslog(LOG_INFO, "rdma-device-port=3D%d", server.args.rdma_port_num); +} + +static void hash_tbl_alloc(void) +{ + + server.umad_agent.gid2fd =3D g_hash_table_new_full(g_int64_hash, + g_int64_equal, + g_free, g_free); + server.umad_agent.commid2fd =3D g_hash_table_new_full(g_int_hash, + g_int_equal, + g_free, g_free); +} + +static void hash_tbl_free(void) +{ + if (server.umad_agent.commid2fd) { + g_hash_table_destroy(server.umad_agent.commid2fd); + } + if (server.umad_agent.gid2fd) { + g_hash_table_destroy(server.umad_agent.gid2fd); + } +} + + +static int _hash_tbl_search_fd_by_ifid(__be64 *gid_ifid) +{ + int *fd; + + fd =3D g_hash_table_lookup(server.umad_agent.gid2fd, gid_ifid); + if (!fd) { + /* Let's try IPv4 */ + *gid_ifid |=3D 0x00000000ffff0000; + fd =3D g_hash_table_lookup(server.umad_agent.gid2fd, gid_ifid); + } + + return fd ? *fd : 0; +} + +static int hash_tbl_search_fd_by_ifid(int *fd, __be64 *gid_ifid) +{ + pthread_rwlock_rdlock(&server.lock); + *fd =3D _hash_tbl_search_fd_by_ifid(gid_ifid); + pthread_rwlock_unlock(&server.lock); + + if (!fd) { + syslog(LOG_WARNING, "Can't find matching for ifid 0x%llx\n", *gid_= ifid); + return -ENOENT; + } + + return 0; +} + +static int hash_tbl_search_fd_by_comm_id(uint32_t comm_id, int *fd, + __be64 *gid_idid) +{ + CommId2FdEntry *fde; + + pthread_rwlock_rdlock(&server.lock); + fde =3D g_hash_table_lookup(server.umad_agent.commid2fd, &comm_id); + pthread_rwlock_unlock(&server.lock); + + if (!fde) { + syslog(LOG_WARNING, "Can't find matching for comm_id 0x%x\n", comm= _id); + return -ENOENT; + } + + *fd =3D fde->fd; + *gid_idid =3D fde->gid_ifid; + + return 0; +} + +static RdmaCmMuxErrCode add_fd_ifid_pair(int fd, __be64 gid_ifid) +{ + int fd1; + + pthread_rwlock_wrlock(&server.lock); + + fd1 =3D _hash_tbl_search_fd_by_ifid(&gid_ifid); + if (fd1) { /* record already exist - an error */ + pthread_rwlock_unlock(&server.lock); + return fd =3D=3D fd1 ? RDMACM_MUX_ERR_CODE_EEXIST : + RDMACM_MUX_ERR_CODE_EACCES; + } + + g_hash_table_insert(server.umad_agent.gid2fd, g_memdup(&gid_ifid, + sizeof(gid_ifid)), g_memdup(&fd, sizeof(fd))); + + pthread_rwlock_unlock(&server.lock); + + syslog(LOG_INFO, "0x%lx registered on socket %d", (uint64_t)gid_ifid, = fd); + + return RDMACM_MUX_ERR_CODE_OK; +} + +static RdmaCmMuxErrCode delete_fd_ifid_pair(int fd, __be64 gid_ifid) +{ + int fd1; + + pthread_rwlock_wrlock(&server.lock); + + fd1 =3D _hash_tbl_search_fd_by_ifid(&gid_ifid); + if (!fd1) { /* record not exist - an error */ + pthread_rwlock_unlock(&server.lock); + return RDMACM_MUX_ERR_CODE_ENOTFOUND; + } + + g_hash_table_remove(server.umad_agent.gid2fd, g_memdup(&gid_ifid, + sizeof(gid_ifid))); + pthread_rwlock_unlock(&server.lock); + + syslog(LOG_INFO, "0x%lx unregistered on socket %d", (uint64_t)gid_ifid= , fd); + + return RDMACM_MUX_ERR_CODE_OK; +} + +static void hash_tbl_save_fd_comm_id_pair(int fd, uint32_t comm_id, + uint64_t gid_ifid) +{ + CommId2FdEntry fde =3D {fd, COMMID_TTL, gid_ifid}; + + pthread_rwlock_wrlock(&server.lock); + g_hash_table_insert(server.umad_agent.commid2fd, + g_memdup(&comm_id, sizeof(comm_id)), + g_memdup(&fde, sizeof(fde))); + pthread_rwlock_unlock(&server.lock); +} + +static gboolean remove_old_comm_ids(gpointer key, gpointer value, + gpointer user_data) +{ + CommId2FdEntry *fde =3D (CommId2FdEntry *)value; + + return !fde->ttl--; +} + +static gboolean remove_entry_from_gid2fd(gpointer key, gpointer value, + gpointer user_data) +{ + if (*(int *)value =3D=3D *(int *)user_data) { + syslog(LOG_INFO, "0x%lx unregistered on socket %d", *(uint64_t *)k= ey, + *(int *)value); + return true; + } + + return false; +} + +static void hash_tbl_remove_fd_ifid_pair(int fd) +{ + pthread_rwlock_wrlock(&server.lock); + g_hash_table_foreach_remove(server.umad_agent.gid2fd, + remove_entry_from_gid2fd, (gpointer)&fd); + pthread_rwlock_unlock(&server.lock); +} + +static int get_fd(const char *mad, int *fd, __be64 *gid_ifid) +{ + struct umad_hdr *hdr =3D (struct umad_hdr *)mad; + char *data =3D (char *)hdr + sizeof(*hdr); + int32_t comm_id; + uint16_t attr_id =3D be16toh(hdr->attr_id); + int rc =3D 0; + + switch (attr_id) { + case UMAD_CM_ATTR_REQ: + memcpy(gid_ifid, data + CM_REQ_DGID_POS, sizeof(*gid_ifid)); + rc =3D hash_tbl_search_fd_by_ifid(fd, gid_ifid); + break; + + case UMAD_CM_ATTR_SIDR_REQ: + memcpy(gid_ifid, data + CM_SIDR_REQ_DGID_POS, sizeof(*gid_ifid)); + rc =3D hash_tbl_search_fd_by_ifid(fd, gid_ifid); + break; + + case UMAD_CM_ATTR_REP: + /* Fall through */ + case UMAD_CM_ATTR_REJ: + /* Fall through */ + case UMAD_CM_ATTR_DREQ: + /* Fall through */ + case UMAD_CM_ATTR_DREP: + /* Fall through */ + case UMAD_CM_ATTR_RTU: + data +=3D sizeof(comm_id); + /* Fall through */ + case UMAD_CM_ATTR_SIDR_REP: + memcpy(&comm_id, data, sizeof(comm_id)); + if (comm_id) { + rc =3D hash_tbl_search_fd_by_comm_id(comm_id, fd, gid_ifid); + } + break; + + default: + rc =3D -EINVAL; + syslog(LOG_WARNING, "Unsupported attr_id 0x%x\n", attr_id); + } + + return rc; +} + +static void *umad_recv_thread_func(void *args) +{ + int rc; + RdmaCmMuxMsg msg =3D {0}; + int fd =3D -2; + + while (server.run) { + do { + msg.umad_len =3D sizeof(msg.umad.mad); + rc =3D umad_recv(server.umad_agent.port_id, &msg.umad, &msg.um= ad_len, + SLEEP_SECS * SCALE_US); + if ((rc =3D=3D -EIO) || (rc =3D=3D -EINVAL)) { + syslog(LOG_CRIT, "Fatal error while trying to read MAD"); + } + + if (rc =3D=3D -ETIMEDOUT) { + g_hash_table_foreach_remove(server.umad_agent.commid2fd, + remove_old_comm_ids, NULL); + } + } while (rc && server.run); + + if (server.run) { + rc =3D get_fd(msg.umad.mad, &fd, &msg.hdr.sgid.global.interfac= e_id); + if (rc) { + continue; + } + + send(fd, &msg, sizeof(msg), 0); + } + } + + return NULL; +} + +static int read_and_process(int fd) +{ + int rc; + RdmaCmMuxMsg msg =3D {0}; + struct umad_hdr *hdr; + uint32_t *comm_id; + uint16_t attr_id; + + rc =3D recv(fd, &msg, sizeof(msg), 0); + + if (rc < 0 && errno !=3D EWOULDBLOCK) { + return -EIO; + } + + if (!rc) { + return -EPIPE; + } + + switch (msg.hdr.msg_type) { + case RDMACM_MUX_MSG_TYPE_REG: + rc =3D add_fd_ifid_pair(fd, msg.hdr.sgid.global.interface_id); + break; + + case RDMACM_MUX_MSG_TYPE_UNREG: + rc =3D delete_fd_ifid_pair(fd, msg.hdr.sgid.global.interface_id); + break; + + case RDMACM_MUX_MSG_TYPE_MAD: + /* If this is REQ or REP then store the pair comm_id,fd to be later + * used for other messages where gid is unknown */ + hdr =3D (struct umad_hdr *)msg.umad.mad; + attr_id =3D be16toh(hdr->attr_id); + if ((attr_id =3D=3D UMAD_CM_ATTR_REQ) || (attr_id =3D=3D UMAD_CM_A= TTR_DREQ) || + (attr_id =3D=3D UMAD_CM_ATTR_SIDR_REQ) || + (attr_id =3D=3D UMAD_CM_ATTR_REP) || (attr_id =3D=3D UMAD_CM_A= TTR_DREP)) { + comm_id =3D (uint32_t *)(msg.umad.mad + sizeof(*hdr)); + hash_tbl_save_fd_comm_id_pair(fd, *comm_id, + msg.hdr.sgid.global.interface_id= ); + } + + rc =3D umad_send(server.umad_agent.port_id, server.umad_agent.agen= t_id, + &msg.umad, msg.umad_len, 1, 0); + if (rc) { + syslog(LOG_WARNING, "Fail to send MAD message, err=3D%d", rc); + } + break; + + default: + syslog(LOG_WARNING, "Got invalid message (%d) from %d", + msg.hdr.msg_type, fd); + rc =3D RDMACM_MUX_ERR_CODE_EINVAL; + } + + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_RESP; + msg.hdr.err_code =3D rc; + rc =3D send(fd, &msg, sizeof(msg), 0); + + return rc =3D=3D sizeof(msg) ? 0 : -EPIPE; +} + +static int accept_all(void) +{ + int fd, rc =3D 0;; + + pthread_rwlock_wrlock(&server.lock); + + do { + if ((server.nfds + 1) > MAX_CLIENTS) { + syslog(LOG_WARNING, "Too many clients (%d)", server.nfds); + rc =3D -EIO; + goto out; + } + + fd =3D accept(server.fds[0].fd, NULL, NULL); + if (fd < 0) { + if (errno !=3D EWOULDBLOCK) { + syslog(LOG_WARNING, "accept() failed"); + rc =3D -EIO; + goto out; + } + break; + } + + syslog(LOG_INFO, "Client connected on socket %d\n", fd); + server.fds[server.nfds].fd =3D fd; + server.fds[server.nfds].events =3D POLLIN; + server.nfds++; + } while (fd !=3D -1); + +out: + pthread_rwlock_unlock(&server.lock); + return rc; +} + +static void compress_fds(void) +{ + int i, j; + int closed =3D 0; + + pthread_rwlock_wrlock(&server.lock); + + for (i =3D 1; i < server.nfds; i++) { + if (!server.fds[i].fd) { + closed++; + for (j =3D i; j < server.nfds; j++) { + server.fds[j].fd =3D server.fds[j + 1].fd; + } + } + } + + server.nfds -=3D closed; + + pthread_rwlock_unlock(&server.lock); +} + +static void close_fd(int idx) +{ + close(server.fds[idx].fd); + syslog(LOG_INFO, "Socket %d closed\n", server.fds[idx].fd); + hash_tbl_remove_fd_ifid_pair(server.fds[idx].fd); + server.fds[idx].fd =3D 0; +} + +static void run(void) +{ + int rc, nfds, i; + bool compress =3D false; + + syslog(LOG_INFO, "Service started"); + + while (server.run) { + rc =3D poll(server.fds, server.nfds, SLEEP_SECS * SCALE_US); + if (rc < 0) { + if (errno !=3D EINTR) { + syslog(LOG_WARNING, "poll() failed"); + } + continue; + } + + if (rc =3D=3D 0) { + continue; + } + + nfds =3D server.nfds; + for (i =3D 0; i < nfds; i++) { + if (server.fds[i].revents =3D=3D 0) { + continue; + } + + if (server.fds[i].revents !=3D POLLIN) { + if (i =3D=3D 0) { + syslog(LOG_NOTICE, "Unexpected poll() event (0x%x)\n", + server.fds[i].revents); + } else { + close_fd(i); + compress =3D true; + } + continue; + } + + if (i =3D=3D 0) { + rc =3D accept_all(); + if (rc) { + continue; + } + } else { + rc =3D read_and_process(server.fds[i].fd); + if (rc) { + close_fd(i); + compress =3D true; + } + } + } + + if (compress) { + compress =3D false; + compress_fds(); + } + } +} + +static void fini_listener(void) +{ + int i; + + if (server.fds[0].fd <=3D 0) { + return; + } + + for (i =3D server.nfds - 1; i >=3D 0; i--) { + if (server.fds[i].fd) { + close(server.fds[i].fd); + } + } + + unlink(server.args.unix_socket_path); +} + +static void fini_umad(void) +{ + if (server.umad_agent.agent_id) { + umad_unregister(server.umad_agent.port_id, server.umad_agent.agent= _id); + } + + if (server.umad_agent.port_id) { + umad_close_port(server.umad_agent.port_id); + } + + hash_tbl_free(); +} + +static void fini(void) +{ + if (server.umad_recv_thread) { + pthread_join(server.umad_recv_thread, NULL); + server.umad_recv_thread =3D 0; + } + fini_umad(); + fini_listener(); + pthread_rwlock_destroy(&server.lock); + + syslog(LOG_INFO, "Service going down"); +} + +static int init_listener(void) +{ + struct sockaddr_un sun; + int rc, on =3D 1; + + server.fds[0].fd =3D socket(AF_UNIX, SOCK_STREAM, 0); + if (server.fds[0].fd < 0) { + syslog(LOG_ALERT, "socket() failed"); + return -EIO; + } + + rc =3D setsockopt(server.fds[0].fd, SOL_SOCKET, SO_REUSEADDR, (char *)= &on, + sizeof(on)); + if (rc < 0) { + syslog(LOG_ALERT, "setsockopt() failed"); + rc =3D -EIO; + goto err; + } + + rc =3D ioctl(server.fds[0].fd, FIONBIO, (char *)&on); + if (rc < 0) { + syslog(LOG_ALERT, "ioctl() failed"); + rc =3D -EIO; + goto err; + } + + if (strlen(server.args.unix_socket_path) >=3D sizeof(sun.sun_path)) { + syslog(LOG_ALERT, + "Invalid unix_socket_path, size must be less than %ld\n", + sizeof(sun.sun_path)); + rc =3D -EINVAL; + goto err; + } + + sun.sun_family =3D AF_UNIX; + rc =3D snprintf(sun.sun_path, sizeof(sun.sun_path), "%s", + server.args.unix_socket_path); + if (rc < 0 || rc >=3D sizeof(sun.sun_path)) { + syslog(LOG_ALERT, "Could not copy unix socket path\n"); + rc =3D -EINVAL; + goto err; + } + + rc =3D bind(server.fds[0].fd, (struct sockaddr *)&sun, sizeof(sun)); + if (rc < 0) { + syslog(LOG_ALERT, "bind() failed"); + rc =3D -EIO; + goto err; + } + + rc =3D listen(server.fds[0].fd, SERVER_LISTEN_BACKLOG); + if (rc < 0) { + syslog(LOG_ALERT, "listen() failed"); + rc =3D -EIO; + goto err; + } + + server.fds[0].events =3D POLLIN; + server.nfds =3D 1; + server.run =3D true; + + return 0; + +err: + close(server.fds[0].fd); + return rc; +} + +static int init_umad(void) +{ + long method_mask[IB_USER_MAD_LONGS_PER_METHOD_MASK]; + + server.umad_agent.port_id =3D umad_open_port(server.args.rdma_dev_name, + server.args.rdma_port_num); + + if (server.umad_agent.port_id < 0) { + syslog(LOG_WARNING, "umad_open_port() failed"); + return -EIO; + } + + memset(&method_mask, 0, sizeof(method_mask)); + method_mask[0] =3D MAD_METHOD_MASK0; + server.umad_agent.agent_id =3D umad_register(server.umad_agent.port_id, + UMAD_CLASS_CM, + UMAD_SA_CLASS_VERSION, + MAD_RMPP_VERSION, method_ma= sk); + if (server.umad_agent.agent_id < 0) { + syslog(LOG_WARNING, "umad_register() failed"); + return -EIO; + } + + hash_tbl_alloc(); + + return 0; +} + +static void signal_handler(int sig, siginfo_t *siginfo, void *context) +{ + static bool warned; + + /* Prevent stop if clients are connected */ + if (server.nfds !=3D 1) { + if (!warned) { + syslog(LOG_WARNING, + "Can't stop while active client exist, resend SIGINT to= overid"); + warned =3D true; + return; + } + } + + if (sig =3D=3D SIGINT) { + server.run =3D false; + fini(); + } + + exit(0); +} + +static int init(void) +{ + int rc; + struct sigaction sig =3D {0}; + + rc =3D init_listener(); + if (rc) { + return rc; + } + + rc =3D init_umad(); + if (rc) { + return rc; + } + + pthread_rwlock_init(&server.lock, 0); + + rc =3D pthread_create(&server.umad_recv_thread, NULL, umad_recv_thread= _func, + NULL); + if (rc) { + syslog(LOG_ERR, "Fail to create UMAD receiver thread (%d)\n", rc); + return rc; + } + + sig.sa_sigaction =3D &signal_handler; + sig.sa_flags =3D SA_SIGINFO; + rc =3D sigaction(SIGINT, &sig, NULL); + if (rc < 0) { + syslog(LOG_ERR, "Fail to install SIGINT handler (%d)\n", errno); + return rc; + } + + return 0; +} + +int main(int argc, char *argv[]) +{ + int rc; + + memset(&server, 0, sizeof(server)); + + parse_args(argc, argv); + + rc =3D init(); + if (rc) { + syslog(LOG_ERR, "Fail to initialize server (%d)\n", rc); + rc =3D -EAGAIN; + goto out; + } + + run(); + +out: + fini(); + + return rc; +} diff --git a/contrib/rdmacm-mux/rdmacm-mux.h b/contrib/rdmacm-mux/rdmacm-mu= x.h new file mode 100644 index 0000000000..03508d52b2 --- /dev/null +++ b/contrib/rdmacm-mux/rdmacm-mux.h @@ -0,0 +1,56 @@ +/* + * QEMU paravirtual RDMA - rdmacm-mux declarations + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef RDMACM_MUX_H +#define RDMACM_MUX_H + +#include "linux/if.h" +#include "infiniband/verbs.h" +#include "infiniband/umad.h" +#include "rdma/rdma_user_cm.h" + +typedef enum RdmaCmMuxMsgType { + RDMACM_MUX_MSG_TYPE_REG =3D 0, + RDMACM_MUX_MSG_TYPE_UNREG =3D 1, + RDMACM_MUX_MSG_TYPE_MAD =3D 2, + RDMACM_MUX_MSG_TYPE_RESP =3D 3, +} RdmaCmMuxMsgType; + +typedef enum RdmaCmMuxErrCode { + RDMACM_MUX_ERR_CODE_OK =3D 0, + RDMACM_MUX_ERR_CODE_EINVAL =3D 1, + RDMACM_MUX_ERR_CODE_EEXIST =3D 2, + RDMACM_MUX_ERR_CODE_EACCES =3D 3, + RDMACM_MUX_ERR_CODE_ENOTFOUND =3D 4, +} RdmaCmMuxErrCode; + +typedef struct RdmaCmMuxHdr { + RdmaCmMuxMsgType msg_type; + union ibv_gid sgid; + RdmaCmMuxErrCode err_code; +} RdmaCmUHdr; + +typedef struct RdmaCmUMad { + struct ib_user_mad hdr; + char mad[RDMA_MAX_PRIVATE_DATA]; +} RdmaCmUMad; + +typedef struct RdmaCmMuxMsg { + RdmaCmUHdr hdr; + int umad_len; + RdmaCmUMad umad; +} RdmaCmMuxMsg; + +#endif --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093386917148.71484141334383; Mon, 12 Nov 2018 23:16:26 -0800 (PST) Received: from localhost ([::1]:52319 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvv-0004t7-QB for importer@patchew.org; Tue, 13 Nov 2018 02:16:25 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51565) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMStn-0003fw-7C for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMStk-0002TJ-EE for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:11 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:59240) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMStk-0002SU-7A for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:08 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DxvV046016; Tue, 13 Nov 2018 07:14:06 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp2120.oracle.com with ESMTP id 2nnw6egpmk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:06 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7E51g008955 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:05 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7E50v017436; Tue, 13 Nov 2018 07:14:05 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:05 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=p7GSCRmh+X1chYNsxs7+qk/BG719GzUY0u8K8adGsjo=; b=u7E802QojoFNY3u/ys5NMVbKmWjWNjpObV6GeyK5oXU08cyrNzomppgt2Pv82pOePtaj N24B3Z6XiDKAoh8M72DFYGs77n0legOGWlzR5Q94c+Zy3tYvRFps4G85YFA7p2LgjAeu q/lnjW9ELWNkvNmmVzn34IYFUy+3UU4Ei3bOcDdc5iDLUzMekgKiloXkYiyAF9tCmhe/ XJAw35MsRMZ+FwG9wnyvUnXgqXu2nzmhMvCyh70M+OAqiF7qXo2pTlwIrA4TvbhEnySm VjcrIRg9kjSC4hfOfMmVLgupxq+lFFgq4EDaxEbFiz+c8P8X7OhV+0D1YyfcN4JDnCwT 2Q== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:51 +0200 Message-Id: <20181113071336.6242-3-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=548 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 02/23] hw/rdma: Add ability to force notification without re-arm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Upon completion of incoming packet the device pushes CQE to driver's RX ring and notify the driver (msix). While for data-path incoming packets the driver needs the ability to control whether it wished to receive interrupts or not, for control-path packets such as incoming MAD the driver needs to be notified anyway, it even do not need to re-arm the notification bit. Enhance the notification field to support this. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 12 ++++++++++-- hw/rdma/rdma_rm_defs.h | 8 +++++++- hw/rdma/vmw/pvrdma_qp_ops.c | 6 ++++-- 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 8d59a42cd1..4f10fcabcc 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -263,7 +263,7 @@ int rdma_rm_alloc_cq(RdmaDeviceResources *dev_res, Rdma= BackendDev *backend_dev, } =20 cq->opaque =3D opaque; - cq->notify =3D false; + cq->notify =3D CNT_CLEAR; =20 rc =3D rdma_backend_create_cq(backend_dev, &cq->backend_cq, cqe); if (rc) { @@ -291,7 +291,10 @@ void rdma_rm_req_notify_cq(RdmaDeviceResources *dev_re= s, uint32_t cq_handle, return; } =20 - cq->notify =3D notify; + if (cq->notify !=3D CNT_SET) { + cq->notify =3D notify ? CNT_ARM : CNT_CLEAR; + } + pr_dbg("notify=3D%d\n", cq->notify); } =20 @@ -349,6 +352,11 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uin= t32_t pd_handle, return -EINVAL; } =20 + if (qp_type =3D=3D IBV_QPT_GSI) { + scq->notify =3D CNT_SET; + rcq->notify =3D CNT_SET; + } + qp =3D res_tbl_alloc(&dev_res->qp_tbl, &rm_qpn); if (!qp) { return -ENOMEM; diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 7228151239..9b399063d3 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -49,10 +49,16 @@ typedef struct RdmaRmPD { uint32_t ctx_handle; } RdmaRmPD; =20 +typedef enum CQNotificationType { + CNT_CLEAR, + CNT_ARM, + CNT_SET, +} CQNotificationType; + typedef struct RdmaRmCQ { RdmaBackendCQ backend_cq; void *opaque; - bool notify; + CQNotificationType notify; } RdmaRmCQ; =20 /* MR (DMA region) */ diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index c668afd0ed..762700a205 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -89,8 +89,10 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_h= andle, pvrdma_ring_write_inc(&dev->dsr_info.cq); =20 pr_dbg("cq->notify=3D%d\n", cq->notify); - if (cq->notify) { - cq->notify =3D false; + if (cq->notify !=3D CNT_CLEAR) { + if (cq->notify =3D=3D CNT_ARM) { + cq->notify =3D CNT_CLEAR; + } post_interrupt(dev, INTR_VEC_CMD_COMPLETION_Q); } =20 --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094693153447.0338623932412; Mon, 12 Nov 2018 23:38:13 -0800 (PST) Received: from localhost ([::1]:52429 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTH1-0005za-Vf for importer@patchew.org; Tue, 13 Nov 2018 02:38:12 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52198) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvJ-0004xB-N5 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvG-0004Pg-Pl for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:45 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60198) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvF-0003xu-El for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:42 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DqUf045984; Tue, 13 Nov 2018 07:15:18 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2120.oracle.com with ESMTP id 2nnw6egps8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:18 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FIx5012825 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:18 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FIME028523; Tue, 13 Nov 2018 07:15:18 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:18 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=p7GSCRmh+X1chYNsxs7+qk/BG719GzUY0u8K8adGsjo=; b=35wdDRSzsngAQOK257s9Q7ABQFHdkHHxpK+9A1MvcBKl/A9Z6eHp4MNQ8lGY55/SQ2Oo 9E0AptWTXdLNpeyE8iFL1oLqyNLeKC1cN3OFsF8KQlKaffAoi4ZU8Ehi7tyhUWMofAS2 MBY9lJDRgTdOPTTW3gY9lVlNiHDjppIo4dY89fmK/61TKO5Ls/Rfp3Rimp2eQ3yW1GOo ncA5RTQEMN1cA4mPWySCfXyFzzYeZO7zlTa2rRi0ejd1MceoxcLOtYtWwu+w52kpJRlL DVen9MplyVfWllRK8ckl5WJHrXz4509zpTfRYqeUbTwDjkOd6L0Ky6Jvr0kPijEsQ0O2 bQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:15 +0200 Message-Id: <20181113071336.6242-27-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=548 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 02/23] hw/rdma: Add ability to force notification without re-arm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Upon completion of incoming packet the device pushes CQE to driver's RX ring and notify the driver (msix). While for data-path incoming packets the driver needs the ability to control whether it wished to receive interrupts or not, for control-path packets such as incoming MAD the driver needs to be notified anyway, it even do not need to re-arm the notification bit. Enhance the notification field to support this. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 12 ++++++++++-- hw/rdma/rdma_rm_defs.h | 8 +++++++- hw/rdma/vmw/pvrdma_qp_ops.c | 6 ++++-- 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 8d59a42cd1..4f10fcabcc 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -263,7 +263,7 @@ int rdma_rm_alloc_cq(RdmaDeviceResources *dev_res, Rdma= BackendDev *backend_dev, } =20 cq->opaque =3D opaque; - cq->notify =3D false; + cq->notify =3D CNT_CLEAR; =20 rc =3D rdma_backend_create_cq(backend_dev, &cq->backend_cq, cqe); if (rc) { @@ -291,7 +291,10 @@ void rdma_rm_req_notify_cq(RdmaDeviceResources *dev_re= s, uint32_t cq_handle, return; } =20 - cq->notify =3D notify; + if (cq->notify !=3D CNT_SET) { + cq->notify =3D notify ? CNT_ARM : CNT_CLEAR; + } + pr_dbg("notify=3D%d\n", cq->notify); } =20 @@ -349,6 +352,11 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uin= t32_t pd_handle, return -EINVAL; } =20 + if (qp_type =3D=3D IBV_QPT_GSI) { + scq->notify =3D CNT_SET; + rcq->notify =3D CNT_SET; + } + qp =3D res_tbl_alloc(&dev_res->qp_tbl, &rm_qpn); if (!qp) { return -ENOMEM; diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 7228151239..9b399063d3 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -49,10 +49,16 @@ typedef struct RdmaRmPD { uint32_t ctx_handle; } RdmaRmPD; =20 +typedef enum CQNotificationType { + CNT_CLEAR, + CNT_ARM, + CNT_SET, +} CQNotificationType; + typedef struct RdmaRmCQ { RdmaBackendCQ backend_cq; void *opaque; - bool notify; + CQNotificationType notify; } RdmaRmCQ; =20 /* MR (DMA region) */ diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index c668afd0ed..762700a205 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -89,8 +89,10 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_h= andle, pvrdma_ring_write_inc(&dev->dsr_info.cq); =20 pr_dbg("cq->notify=3D%d\n", cq->notify); - if (cq->notify) { - cq->notify =3D false; + if (cq->notify !=3D CNT_CLEAR) { + if (cq->notify =3D=3D CNT_ARM) { + cq->notify =3D CNT_CLEAR; + } post_interrupt(dev, INTR_VEC_CMD_COMPLETION_Q); } =20 --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094576177502.1176701296332; Mon, 12 Nov 2018 23:36:16 -0800 (PST) Received: from localhost ([::1]:52420 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTF8-0004aB-NL for importer@patchew.org; Tue, 13 Nov 2018 02:36:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52108) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvG-0004vj-L5 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvC-0004KT-8T for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:41 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44238) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvB-000417-Bu for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:38 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Dlma078853; Tue, 13 Nov 2018 07:15:22 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nnwg18mb0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:22 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FLEY017921 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:21 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FLWP028545; Tue, 13 Nov 2018 07:15:21 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=zifS50FkBv15SH/Pk4NuMSiO5SRrHGjgTr4Z+Yu5xoA=; b=osBv6fZGoWcUnmM6Sfm64JJBfNGDou7aLij6ptwhpGokXKjGNn1CbesO8ye6McfjKhfs mW6iXkGbKwts3FvfVYbxJu/S5lTNsar+WVXRKq32ciCX3gEoTdDifvHgHTE+lC2ZB7VI 88QA9Iy/1rabn5FRU82XUhEJQmXdCGnZP6K78gNzZTAmWvBRVHRvSch4qc3DF+2ng73K ID4Einp7G5TIidPdYi+pAGtPMM6s1aY1PDIddlUIoUkG/p9FJhJXoDbeOzvewxpQY1c0 Llf2g3VYLxduWWk44zgHvJ4ycgP/wI37z7q6dRNBCdJuzrPlE5aQNbCstXhwvZih8AY7 Og== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:16 +0200 Message-Id: <20181113071336.6242-28-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=943 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 03/23] hw/rdma: Return qpn 1 if ibqp is NULL X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Device is not supporting QP0, only QP1. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_backend.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 86e8fe8ab6..3ccc9a2494 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -33,7 +33,7 @@ static inline union ibv_gid *rdma_backend_gid(RdmaBackend= Dev *dev) =20 static inline uint32_t rdma_backend_qpn(const RdmaBackendQP *qp) { - return qp->ibqp ? qp->ibqp->qp_num : 0; + return qp->ibqp ? qp->ibqp->qp_num : 1; } =20 static inline uint32_t rdma_backend_mr_lkey(const RdmaBackendMR *mr) --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093579441950.7816665212054; Mon, 12 Nov 2018 23:19:39 -0800 (PST) Received: from localhost ([::1]:52331 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSz4-0007Oy-7d for importer@patchew.org; Tue, 13 Nov 2018 02:19:38 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51615) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSu2-0003pb-4l for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMStu-0002bu-NO for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:23 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:43464) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMStu-0002ZT-00 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:18 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EF0J079262; Tue, 13 Nov 2018 07:14:15 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nnwg18m74-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:15 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7E9AB019838 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:09 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7E8rq011544; Tue, 13 Nov 2018 07:14:08 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:08 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=zifS50FkBv15SH/Pk4NuMSiO5SRrHGjgTr4Z+Yu5xoA=; b=qLzy9nBDzEQp3IT8vLZIhGzR00t5qkU3uk4JN88lS/Uq2T+siyMcQGd75tjQHKdHN/Pe KBmyUp8uhtxDT78KeshNCV2J3lc8Z36KLAvsKT/ztJJ0AaHhuCNKEsZlno20hokaOlIL h4AdqqGzu5XJeyvUTX1fpyJR4Pq4gzwxoSgjp3xw0SXVEmeGA3L/y/dG8jnW/wEKYS4X cE2uM3M9cQECbCcXO2ylwb9BEprGOhyPN36qmpRu91Me99HczJe66RtybU60RixcaKAs yhAC3xxvhi1cQvPDYZGcEVfNShkOnFsZxuCTucPxnnW5pyEuyKVYzgu80ysgFlHoL8ha Ww== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:52 +0200 Message-Id: <20181113071336.6242-4-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=943 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 03/23] hw/rdma: Return qpn 1 if ibqp is NULL X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Device is not supporting QP0, only QP1. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 86e8fe8ab6..3ccc9a2494 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -33,7 +33,7 @@ static inline union ibv_gid *rdma_backend_gid(RdmaBackend= Dev *dev) =20 static inline uint32_t rdma_backend_qpn(const RdmaBackendQP *qp) { - return qp->ibqp ? qp->ibqp->qp_num : 0; + return qp->ibqp ? qp->ibqp->qp_num : 1; } =20 static inline uint32_t rdma_backend_mr_lkey(const RdmaBackendMR *mr) --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093404632777.0081806386052; Mon, 12 Nov 2018 23:16:44 -0800 (PST) Received: from localhost ([::1]:52321 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSwF-00057r-BV for importer@patchew.org; Tue, 13 Nov 2018 02:16:43 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51608) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSu0-0003o2-8G for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMStt-0002b2-Oh for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:20 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:44610) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMStr-0002YC-A9 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:16 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EDBQ037425; Tue, 13 Nov 2018 07:14:13 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2nnwc0gmyb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:13 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7ECHg019906 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:12 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EBCU011564; Tue, 13 Nov 2018 07:14:11 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:11 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=6lUPD/8fm9Ml8bbOYnOvpGwtC51teMm7eBve8UQAC9Q=; b=z/CgwqUIir9HWSqix7aHwGQ7hrEB/nMes+YXNSyPJa8C5UdAQXSfzLNBRZtKngRwiiFH DfDE+Nttg0lIDf7M3EL/LmkB5hkgSMd3GZ3u2zX9xwLp26lMLrXNfXdKXUMr82IyQOg1 EGoP/1dlAPghpzqPJZc7mEdHZogOA8E8gncb0PSpApUIDJWmP+3DOjApwYF+ByYAebxZ rMfDVCGvfbEJrPLu96YhAoIpEf5CSyB/myXddFKvwZRXpo/hETEa3v4IjSL7q1aRaw5U mf0LHYktTKrleJGpQ2s+rkJDTCSs56mVUyxiDfeiGj1jkDQaFRpfHa3jTQUSjknfaeqO aQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:53 +0200 Message-Id: <20181113071336.6242-5-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=558 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 04/23] hw/rdma: Abort send-op if fail to create addr handler X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Function create_ah might return NULL, let's exit with an error. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index d7a4bbd91f..1e148398a2 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -338,6 +338,10 @@ void rdma_backend_post_send(RdmaBackendDev *backend_de= v, if (qp_type =3D=3D IBV_QPT_UD) { wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, backend_dev->backend_gid_idx, dgid); + if (!wr.wr.ud.ah) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + goto out_dealloc_cqe_ctx; + } wr.wr.ud.remote_qpn =3D dqpn; wr.wr.ud.remote_qkey =3D dqkey; } --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094890836804.8055747313944; Mon, 12 Nov 2018 23:41:30 -0800 (PST) Received: from localhost ([::1]:52447 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTKD-00006U-JP for importer@patchew.org; Tue, 13 Nov 2018 02:41:29 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52169) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvI-0004wB-JN for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvH-0004QA-40 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:44 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44270) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvG-000469-PL for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:42 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DlIZ078852; Tue, 13 Nov 2018 07:15:25 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nnwg18mb5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:25 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FOU2017985 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:24 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7FOI1030749; Tue, 13 Nov 2018 07:15:24 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=6lUPD/8fm9Ml8bbOYnOvpGwtC51teMm7eBve8UQAC9Q=; b=El9+W2HvvKaGdaAv9RT27PFnlsAva0YQRbRv5Zi5pQBexEJWRdTbKDmoqbj4mxaHzA3L 6XjspHV4rH52Yt+rHBxEDs2cGcg8LG2AhuHsm3zhlPjDUjUfSgFhqaaNITWyWrphoRwq k9SCzCHg37tF45xaBBnXJTeek/1k720iHrG9Ka0L2+0VOIZjnYAA6H+OPcSDVfcVehYB mX6wkO+RiDm2gISwptIPKVM3yLjHUjwb8vQb87R/HmzTO3nb5A0jkhRdYUQm3V6Pga8X 4+ssNfLm+zD+DtL+XzEAnH3mBF9jDXdZhe0HALuau/xK+fGDBUKgsVIOuECRNu6xhLGE uw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:17 +0200 Message-Id: <20181113071336.6242-29-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=558 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 04/23] hw/rdma: Abort send-op if fail to create addr handler X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Function create_ah might return NULL, let's exit with an error. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index d7a4bbd91f..1e148398a2 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -338,6 +338,10 @@ void rdma_backend_post_send(RdmaBackendDev *backend_de= v, if (qp_type =3D=3D IBV_QPT_UD) { wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, backend_dev->backend_gid_idx, dgid); + if (!wr.wr.ud.ah) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + goto out_dealloc_cqe_ctx; + } wr.wr.ud.remote_qpn =3D dqpn; wr.wr.ud.remote_qkey =3D dqkey; } --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094843726906.079954466503; Mon, 12 Nov 2018 23:40:43 -0800 (PST) Received: from localhost ([::1]:52440 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTJL-0007mH-TY for importer@patchew.org; Tue, 13 Nov 2018 02:40:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52138) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvH-0004vn-GE for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvE-0004Mt-Mh for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:43 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44290) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvD-0004Ak-VB for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:40 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Do82078867; Tue, 13 Nov 2018 07:15:28 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nnwg18mb9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:28 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FRCd023108 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:27 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FRKa028567; Tue, 13 Nov 2018 07:15:27 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:27 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=rOHY0FuAgAXuH+Z3jUp0X6wXHpoLklkNJWEZttnsIfA=; b=RqQgQsFB3NLysX5Va40GJI3I2pk4hJS4Obn0owrvIy6lU0KXAnPCXD0jLBoQspM0ofkW wOHEPSM9H5IS2+0/1vwrktn1W0A176Zvae36xm7zz0cogLlAxegOEKIGgiqzcVWmdOmZ yERPfMmwRKOaHnubdztaD7bin6Ez4LUskVAyGo9JgnYBk8wUl5rR6dV9TQaH0wi0odSR +PzEjKTLIySdNmurmkTh1r1Jaj2FKh5T51PklHSw5VDbtJapSUUYzh3ls4WdI8ehdGzB VCCwWo1n1LG6QLGhvU6djLbKiqaKvWenWgjWBrXtgmVYMKWvlDn2rC4OvT9YHPHjVW3U fw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:18 +0200 Message-Id: <20181113071336.6242-30-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 05/23] hw/rdma: Add support for MAD packets X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" MAD (Management Datagram) packets are widely used by various modules both in kernel and in user space for example the rdma_* API which is used to create and maintain "connection" layer on top of RDMA uses several types of MAD packets. To support MAD packets the device uses an external utility (contrib/rdmacm-mux) to relay packets from and to the guest driver. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_backend.c | 263 +++++++++++++++++++++++++++++++++++- hw/rdma/rdma_backend.h | 4 +- hw/rdma/rdma_backend_defs.h | 10 +- hw/rdma/vmw/pvrdma.h | 2 + hw/rdma/vmw/pvrdma_main.c | 4 +- 5 files changed, 273 insertions(+), 10 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 1e148398a2..3eb0099f8d 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -16,8 +16,13 @@ #include "qemu/osdep.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" =20 #include +#include +#include +#include =20 #include "trace.h" #include "rdma_utils.h" @@ -33,16 +38,25 @@ #define VENDOR_ERR_MAD_SEND 0x206 #define VENDOR_ERR_INVLKEY 0x207 #define VENDOR_ERR_MR_SMALL 0x208 +#define VENDOR_ERR_INV_MAD_BUFF 0x209 +#define VENDOR_ERR_INV_NUM_SGE 0x210 =20 #define THR_NAME_LEN 16 #define THR_POLL_TO 5000 =20 +#define MAD_HDR_SIZE sizeof(struct ibv_grh) + typedef struct BackendCtx { - uint64_t req_id; void *up_ctx; bool is_tx_req; + struct ibv_sge sge; /* Used to save MAD recv buffer */ } BackendCtx; =20 +struct backend_umad { + struct ib_user_mad hdr; + char mad[RDMA_MAX_PRIVATE_DATA]; +}; + static void (*comp_handler)(int status, unsigned int vendor_err, void *ctx= ); =20 static void dummy_comp_handler(int status, unsigned int vendor_err, void *= ctx) @@ -286,6 +300,49 @@ static int build_host_sge_array(RdmaDeviceResources *r= dma_dev_res, return 0; } =20 +static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, + uint32_t num_sge) +{ + struct backend_umad umad =3D {0}; + char *hdr, *msg; + int ret; + + pr_dbg("num_sge=3D%d\n", num_sge); + + if (num_sge !=3D 2) { + return -EINVAL; + } + + umad.hdr.length =3D sge[0].length + sge[1].length; + pr_dbg("msg_len=3D%d\n", umad.hdr.length); + + if (umad.hdr.length > sizeof(umad.mad)) { + return -ENOMEM; + } + + umad.hdr.addr.qpn =3D htobe32(1); + umad.hdr.addr.grh_present =3D 1; + umad.hdr.addr.gid_index =3D backend_dev->backend_gid_idx; + memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); + umad.hdr.addr.hop_limit =3D 1; + + hdr =3D rdma_pci_dma_map(backend_dev->dev, sge[0].addr, sge[0].length); + msg =3D rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length); + + memcpy(&umad.mad[0], hdr, sge[0].length); + memcpy(&umad.mad[sge[0].length], msg, sge[1].length); + + rdma_pci_dma_unmap(backend_dev->dev, msg, sge[1].length); + rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); + + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, + sizeof(umad)); + + pr_dbg("qemu_chr_fe_write=3D%d\n", ret); + + return (ret !=3D sizeof(umad)); +} + void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, @@ -304,9 +361,13 @@ void rdma_backend_post_send(RdmaBackendDev *backend_de= v, comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + rc =3D mad_send(backend_dev, sge, num_sge); + if (rc) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + } else { + comp_handler(IBV_WC_SUCCESS, 0, ctx); + } } - pr_dbg("qp->ibqp is NULL for qp_type %d!!!\n", qp_type); return; } =20 @@ -370,6 +431,48 @@ out_free_bctx: g_free(bctx); } =20 +static unsigned int save_mad_recv_buffer(RdmaBackendDev *backend_dev, + struct ibv_sge *sge, uint32_t num= _sge, + void *ctx) +{ + BackendCtx *bctx; + int rc; + uint32_t bctx_id; + + if (num_sge !=3D 1) { + pr_dbg("Invalid num_sge (%d), expecting 1\n", num_sge); + return VENDOR_ERR_INV_NUM_SGE; + } + + if (sge[0].length < RDMA_MAX_PRIVATE_DATA + sizeof(struct ibv_grh)) { + pr_dbg("Too small buffer for MAD\n"); + return VENDOR_ERR_INV_MAD_BUFF; + } + + pr_dbg("addr=3D0x%" PRIx64"\n", sge[0].addr); + pr_dbg("length=3D%d\n", sge[0].length); + pr_dbg("lkey=3D%d\n", sge[0].lkey); + + bctx =3D g_malloc0(sizeof(*bctx)); + + rc =3D rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx= ); + if (unlikely(rc)) { + g_free(bctx); + pr_dbg("Fail to allocate cqe_ctx\n"); + return VENDOR_ERR_NOMEM; + } + + pr_dbg("bctx_id %d, bctx %p, ctx %p\n", bctx_id, bctx, ctx); + bctx->up_ctx =3D ctx; + bctx->sge =3D *sge; + + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + qlist_append_int(backend_dev->recv_mads_list.list, bctx_id); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + + return 0; +} + void rdma_backend_post_recv(RdmaBackendDev *backend_dev, RdmaDeviceResources *rdma_dev_res, RdmaBackendQP *qp, uint8_t qp_type, @@ -388,7 +491,10 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_de= v, } if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + rc =3D save_mad_recv_buffer(backend_dev, sge, num_sge, ctx); + if (rc) { + comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + } } return; } @@ -517,7 +623,6 @@ int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t q= p_type, =20 switch (qp_type) { case IBV_QPT_GSI: - pr_dbg("QP1 unsupported\n"); return 0; =20 case IBV_QPT_RC: @@ -748,11 +853,146 @@ static int init_device_caps(RdmaBackendDev *backend_= dev, return 0; } =20 +static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, + union ibv_gid *my_gid, int paylen) +{ + grh->paylen =3D htons(paylen); + grh->sgid =3D *sgid; + grh->dgid =3D *my_gid; + + pr_dbg("paylen=3D%d (net=3D0x%x)\n", paylen, grh->paylen); + pr_dbg("my_gid=3D0x%llx\n", my_gid->global.interface_id); + pr_dbg("gid=3D0x%llx\n", sgid->global.interface_id); +} + +static inline int mad_can_receieve(void *opaque) +{ + return sizeof(struct backend_umad); +} + +static void mad_read(void *opaque, const uint8_t *buf, int size) +{ + RdmaBackendDev *backend_dev =3D (RdmaBackendDev *)opaque; + QObject *o_ctx_id; + unsigned long cqe_ctx_id; + BackendCtx *bctx; + char *mad; + struct backend_umad *umad; + + assert(size !=3D sizeof(umad)); + umad =3D (struct backend_umad *)buf; + + pr_dbg("Got %d bytes\n", size); + pr_dbg("umad->hdr.length=3D%d\n", umad->hdr.length); + +#ifdef PVRDMA_DEBUG + struct umad_hdr *hdr =3D (struct umad_hdr *)&msg->umad.mad; + pr_dbg("bv %x cls %x cv %x mtd %x st %d tid %" PRIx64 " at %x atm %x\n= ", + hdr->base_version, hdr->mgmt_class, hdr->class_version, + hdr->method, hdr->status, be64toh(hdr->tid), + hdr->attr_id, hdr->attr_mod); +#endif + + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + o_ctx_id =3D qlist_pop(backend_dev->recv_mads_list.list); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + if (!o_ctx_id) { + pr_dbg("No more free MADs buffers, waiting for a while\n"); + sleep(THR_POLL_TO); + return; + } + + cqe_ctx_id =3D qnum_get_uint(qobject_to(QNum, o_ctx_id)); + bctx =3D rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); + if (unlikely(!bctx)) { + pr_dbg("Error: Fail to find ctx for %ld\n", cqe_ctx_id); + return; + } + + pr_dbg("id %ld, bctx %p, ctx %p\n", cqe_ctx_id, bctx, bctx->up_ctx); + + mad =3D rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, + bctx->sge.length); + if (!mad || bctx->sge.length < umad->hdr.length + MAD_HDR_SIZE) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, + bctx->up_ctx); + } else { + memset(mad, 0, bctx->sge.length); + build_mad_hdr((struct ibv_grh *)mad, + (union ibv_gid *)&umad->hdr.addr.gid, + &backend_dev->gid, umad->hdr.length); + memcpy(&mad[MAD_HDR_SIZE], umad->mad, umad->hdr.length); + rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); + + comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); + } + + g_free(bctx); + rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); +} + +static int mad_init(RdmaBackendDev *backend_dev) +{ + struct backend_umad umad =3D {0}; + int ret; + + if (!qemu_chr_fe_backend_connected(backend_dev->mad_chr_be)) { + pr_dbg("Missing chardev for MAD multiplexer\n"); + return -EIO; + } + + qemu_chr_fe_set_handlers(backend_dev->mad_chr_be, mad_can_receieve, + mad_read, NULL, NULL, backend_dev, NULL, true= ); + + /* Register ourself */ + memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, + sizeof(umad.hdr)); + if (ret !=3D sizeof(umad.hdr)) { + pr_dbg("Fail to register to rdma_umadmux (%d)\n", ret); + } + + qemu_mutex_init(&backend_dev->recv_mads_list.lock); + backend_dev->recv_mads_list.list =3D qlist_new(); + + return 0; +} + +static void mad_stop(RdmaBackendDev *backend_dev) +{ + QObject *o_ctx_id; + unsigned long cqe_ctx_id; + BackendCtx *bctx; + + pr_dbg("Closing MAD\n"); + + /* Clear MAD buffers list */ + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + do { + o_ctx_id =3D qlist_pop(backend_dev->recv_mads_list.list); + if (o_ctx_id) { + cqe_ctx_id =3D qnum_get_uint(qobject_to(QNum, o_ctx_id)); + bctx =3D rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ct= x_id); + if (bctx) { + rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx= _id); + g_free(bctx); + } + } + } while (o_ctx_id); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); +} + +static void mad_fini(RdmaBackendDev *backend_dev) +{ + qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); + qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); +} + int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - Error **errp) + CharBackend *mad_chr_be, Error **errp) { int i; int ret =3D 0; @@ -763,7 +1003,7 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCI= Device *pdev, memset(backend_dev, 0, sizeof(*backend_dev)); =20 backend_dev->dev =3D pdev; - + backend_dev->mad_chr_be =3D mad_chr_be; backend_dev->backend_gid_idx =3D backend_gid_idx; backend_dev->port_num =3D port_num; backend_dev->rdma_dev_res =3D rdma_dev_res; @@ -854,6 +1094,13 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PC= IDevice *pdev, pr_dbg("interface_id=3D0x%" PRIx64 "\n", be64_to_cpu(backend_dev->gid.global.interface_id)); =20 + ret =3D mad_init(backend_dev); + if (ret) { + error_setg(errp, "Fail to initialize mad"); + ret =3D -EIO; + goto out_destroy_comm_channel; + } + backend_dev->comp_thread.run =3D false; backend_dev->comp_thread.is_running =3D false; =20 @@ -885,11 +1132,13 @@ void rdma_backend_stop(RdmaBackendDev *backend_dev) { pr_dbg("Stopping rdma_backend\n"); stop_backend_thread(&backend_dev->comp_thread); + mad_stop(backend_dev); } =20 void rdma_backend_fini(RdmaBackendDev *backend_dev) { rdma_backend_stop(backend_dev); + mad_fini(backend_dev); g_hash_table_destroy(ah_hash); ibv_destroy_comp_channel(backend_dev->channel); ibv_close_device(backend_dev->context); diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 3ccc9a2494..fc83330251 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -17,6 +17,8 @@ #define RDMA_BACKEND_H =20 #include "qapi/error.h" +#include "chardev/char-fe.h" + #include "rdma_rm_defs.h" #include "rdma_backend_defs.h" =20 @@ -50,7 +52,7 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDev= ice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - Error **errp); + CharBackend *mad_chr_be, Error **errp); void rdma_backend_fini(RdmaBackendDev *backend_dev); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 7404f64002..2a7e667075 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -16,8 +16,9 @@ #ifndef RDMA_BACKEND_DEFS_H #define RDMA_BACKEND_DEFS_H =20 -#include #include "qemu/thread.h" +#include "chardev/char-fe.h" +#include =20 typedef struct RdmaDeviceResources RdmaDeviceResources; =20 @@ -28,6 +29,11 @@ typedef struct RdmaBackendThread { bool is_running; /* Set by the thread to report its status */ } RdmaBackendThread; =20 +typedef struct RecvMadList { + QemuMutex lock; + QList *list; +} RecvMadList; + typedef struct RdmaBackendDev { struct ibv_device_attr dev_attr; RdmaBackendThread comp_thread; @@ -39,6 +45,8 @@ typedef struct RdmaBackendDev { struct ibv_comp_channel *channel; uint8_t port_num; uint8_t backend_gid_idx; + RecvMadList recv_mads_list; + CharBackend *mad_chr_be; } RdmaBackendDev; =20 typedef struct RdmaBackendPD { diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index e2d9f93cdf..e3742d893a 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -19,6 +19,7 @@ #include "qemu/units.h" #include "hw/pci/pci.h" #include "hw/pci/msix.h" +#include "chardev/char-fe.h" =20 #include "../rdma_backend_defs.h" #include "../rdma_rm_defs.h" @@ -83,6 +84,7 @@ typedef struct PVRDMADev { uint8_t backend_port_num; RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; + CharBackend mad_chr; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index ca5fa8d981..6c8c0154fa 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -51,6 +51,7 @@ static Property pvrdma_dev_properties[] =3D { DEFINE_PROP_INT32("dev-caps-max-qp-init-rd-atom", PVRDMADev, dev_attr.max_qp_init_rd_atom, MAX_QP_INIT_RD_ATOM), DEFINE_PROP_INT32("dev-caps-max-ah", PVRDMADev, dev_attr.max_ah, MAX_A= H), + DEFINE_PROP_CHR("mad-chardev", PVRDMADev, mad_chr), DEFINE_PROP_END_OF_LIST(), }; =20 @@ -613,7 +614,8 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) =20 rc =3D rdma_backend_init(&dev->backend_dev, pdev, &dev->rdma_dev_res, dev->backend_device_name, dev->backend_port_num, - dev->backend_gid_idx, &dev->dev_attr, errp); + dev->backend_gid_idx, &dev->dev_attr, &dev->mad= _chr, + errp); if (rc) { goto out; } --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093775811807.2782222272556; Mon, 12 Nov 2018 23:22:55 -0800 (PST) Received: from localhost ([::1]:52349 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT29-0001YY-71 for importer@patchew.org; Tue, 13 Nov 2018 02:22:49 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51614) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSu2-0003pa-4m for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMStu-0002c2-Pk for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:23 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:44638) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMStu-0002Zy-0p for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:18 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EFZS037452; Tue, 13 Nov 2018 07:14:15 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2nnwc0gmyf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:15 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EFAj009268 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:15 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7EEL5017470; Tue, 13 Nov 2018 07:14:14 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:14 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=rOHY0FuAgAXuH+Z3jUp0X6wXHpoLklkNJWEZttnsIfA=; b=vwpSbRdg7jgMMxxssqHxnyDizmcJnH2jRy75lF9I+7bjhzR4Jt63f2HHYRL7kS5xyr/7 5uRq+PMOnsdBkGyC0SnIfPcuioKHm4V4g6VV08A0w05I34baHgHLBbx7TuELNlJkXnW+ 3AbePqdu8J6of3mUzKvlLvmXQfHcxED/bkSFpgUGktrhj7MEYttSSvnVP003xehSiRhE IVWI7eZtxO1h9/VjfxqNpvxxHyZ1mHkBIirnX+mPMIUouqZ2NCDZL7WGObtatUmfofzY f/iRyMytntJh858sg+C4+lV0bmJtyaKcJHNlg+8NpyaE/1/jy/dDdP52OpnUg0mfg9Pt 4A== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:54 +0200 Message-Id: <20181113071336.6242-6-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 05/23] hw/rdma: Add support for MAD packets X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" MAD (Management Datagram) packets are widely used by various modules both in kernel and in user space for example the rdma_* API which is used to create and maintain "connection" layer on top of RDMA uses several types of MAD packets. To support MAD packets the device uses an external utility (contrib/rdmacm-mux) to relay packets from and to the guest driver. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_backend.c | 263 +++++++++++++++++++++++++++++++++++- hw/rdma/rdma_backend.h | 4 +- hw/rdma/rdma_backend_defs.h | 10 +- hw/rdma/vmw/pvrdma.h | 2 + hw/rdma/vmw/pvrdma_main.c | 4 +- 5 files changed, 273 insertions(+), 10 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 1e148398a2..3eb0099f8d 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -16,8 +16,13 @@ #include "qemu/osdep.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" =20 #include +#include +#include +#include =20 #include "trace.h" #include "rdma_utils.h" @@ -33,16 +38,25 @@ #define VENDOR_ERR_MAD_SEND 0x206 #define VENDOR_ERR_INVLKEY 0x207 #define VENDOR_ERR_MR_SMALL 0x208 +#define VENDOR_ERR_INV_MAD_BUFF 0x209 +#define VENDOR_ERR_INV_NUM_SGE 0x210 =20 #define THR_NAME_LEN 16 #define THR_POLL_TO 5000 =20 +#define MAD_HDR_SIZE sizeof(struct ibv_grh) + typedef struct BackendCtx { - uint64_t req_id; void *up_ctx; bool is_tx_req; + struct ibv_sge sge; /* Used to save MAD recv buffer */ } BackendCtx; =20 +struct backend_umad { + struct ib_user_mad hdr; + char mad[RDMA_MAX_PRIVATE_DATA]; +}; + static void (*comp_handler)(int status, unsigned int vendor_err, void *ctx= ); =20 static void dummy_comp_handler(int status, unsigned int vendor_err, void *= ctx) @@ -286,6 +300,49 @@ static int build_host_sge_array(RdmaDeviceResources *r= dma_dev_res, return 0; } =20 +static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, + uint32_t num_sge) +{ + struct backend_umad umad =3D {0}; + char *hdr, *msg; + int ret; + + pr_dbg("num_sge=3D%d\n", num_sge); + + if (num_sge !=3D 2) { + return -EINVAL; + } + + umad.hdr.length =3D sge[0].length + sge[1].length; + pr_dbg("msg_len=3D%d\n", umad.hdr.length); + + if (umad.hdr.length > sizeof(umad.mad)) { + return -ENOMEM; + } + + umad.hdr.addr.qpn =3D htobe32(1); + umad.hdr.addr.grh_present =3D 1; + umad.hdr.addr.gid_index =3D backend_dev->backend_gid_idx; + memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); + umad.hdr.addr.hop_limit =3D 1; + + hdr =3D rdma_pci_dma_map(backend_dev->dev, sge[0].addr, sge[0].length); + msg =3D rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length); + + memcpy(&umad.mad[0], hdr, sge[0].length); + memcpy(&umad.mad[sge[0].length], msg, sge[1].length); + + rdma_pci_dma_unmap(backend_dev->dev, msg, sge[1].length); + rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); + + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, + sizeof(umad)); + + pr_dbg("qemu_chr_fe_write=3D%d\n", ret); + + return (ret !=3D sizeof(umad)); +} + void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, @@ -304,9 +361,13 @@ void rdma_backend_post_send(RdmaBackendDev *backend_de= v, comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + rc =3D mad_send(backend_dev, sge, num_sge); + if (rc) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + } else { + comp_handler(IBV_WC_SUCCESS, 0, ctx); + } } - pr_dbg("qp->ibqp is NULL for qp_type %d!!!\n", qp_type); return; } =20 @@ -370,6 +431,48 @@ out_free_bctx: g_free(bctx); } =20 +static unsigned int save_mad_recv_buffer(RdmaBackendDev *backend_dev, + struct ibv_sge *sge, uint32_t num= _sge, + void *ctx) +{ + BackendCtx *bctx; + int rc; + uint32_t bctx_id; + + if (num_sge !=3D 1) { + pr_dbg("Invalid num_sge (%d), expecting 1\n", num_sge); + return VENDOR_ERR_INV_NUM_SGE; + } + + if (sge[0].length < RDMA_MAX_PRIVATE_DATA + sizeof(struct ibv_grh)) { + pr_dbg("Too small buffer for MAD\n"); + return VENDOR_ERR_INV_MAD_BUFF; + } + + pr_dbg("addr=3D0x%" PRIx64"\n", sge[0].addr); + pr_dbg("length=3D%d\n", sge[0].length); + pr_dbg("lkey=3D%d\n", sge[0].lkey); + + bctx =3D g_malloc0(sizeof(*bctx)); + + rc =3D rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx= ); + if (unlikely(rc)) { + g_free(bctx); + pr_dbg("Fail to allocate cqe_ctx\n"); + return VENDOR_ERR_NOMEM; + } + + pr_dbg("bctx_id %d, bctx %p, ctx %p\n", bctx_id, bctx, ctx); + bctx->up_ctx =3D ctx; + bctx->sge =3D *sge; + + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + qlist_append_int(backend_dev->recv_mads_list.list, bctx_id); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + + return 0; +} + void rdma_backend_post_recv(RdmaBackendDev *backend_dev, RdmaDeviceResources *rdma_dev_res, RdmaBackendQP *qp, uint8_t qp_type, @@ -388,7 +491,10 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_de= v, } if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + rc =3D save_mad_recv_buffer(backend_dev, sge, num_sge, ctx); + if (rc) { + comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + } } return; } @@ -517,7 +623,6 @@ int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t q= p_type, =20 switch (qp_type) { case IBV_QPT_GSI: - pr_dbg("QP1 unsupported\n"); return 0; =20 case IBV_QPT_RC: @@ -748,11 +853,146 @@ static int init_device_caps(RdmaBackendDev *backend_= dev, return 0; } =20 +static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, + union ibv_gid *my_gid, int paylen) +{ + grh->paylen =3D htons(paylen); + grh->sgid =3D *sgid; + grh->dgid =3D *my_gid; + + pr_dbg("paylen=3D%d (net=3D0x%x)\n", paylen, grh->paylen); + pr_dbg("my_gid=3D0x%llx\n", my_gid->global.interface_id); + pr_dbg("gid=3D0x%llx\n", sgid->global.interface_id); +} + +static inline int mad_can_receieve(void *opaque) +{ + return sizeof(struct backend_umad); +} + +static void mad_read(void *opaque, const uint8_t *buf, int size) +{ + RdmaBackendDev *backend_dev =3D (RdmaBackendDev *)opaque; + QObject *o_ctx_id; + unsigned long cqe_ctx_id; + BackendCtx *bctx; + char *mad; + struct backend_umad *umad; + + assert(size !=3D sizeof(umad)); + umad =3D (struct backend_umad *)buf; + + pr_dbg("Got %d bytes\n", size); + pr_dbg("umad->hdr.length=3D%d\n", umad->hdr.length); + +#ifdef PVRDMA_DEBUG + struct umad_hdr *hdr =3D (struct umad_hdr *)&msg->umad.mad; + pr_dbg("bv %x cls %x cv %x mtd %x st %d tid %" PRIx64 " at %x atm %x\n= ", + hdr->base_version, hdr->mgmt_class, hdr->class_version, + hdr->method, hdr->status, be64toh(hdr->tid), + hdr->attr_id, hdr->attr_mod); +#endif + + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + o_ctx_id =3D qlist_pop(backend_dev->recv_mads_list.list); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + if (!o_ctx_id) { + pr_dbg("No more free MADs buffers, waiting for a while\n"); + sleep(THR_POLL_TO); + return; + } + + cqe_ctx_id =3D qnum_get_uint(qobject_to(QNum, o_ctx_id)); + bctx =3D rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); + if (unlikely(!bctx)) { + pr_dbg("Error: Fail to find ctx for %ld\n", cqe_ctx_id); + return; + } + + pr_dbg("id %ld, bctx %p, ctx %p\n", cqe_ctx_id, bctx, bctx->up_ctx); + + mad =3D rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, + bctx->sge.length); + if (!mad || bctx->sge.length < umad->hdr.length + MAD_HDR_SIZE) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, + bctx->up_ctx); + } else { + memset(mad, 0, bctx->sge.length); + build_mad_hdr((struct ibv_grh *)mad, + (union ibv_gid *)&umad->hdr.addr.gid, + &backend_dev->gid, umad->hdr.length); + memcpy(&mad[MAD_HDR_SIZE], umad->mad, umad->hdr.length); + rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); + + comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); + } + + g_free(bctx); + rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); +} + +static int mad_init(RdmaBackendDev *backend_dev) +{ + struct backend_umad umad =3D {0}; + int ret; + + if (!qemu_chr_fe_backend_connected(backend_dev->mad_chr_be)) { + pr_dbg("Missing chardev for MAD multiplexer\n"); + return -EIO; + } + + qemu_chr_fe_set_handlers(backend_dev->mad_chr_be, mad_can_receieve, + mad_read, NULL, NULL, backend_dev, NULL, true= ); + + /* Register ourself */ + memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, + sizeof(umad.hdr)); + if (ret !=3D sizeof(umad.hdr)) { + pr_dbg("Fail to register to rdma_umadmux (%d)\n", ret); + } + + qemu_mutex_init(&backend_dev->recv_mads_list.lock); + backend_dev->recv_mads_list.list =3D qlist_new(); + + return 0; +} + +static void mad_stop(RdmaBackendDev *backend_dev) +{ + QObject *o_ctx_id; + unsigned long cqe_ctx_id; + BackendCtx *bctx; + + pr_dbg("Closing MAD\n"); + + /* Clear MAD buffers list */ + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + do { + o_ctx_id =3D qlist_pop(backend_dev->recv_mads_list.list); + if (o_ctx_id) { + cqe_ctx_id =3D qnum_get_uint(qobject_to(QNum, o_ctx_id)); + bctx =3D rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ct= x_id); + if (bctx) { + rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx= _id); + g_free(bctx); + } + } + } while (o_ctx_id); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); +} + +static void mad_fini(RdmaBackendDev *backend_dev) +{ + qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); + qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); +} + int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - Error **errp) + CharBackend *mad_chr_be, Error **errp) { int i; int ret =3D 0; @@ -763,7 +1003,7 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCI= Device *pdev, memset(backend_dev, 0, sizeof(*backend_dev)); =20 backend_dev->dev =3D pdev; - + backend_dev->mad_chr_be =3D mad_chr_be; backend_dev->backend_gid_idx =3D backend_gid_idx; backend_dev->port_num =3D port_num; backend_dev->rdma_dev_res =3D rdma_dev_res; @@ -854,6 +1094,13 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PC= IDevice *pdev, pr_dbg("interface_id=3D0x%" PRIx64 "\n", be64_to_cpu(backend_dev->gid.global.interface_id)); =20 + ret =3D mad_init(backend_dev); + if (ret) { + error_setg(errp, "Fail to initialize mad"); + ret =3D -EIO; + goto out_destroy_comm_channel; + } + backend_dev->comp_thread.run =3D false; backend_dev->comp_thread.is_running =3D false; =20 @@ -885,11 +1132,13 @@ void rdma_backend_stop(RdmaBackendDev *backend_dev) { pr_dbg("Stopping rdma_backend\n"); stop_backend_thread(&backend_dev->comp_thread); + mad_stop(backend_dev); } =20 void rdma_backend_fini(RdmaBackendDev *backend_dev) { rdma_backend_stop(backend_dev); + mad_fini(backend_dev); g_hash_table_destroy(ah_hash); ibv_destroy_comp_channel(backend_dev->channel); ibv_close_device(backend_dev->context); diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 3ccc9a2494..fc83330251 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -17,6 +17,8 @@ #define RDMA_BACKEND_H =20 #include "qapi/error.h" +#include "chardev/char-fe.h" + #include "rdma_rm_defs.h" #include "rdma_backend_defs.h" =20 @@ -50,7 +52,7 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDev= ice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - Error **errp); + CharBackend *mad_chr_be, Error **errp); void rdma_backend_fini(RdmaBackendDev *backend_dev); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 7404f64002..2a7e667075 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -16,8 +16,9 @@ #ifndef RDMA_BACKEND_DEFS_H #define RDMA_BACKEND_DEFS_H =20 -#include #include "qemu/thread.h" +#include "chardev/char-fe.h" +#include =20 typedef struct RdmaDeviceResources RdmaDeviceResources; =20 @@ -28,6 +29,11 @@ typedef struct RdmaBackendThread { bool is_running; /* Set by the thread to report its status */ } RdmaBackendThread; =20 +typedef struct RecvMadList { + QemuMutex lock; + QList *list; +} RecvMadList; + typedef struct RdmaBackendDev { struct ibv_device_attr dev_attr; RdmaBackendThread comp_thread; @@ -39,6 +45,8 @@ typedef struct RdmaBackendDev { struct ibv_comp_channel *channel; uint8_t port_num; uint8_t backend_gid_idx; + RecvMadList recv_mads_list; + CharBackend *mad_chr_be; } RdmaBackendDev; =20 typedef struct RdmaBackendPD { diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index e2d9f93cdf..e3742d893a 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -19,6 +19,7 @@ #include "qemu/units.h" #include "hw/pci/pci.h" #include "hw/pci/msix.h" +#include "chardev/char-fe.h" =20 #include "../rdma_backend_defs.h" #include "../rdma_rm_defs.h" @@ -83,6 +84,7 @@ typedef struct PVRDMADev { uint8_t backend_port_num; RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; + CharBackend mad_chr; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index ca5fa8d981..6c8c0154fa 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -51,6 +51,7 @@ static Property pvrdma_dev_properties[] =3D { DEFINE_PROP_INT32("dev-caps-max-qp-init-rd-atom", PVRDMADev, dev_attr.max_qp_init_rd_atom, MAX_QP_INIT_RD_ATOM), DEFINE_PROP_INT32("dev-caps-max-ah", PVRDMADev, dev_attr.max_ah, MAX_A= H), + DEFINE_PROP_CHR("mad-chardev", PVRDMADev, mad_chr), DEFINE_PROP_END_OF_LIST(), }; =20 @@ -613,7 +614,8 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) =20 rc =3D rdma_backend_init(&dev->backend_dev, pdev, &dev->rdma_dev_res, dev->backend_device_name, dev->backend_port_num, - dev->backend_gid_idx, &dev->dev_attr, errp); + dev->backend_gid_idx, &dev->dev_attr, &dev->mad= _chr, + errp); if (rc) { goto out; } --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 154209359186871.46362102274475; Mon, 12 Nov 2018 23:19:51 -0800 (PST) Received: from localhost ([::1]:52332 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSzG-0007b6-DR for importer@patchew.org; Tue, 13 Nov 2018 02:19:50 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51656) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSu8-0003vl-Q6 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSu4-0002oO-Cv for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:32 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:44702) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSu4-0002e0-2e for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:28 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7E6de037388; Tue, 13 Nov 2018 07:14:19 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2nnwc0gmyp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:19 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EICm020064 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:18 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7EHb3017572; Tue, 13 Nov 2018 07:14:17 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:17 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=WSQVc6fFfhV/W33OJ/u8o9PSALsubtJHR6DYTkyqNvI=; b=W7+12AsjxqPVatJZw8hnibF3h+yO2dOI5s0mzEH6KIBvpice2PoWXPfV9upusaxqE6Fl XKF7VlBikNoNXmytl8HR32WKouUoU/9e1435Z6RQB9fAkMNK+f1qMquNAbdxIYl5cIch pMYep7tqmhCqM04Bu0ptUN2Qnh+2jEkoVG37rpsWyQIIUiW1XBIH2t+9lvZPaZbTudmZ phY8c5qDutqsQlgSRjPqmgewC0XADmyxbJ4ZjssMwa9H8MthIJoUaFeghkYJiCCUWi6L UN7eXHPBscNJhwsXweZUXVxBqkapHUW0reRJASXupcqCMA0U3b5VHYoCGJXHGRkfhQwq fQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:55 +0200 Message-Id: <20181113071336.6242-7-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=903 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 06/23] hw/pvrdma: Make function reset_device return void X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This function cannot fail - fix it to return void Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_main.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 6c8c0154fa..fc2abd34af 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -369,13 +369,11 @@ static int unquiesce_device(PVRDMADev *dev) return 0; } =20 -static int reset_device(PVRDMADev *dev) +static void reset_device(PVRDMADev *dev) { pvrdma_stop(dev); =20 pr_dbg("Device reset complete\n"); - - return 0; } =20 static uint64_t regs_read(void *opaque, hwaddr addr, unsigned size) --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094885321866.5103634710666; Mon, 12 Nov 2018 23:41:25 -0800 (PST) Received: from localhost ([::1]:52446 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTK0-0008Ng-Cd for importer@patchew.org; Tue, 13 Nov 2018 02:41:16 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52172) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvI-0004wN-S9 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvH-0004QI-7E for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:44 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44390) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvG-0004J7-PM for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:43 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Dlmd078853; Tue, 13 Nov 2018 07:15:36 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2nnwg18mbn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:36 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FUTF012819 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:30 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FU6Z028642; Tue, 13 Nov 2018 07:15:30 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:30 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=WSQVc6fFfhV/W33OJ/u8o9PSALsubtJHR6DYTkyqNvI=; b=yjLe6i+93+tqiRFj6m6k3KJgU8Kp+dmLkgz40fQewuzjaWjw5wo2mzZ9U8Q68V7HGhnX QTxjE9oQTPfML8d/YdKNT7yfazf46aDH9KXcBJlXVfIIEUvq/PhhxRE6XBq/RxnT7cDG ZrSA70Qss/TKy1pfJHcs4NoYOn5ohFcnuIj3lXDe9Wcf9KcmGagJrcdaJc2zSuCrMx32 i43pJiOpfazEt5WcfZctj+VwFwupMUfKlxOaVOrBycAq2OY6brJzBhcrpRE6nJSAlgBj bew0C4YbjUAq6EDdNKvcdwG0NkK6JJIifLWGNFCsE2E0jaiVsVqW+eMJAAuJXWtoYL+P og== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:19 +0200 Message-Id: <20181113071336.6242-31-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=903 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 06/23] hw/pvrdma: Make function reset_device return void X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This function cannot fail - fix it to return void Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_main.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 6c8c0154fa..fc2abd34af 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -369,13 +369,11 @@ static int unquiesce_device(PVRDMADev *dev) return 0; } =20 -static int reset_device(PVRDMADev *dev) +static void reset_device(PVRDMADev *dev) { pvrdma_stop(dev); =20 pr_dbg("Device reset complete\n"); - - return 0; } =20 static uint64_t regs_read(void *opaque, hwaddr addr, unsigned size) --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093943733187.60963120835356; Mon, 12 Nov 2018 23:25:43 -0800 (PST) Received: from localhost ([::1]:52364 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT4w-0004PA-J8 for importer@patchew.org; Tue, 13 Nov 2018 02:25:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51657) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSu8-0003vm-Q3 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSu4-0002nx-63 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:32 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:43546) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSu2-0002fi-7l for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:28 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EL1H079286; Tue, 13 Nov 2018 07:14:21 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2nnwg18m7h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:21 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EL2J009530 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:21 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EKCN011714; Tue, 13 Nov 2018 07:14:20 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:20 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=lres8zxXcMwUJghKrp3MIBZTlaNpzoF3DKwfz3MbE1U=; b=GyBX61JqHlZf+Nt0TfmBsDXV8zW01PMSAIs1+jftsq84+HZAsFyllo7tIwuV5HgsSQX4 RbDIP9GMxQWxHAmBWheozLN24cRPYzTiOazBcLkLdwYId1G/NgA8WoUar+sH/UjyNNyt gWF03nTgkveOJpptgOXPVvkBfsvGK9blwMPUqbjxahliq6rQCFcv1Df4koJ57lUzqkoF wrL5QpCKMoXSL/zdv1jBqrpt9COWNyseJdvO//lkGeFBuKEgjHQCa4+ebc8kk533oWyD Fvi9g225oOLQRh3fnBWrpUUyPVLAeJqXyGkIzOC2zP0VunRw3q2Y03rNJYYnN5Iz4w11 yQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:56 +0200 Message-Id: <20181113071336.6242-8-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=603 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 07/23] hw/pvrdma: Make default pkey 0xFFFF X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Commit 6e7dba23af ("hw/pvrdma: Make default pkey 0xFFFF") exports default pkey as external definition but omit the change from 0x7FFF to 0xFFFF. Fixes: 6e7dba23af ("hw/pvrdma: Make default pkey 0xFFFF") Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index e3742d893a..15c3f28b86 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -52,7 +52,7 @@ #define PVRDMA_FW_VERSION 14 =20 /* Some defaults */ -#define PVRDMA_PKEY 0x7FFF +#define PVRDMA_PKEY 0xFFFF =20 typedef struct DSRInfo { dma_addr_t dma; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094144654320.7638753504259; Mon, 12 Nov 2018 23:29:04 -0800 (PST) Received: from localhost ([::1]:52383 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT8B-0007IK-Jn for importer@patchew.org; Tue, 13 Nov 2018 02:29:03 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52135) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvH-0004vm-EK for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvE-0004Mn-L7 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:43 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44368) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvC-0004HI-RV for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:39 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Ep8t079429; Tue, 13 Nov 2018 07:15:34 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2nnwg18mbj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:34 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FXCY012959 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:33 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7FXus030798; Tue, 13 Nov 2018 07:15:33 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=lres8zxXcMwUJghKrp3MIBZTlaNpzoF3DKwfz3MbE1U=; b=LFLNnfY6fGjyXTjkYW+jbVJCbCuq8H3IE4bCiQaaVTDqDLez/19TzuJzutOsDyQbjh/h pYFptuNBb5tGLJtNgahQLKEwQsFYDrXUNIGWHTqjfasRWPqbHLNoibxeAAo1N10e3GgP uvvgSmHtjJ4qvY1l5uGwuURb/fignSOqUjxUkaM+cg7HULuUBg6JOtfL+6PWfjkOFo/X a+yRx7Z6E+IJIYpR8/aDs0BSG4UKvCKwWpSRCrzQHXHUtHzaByvZfI/L1c8UC6iYyOnN tDcOALbv2OFGGt9OO6HAFyPq3NTlAKW368v73VRVDGe7zQyXWUgPVGmdqL1hhHfmjNBI hg== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:20 +0200 Message-Id: <20181113071336.6242-32-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=603 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 07/23] hw/pvrdma: Make default pkey 0xFFFF X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Commit 6e7dba23af ("hw/pvrdma: Make default pkey 0xFFFF") exports default pkey as external definition but omit the change from 0x7FFF to 0xFFFF. Fixes: 6e7dba23af ("hw/pvrdma: Make default pkey 0xFFFF") Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index e3742d893a..15c3f28b86 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -52,7 +52,7 @@ #define PVRDMA_FW_VERSION 14 =20 /* Some defaults */ -#define PVRDMA_PKEY 0x7FFF +#define PVRDMA_PKEY 0xFFFF =20 typedef struct DSRInfo { dma_addr_t dma; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095060788495.1711472830907; Mon, 12 Nov 2018 23:44:20 -0800 (PST) Received: from localhost ([::1]:52459 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTMx-0002Nx-MJ for importer@patchew.org; Tue, 13 Nov 2018 02:44:19 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52243) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvL-0004yx-D7 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvI-0004Sj-UY for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:47 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60478) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvI-0004RT-Im for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:44 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Dpan045981; Tue, 13 Nov 2018 07:15:43 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2120.oracle.com with ESMTP id 2nnw6egpu2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:43 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FaPj018375 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:37 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7FaZr030811; Tue, 13 Nov 2018 07:15:36 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:36 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=xDiBfg1kHb6oDTu4/E5lBynVjVnrP3sAMLNJA6kUlXs=; b=2HpLzuhZ1oMOWdUVHVt53iHJ8sLS09AbJ6yyxxd601zh7wifYg2e5Rb3ZLNkB/KvJ1+X 7At4koLOkNRxX3czAcGlZjjq8ehR2RR20fZ7rMBy4otT3cAxWdQgL5djppbUq2yW3g7C TALfyiMyD+ZYoj70srLr7cGFjN7901Pk7AwSVNnFwe86076sqO4UfOVNb5dbGOyaKO3J zVubmF6c5jLJDtyFm55KY+Oo0c0xwnSuXqG6Sz0DIPXJQJD0ofxYJGJ5N2mBUcKiItml c0TB3yNq8HEDia7ITnEyzyJo9Z6Daqg3AEtkO1N6D9s+7EJ3YTEsKG/0K92kY/XH2aJd xg== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:21 +0200 Message-Id: <20181113071336.6242-33-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=783 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 08/23] hw/pvrdma: Set the correct opcode for recv completion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The function pvrdma_post_cqe populates CQE entry with opcode from the given completion element. For receive operation value was not set. Fix it by setting it to IBV_WC_RECV. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_qp_ops.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 762700a205..7b0f440fda 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -196,8 +196,9 @@ int pvrdma_qp_recv(PVRDMADev *dev, uint32_t qp_handle) comp_ctx =3D g_malloc(sizeof(CompHandlerCtx)); comp_ctx->dev =3D dev; comp_ctx->cq_handle =3D qp->recv_cq_handle; - comp_ctx->cqe.qp =3D qp_handle; comp_ctx->cqe.wr_id =3D wqe->hdr.wr_id; + comp_ctx->cqe.qp =3D qp_handle; + comp_ctx->cqe.opcode =3D IBV_WC_RECV; =20 rdma_backend_post_recv(&dev->backend_dev, &dev->rdma_dev_res, &qp->backend_qp, qp->qp_type, --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093781211137.64627235285377; Mon, 12 Nov 2018 23:23:01 -0800 (PST) Received: from localhost ([::1]:52350 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT2G-0001eZ-Al for importer@patchew.org; Tue, 13 Nov 2018 02:22:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51707) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuL-00045i-AY for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuH-00034h-Dv for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:45 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:43652) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuB-0002sE-SC for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:38 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DlUd078863; Tue, 13 Nov 2018 07:14:30 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nnwg18m7w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:30 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EOBV020261 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:24 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7EN2N017587; Tue, 13 Nov 2018 07:14:23 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:23 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=xDiBfg1kHb6oDTu4/E5lBynVjVnrP3sAMLNJA6kUlXs=; b=nusDJB0yRzPv1xgi/AqoMBN0S0T1Gt2xqpgRAc6T2mhDlGjRIEf7nC4y8JxbkOyt1chE YZTqgh6GTOYhBk/0RJPO9oQApPOGYv6yjb9ptOifuI9VORG903zZd3OYuq/quasT67uV 61CIdZVyQ9ogCtE3sCpHWTJ50FO2DUlV+h6A7SRkOAN2uKNK6Lk30rCOtkKrzyuQ/E/I 0DU3McdksV8F+3YH1DelBjutSWkIbOV9YqWKVM4PgqK9F9yDzj6ySFw3/HjW3soc1MD9 7Lx9IlBUudwD6MZFOQVDIQtj3I6ph4sNL8/llub6Bh9dGgKKUUX7WS1lfOrpVJhemJKd Ug== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:57 +0200 Message-Id: <20181113071336.6242-9-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=783 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 08/23] hw/pvrdma: Set the correct opcode for recv completion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The function pvrdma_post_cqe populates CQE entry with opcode from the given completion element. For receive operation value was not set. Fix it by setting it to IBV_WC_RECV. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_qp_ops.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 762700a205..7b0f440fda 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -196,8 +196,9 @@ int pvrdma_qp_recv(PVRDMADev *dev, uint32_t qp_handle) comp_ctx =3D g_malloc(sizeof(CompHandlerCtx)); comp_ctx->dev =3D dev; comp_ctx->cq_handle =3D qp->recv_cq_handle; - comp_ctx->cqe.qp =3D qp_handle; comp_ctx->cqe.wr_id =3D wqe->hdr.wr_id; + comp_ctx->cqe.qp =3D qp_handle; + comp_ctx->cqe.opcode =3D IBV_WC_RECV; =20 rdma_backend_post_recv(&dev->backend_dev, &dev->rdma_dev_res, &qp->backend_qp, qp->qp_type, --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094333543810.3942400731027; Mon, 12 Nov 2018 23:32:13 -0800 (PST) Received: from localhost ([::1]:52400 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTBE-0001GT-B2 for importer@patchew.org; Tue, 13 Nov 2018 02:32:12 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52225) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvK-0004xq-DU for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvH-0004QQ-97 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:46 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44478) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvG-0004Ne-RA for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:43 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Ep8v079429; Tue, 13 Nov 2018 07:15:40 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nnwg18mbw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:40 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Fe2Z018481 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:40 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7FdUu018041; Tue, 13 Nov 2018 07:15:39 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:39 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=E/PSQ9TPJmhF1T4H9YbFyqfTDOJqkPZBM5hsfVQ46fE=; b=XY3R8LEjOQW0rvMWR19ErYTD+FMlavR1cBRGVM61nRlc7WHJtDtmwgq7fewmTGO31xF7 yreZTqZcjX3r2K7hCmPTIEjkb2Slec76ie0vbG7zpwncqJSkUOxyO4T+xZ6Qgn7JinG3 miNkW2v6eUWijpcnbFrBIq35OZIVmAntTw4rVZwiPJmJCVcYydbjKRcNIYE1Im3KU+Np lhpe/rKXOwXHMbXBomGFouxEwnZEKZpkGFBHFkdWXXTZeNhzf9GHyWd8kNT1B4o75VM+ XrjPTUV4g2xDw4p/KB5lAfB0FtFKcN5qJxi4Y+owVoQep7ThzvdlRhI3+2uvIk4wuCoS 4Q== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:22 +0200 Message-Id: <20181113071336.6242-34-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=909 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 09/23] hw/pvrdma: Set the correct opcode for send completion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" opcode for WC should be set by the device and not taken from work element. Signed-off-by: Yuval Shaia --- hw/rdma/vmw/pvrdma_qp_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 7b0f440fda..3388be1926 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -154,7 +154,7 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) comp_ctx->cq_handle =3D qp->send_cq_handle; comp_ctx->cqe.wr_id =3D wqe->hdr.wr_id; comp_ctx->cqe.qp =3D qp_handle; - comp_ctx->cqe.opcode =3D wqe->hdr.opcode; + comp_ctx->cqe.opcode =3D IBV_WC_SEND; =20 rdma_backend_post_send(&dev->backend_dev, &qp->backend_qp, qp->qp_= type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.nu= m_sge, --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093591581435.103772307098; Mon, 12 Nov 2018 23:19:51 -0800 (PST) Received: from localhost ([::1]:52333 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSzG-0007bm-EH for importer@patchew.org; Tue, 13 Nov 2018 02:19:50 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51698) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuJ-00044Y-6K for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuD-0002zt-N7 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:41 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:59482) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSu8-0002pq-S8 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:33 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EShB046231; Tue, 13 Nov 2018 07:14:28 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2120.oracle.com with ESMTP id 2nnw6egpnu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:28 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7ERVn015242 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:27 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EQ1f011763; Tue, 13 Nov 2018 07:14:26 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:26 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=E/PSQ9TPJmhF1T4H9YbFyqfTDOJqkPZBM5hsfVQ46fE=; b=jyTIJUNashD3x6u1PUpXitubWjHmOHJyzW5Cbw47efWNB1T16/z1EwymVfdIQb22P52n C0yI6u52q4qwm7PCY9dlBcyH2R3IPpdgxcY4dMU+S4nEfXc+xuVL9EPtL60UW5ILHAdl W76imgFzgQaX0ocTAkR2ltN37Z1knvuH1/7CXbxJ+k+eI1BxZDSggPepELrcytu71sIl vuYO8W3RUyEn3EV7qMFuJyzN15e1wIM6R/oHAJTqbfF/D2SfvCCWPkRqexMw6h/5XHfe KuMyMID5FGcJzyCuYaXWW4fDXgjXQp2xNKnJvHtwqWcXWZOHSBRFBTgANWkFmIa7FBHe rQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:58 +0200 Message-Id: <20181113071336.6242-10-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=909 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 09/23] hw/pvrdma: Set the correct opcode for send completion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" opcode for WC should be set by the device and not taken from work element. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_qp_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 7b0f440fda..3388be1926 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -154,7 +154,7 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) comp_ctx->cq_handle =3D qp->send_cq_handle; comp_ctx->cqe.wr_id =3D wqe->hdr.wr_id; comp_ctx->cqe.qp =3D qp_handle; - comp_ctx->cqe.opcode =3D wqe->hdr.opcode; + comp_ctx->cqe.opcode =3D IBV_WC_SEND; =20 rdma_backend_post_send(&dev->backend_dev, &qp->backend_qp, qp->qp_= type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.nu= m_sge, --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093954731267.5462312201165; Mon, 12 Nov 2018 23:25:54 -0800 (PST) Received: from localhost ([::1]:52365 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT57-0004UJ-Iz for importer@patchew.org; Tue, 13 Nov 2018 02:25:53 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51818) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuS-0004BQ-Kb for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuP-0003FB-SI for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:52 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:44876) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuP-0002yf-HD for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:49 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DsEo037311; Tue, 13 Nov 2018 07:14:35 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2nnwc0gn08-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:35 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EUHA015381 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:30 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7ETu4011779; Tue, 13 Nov 2018 07:14:29 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:29 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=1PabdQpHYcYZtlqyD2tQL2JnpPCGGBG24Ep0JYOllsY=; b=Sq/r8fX1HkpZbuYPlmqcjbAL6CgSFMJkRk2IVu8oB4rXgvIschw3cqCrbO6WLHiQbxQ1 sGJIpjuTxtpg8w4XGk/HWz82D8O9U0lNMjxZDq9jF8Kru5dsNTgZg2xju8mbnoifiNt5 chqJ1JiyuB12+y3MrWxn16coXeXxrayAUsCszK7ZppbC9/86ccR1+AkVckrbv7D2ID7a Dp2iiaQeGPbhBrNs3uvBLBNVHl2CiRMaJWx0E4dv2ocpkJEdqix1b9kQy6p7NUwv03kP 42GdPUrk/14Ic/rEAkHQLMGzo4527y2A6pfAVK1o+8QJUAVXXOrtWbzDloNfPkSB34mo Yg== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:12:59 +0200 Message-Id: <20181113071336.6242-11-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 10/23] json: Define new QMP message for pvrdma X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" pvrdma requires that the same GID attached to it will be attached to the backend device in the host. A new QMP messages is defined so pvrdma device can broadcast any change made to its GID table. This event is captured by libvirt which in turn will update the GID table in the backend device. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- MAINTAINERS | 1 + Makefile | 3 ++- Makefile.objs | 4 ++++ qapi/qapi-schema.json | 1 + qapi/rdma.json | 38 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 46 insertions(+), 1 deletion(-) create mode 100644 qapi/rdma.json diff --git a/MAINTAINERS b/MAINTAINERS index e087d58ac6..a149f68a8f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2232,6 +2232,7 @@ F: hw/rdma/* F: hw/rdma/vmw/* F: docs/pvrdma.txt F: contrib/rdmacm-mux/* +F: qapi/rdma.json =20 Build and test automation ------------------------- diff --git a/Makefile b/Makefile index 94072776ff..db4ce60ee5 100644 --- a/Makefile +++ b/Makefile @@ -599,7 +599,8 @@ qapi-modules =3D $(SRC_PATH)/qapi/qapi-schema.json $(SR= C_PATH)/qapi/common.json \ $(SRC_PATH)/qapi/tpm.json \ $(SRC_PATH)/qapi/trace.json \ $(SRC_PATH)/qapi/transaction.json \ - $(SRC_PATH)/qapi/ui.json + $(SRC_PATH)/qapi/ui.json \ + $(SRC_PATH)/qapi/rdma.json =20 qapi/qapi-builtin-types.c qapi/qapi-builtin-types.h \ qapi/qapi-types.c qapi/qapi-types.h \ diff --git a/Makefile.objs b/Makefile.objs index cc7df3ad80..76d8028f2f 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -21,6 +21,7 @@ util-obj-y +=3D qapi/qapi-types-tpm.o util-obj-y +=3D qapi/qapi-types-trace.o util-obj-y +=3D qapi/qapi-types-transaction.o util-obj-y +=3D qapi/qapi-types-ui.o +util-obj-y +=3D qapi/qapi-types-rdma.o util-obj-y +=3D qapi/qapi-builtin-visit.o util-obj-y +=3D qapi/qapi-visit.o util-obj-y +=3D qapi/qapi-visit-block-core.o @@ -40,6 +41,7 @@ util-obj-y +=3D qapi/qapi-visit-tpm.o util-obj-y +=3D qapi/qapi-visit-trace.o util-obj-y +=3D qapi/qapi-visit-transaction.o util-obj-y +=3D qapi/qapi-visit-ui.o +util-obj-y +=3D qapi/qapi-visit-rdma.o util-obj-y +=3D qapi/qapi-events.o util-obj-y +=3D qapi/qapi-events-block-core.o util-obj-y +=3D qapi/qapi-events-block.o @@ -58,6 +60,7 @@ util-obj-y +=3D qapi/qapi-events-tpm.o util-obj-y +=3D qapi/qapi-events-trace.o util-obj-y +=3D qapi/qapi-events-transaction.o util-obj-y +=3D qapi/qapi-events-ui.o +util-obj-y +=3D qapi/qapi-events-rdma.o util-obj-y +=3D qapi/qapi-introspect.o =20 chardev-obj-y =3D chardev/ @@ -155,6 +158,7 @@ common-obj-y +=3D qapi/qapi-commands-tpm.o common-obj-y +=3D qapi/qapi-commands-trace.o common-obj-y +=3D qapi/qapi-commands-transaction.o common-obj-y +=3D qapi/qapi-commands-ui.o +common-obj-y +=3D qapi/qapi-commands-rdma.o common-obj-y +=3D qapi/qapi-introspect.o common-obj-y +=3D qmp.o hmp.o endif diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json index 65b6dc2f6f..a650d80f83 100644 --- a/qapi/qapi-schema.json +++ b/qapi/qapi-schema.json @@ -94,3 +94,4 @@ { 'include': 'trace.json' } { 'include': 'introspect.json' } { 'include': 'misc.json' } +{ 'include': 'rdma.json' } diff --git a/qapi/rdma.json b/qapi/rdma.json new file mode 100644 index 0000000000..804c68ab36 --- /dev/null +++ b/qapi/rdma.json @@ -0,0 +1,38 @@ +# -*- Mode: Python -*- +# + +## +# =3D RDMA device +## + +## +# @RDMA_GID_STATUS_CHANGED: +# +# Emitted when guest driver adds/deletes GID to/from device +# +# @netdev: RoCE Network Device name - char * +# +# @gid-status: Add or delete indication - bool +# +# @subnet-prefix: Subnet Prefix - uint64 +# +# @interface-id : Interface ID - uint64 +# +# Since: 3.2 +# +# Example: +# +# <- {"timestamp": {"seconds": 1541579657, "microseconds": 986760}, +# "event": "RDMA_GID_STATUS_CHANGED", +# "data": +# {"netdev": "bridge0", +# "interface-id": 15880512517475447892, +# "gid-status": true, +# "subnet-prefix": 33022}} +# +## +{ 'event': 'RDMA_GID_STATUS_CHANGED', + 'data': { 'netdev' : 'str', + 'gid-status' : 'bool', + 'subnet-prefix' : 'uint64', + 'interface-id' : 'uint64' } } --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094538141819.1039718861722; Mon, 12 Nov 2018 23:35:38 -0800 (PST) Received: from localhost ([::1]:52412 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTES-0003zB-DI for importer@patchew.org; Tue, 13 Nov 2018 02:35:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52245) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvL-0004yz-Db for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvJ-0004TF-Ap for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:47 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60482) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvJ-0004S2-00 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:45 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7E2Gl046085; Tue, 13 Nov 2018 07:15:43 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2120.oracle.com with ESMTP id 2nnw6egpu3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:43 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FgFG014256 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:42 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Fgnd028783; Tue, 13 Nov 2018 07:15:42 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:42 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=1PabdQpHYcYZtlqyD2tQL2JnpPCGGBG24Ep0JYOllsY=; b=bBrAT7cZ7A+eVC6RyMgDEuIQVHch93/mzDvX8F/yRLoHotPo89Trk+SpuPRj+plaXTuV IAXcP9hVkv2W1B7EpQvWdAyVefwjH3SgwbO8kY6+/fd5Ff3rt/nFVVGlCF/4fkZR86W5 pD2RwkQGODXi9/huTlF7E+Tq+lXH9JcqnaKXeNPi7N9w5vFChbcbKLl3DgLb7w1el4tT g14PBEcebzIw/srF5u2uFXQ1aVXIYyplNRWEzF4X9WgkXRagRsZD7oOG31xiWBKUNAmr 6yhFTL2+HHhyImjQ0IUhfpYMDBMyqFHpAd0JejASpMl+Efk7hxDVDtvwZwAUYNDUk+ye /A== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:23 +0200 Message-Id: <20181113071336.6242-35-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 10/23] json: Define new QMP message for pvrdma X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" pvrdma requires that the same GID attached to it will be attached to the backend device in the host. A new QMP messages is defined so pvrdma device can broadcast any change made to its GID table. This event is captured by libvirt which in turn will update the GID table in the backend device. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- MAINTAINERS | 1 + Makefile | 3 ++- Makefile.objs | 4 ++++ qapi/qapi-schema.json | 1 + qapi/rdma.json | 38 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 46 insertions(+), 1 deletion(-) create mode 100644 qapi/rdma.json diff --git a/MAINTAINERS b/MAINTAINERS index e087d58ac6..a149f68a8f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2232,6 +2232,7 @@ F: hw/rdma/* F: hw/rdma/vmw/* F: docs/pvrdma.txt F: contrib/rdmacm-mux/* +F: qapi/rdma.json =20 Build and test automation ------------------------- diff --git a/Makefile b/Makefile index 94072776ff..db4ce60ee5 100644 --- a/Makefile +++ b/Makefile @@ -599,7 +599,8 @@ qapi-modules =3D $(SRC_PATH)/qapi/qapi-schema.json $(SR= C_PATH)/qapi/common.json \ $(SRC_PATH)/qapi/tpm.json \ $(SRC_PATH)/qapi/trace.json \ $(SRC_PATH)/qapi/transaction.json \ - $(SRC_PATH)/qapi/ui.json + $(SRC_PATH)/qapi/ui.json \ + $(SRC_PATH)/qapi/rdma.json =20 qapi/qapi-builtin-types.c qapi/qapi-builtin-types.h \ qapi/qapi-types.c qapi/qapi-types.h \ diff --git a/Makefile.objs b/Makefile.objs index cc7df3ad80..76d8028f2f 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -21,6 +21,7 @@ util-obj-y +=3D qapi/qapi-types-tpm.o util-obj-y +=3D qapi/qapi-types-trace.o util-obj-y +=3D qapi/qapi-types-transaction.o util-obj-y +=3D qapi/qapi-types-ui.o +util-obj-y +=3D qapi/qapi-types-rdma.o util-obj-y +=3D qapi/qapi-builtin-visit.o util-obj-y +=3D qapi/qapi-visit.o util-obj-y +=3D qapi/qapi-visit-block-core.o @@ -40,6 +41,7 @@ util-obj-y +=3D qapi/qapi-visit-tpm.o util-obj-y +=3D qapi/qapi-visit-trace.o util-obj-y +=3D qapi/qapi-visit-transaction.o util-obj-y +=3D qapi/qapi-visit-ui.o +util-obj-y +=3D qapi/qapi-visit-rdma.o util-obj-y +=3D qapi/qapi-events.o util-obj-y +=3D qapi/qapi-events-block-core.o util-obj-y +=3D qapi/qapi-events-block.o @@ -58,6 +60,7 @@ util-obj-y +=3D qapi/qapi-events-tpm.o util-obj-y +=3D qapi/qapi-events-trace.o util-obj-y +=3D qapi/qapi-events-transaction.o util-obj-y +=3D qapi/qapi-events-ui.o +util-obj-y +=3D qapi/qapi-events-rdma.o util-obj-y +=3D qapi/qapi-introspect.o =20 chardev-obj-y =3D chardev/ @@ -155,6 +158,7 @@ common-obj-y +=3D qapi/qapi-commands-tpm.o common-obj-y +=3D qapi/qapi-commands-trace.o common-obj-y +=3D qapi/qapi-commands-transaction.o common-obj-y +=3D qapi/qapi-commands-ui.o +common-obj-y +=3D qapi/qapi-commands-rdma.o common-obj-y +=3D qapi/qapi-introspect.o common-obj-y +=3D qmp.o hmp.o endif diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json index 65b6dc2f6f..a650d80f83 100644 --- a/qapi/qapi-schema.json +++ b/qapi/qapi-schema.json @@ -94,3 +94,4 @@ { 'include': 'trace.json' } { 'include': 'introspect.json' } { 'include': 'misc.json' } +{ 'include': 'rdma.json' } diff --git a/qapi/rdma.json b/qapi/rdma.json new file mode 100644 index 0000000000..804c68ab36 --- /dev/null +++ b/qapi/rdma.json @@ -0,0 +1,38 @@ +# -*- Mode: Python -*- +# + +## +# =3D RDMA device +## + +## +# @RDMA_GID_STATUS_CHANGED: +# +# Emitted when guest driver adds/deletes GID to/from device +# +# @netdev: RoCE Network Device name - char * +# +# @gid-status: Add or delete indication - bool +# +# @subnet-prefix: Subnet Prefix - uint64 +# +# @interface-id : Interface ID - uint64 +# +# Since: 3.2 +# +# Example: +# +# <- {"timestamp": {"seconds": 1541579657, "microseconds": 986760}, +# "event": "RDMA_GID_STATUS_CHANGED", +# "data": +# {"netdev": "bridge0", +# "interface-id": 15880512517475447892, +# "gid-status": true, +# "subnet-prefix": 33022}} +# +## +{ 'event': 'RDMA_GID_STATUS_CHANGED', + 'data': { 'netdev' : 'str', + 'gid-status' : 'bool', + 'subnet-prefix' : 'uint64', + 'interface-id' : 'uint64' } } --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095246822634.3715931751382; Mon, 12 Nov 2018 23:47:26 -0800 (PST) Received: from localhost ([::1]:52477 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTPx-0004lU-B3 for importer@patchew.org; Tue, 13 Nov 2018 02:47:25 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52314) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvS-000554-BA for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvM-0004Xp-Tg for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:54 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45826) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvM-0004Wi-G5 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:48 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DlPs037023; Tue, 13 Nov 2018 07:15:47 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2nnwc0gn5c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:46 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FjAk013298 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:46 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Fjjq028788; Tue, 13 Nov 2018 07:15:45 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:45 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=1X65f5TTbCC49qez/k0PtHaB1YEwFnmmC6uA5RcK8M0=; b=wvu9BIefeVFYoWm9SOAYMPj6aimJ5zFXNehtn0dz/T27g9cpd32laBzwMjy2RDTNBMfm N+UoG/ix3aLpF8V/lJvycoxMKvOiSc30xBS13cQgTmM0Tf69tMMv+EhjJL7rWpMWhl9H c8+2uXw672EPxiU+FCGFGpVeKy4QHgO62XujIodPRTtbILn93OYIESGXrprqVFx+XkQ4 49yVzWDxt4A9eIawFiU+SAv1YxCv3X6UmPSAW2YSvLb8PZ3QAFkembR71y8uynM025HX 3+zjZCW/eXetu96ORfw03e9UQ+Eob6JOExeNBlLnlsDzvi6VEOUiBNMPTUQE4dk6z+du ag== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:24 +0200 Message-Id: <20181113071336.6242-36-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 11/23] hw/pvrdma: Add support to allow guest to configure GID table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The control over the RDMA device's GID table is done by updating the device's Ethernet function addresses. Usually the first GID entry is determine by the MAC address, the second by the first IPv6 address and the third by the IPv4 address. Other entries can be added by adding more IP addresses. The opposite is the same, i.e. whenever an address is removed, the corresponding GID entry is removed. The process is done by the network and RDMA stacks. Whenever an address is added the ib_core driver is notified and calls the device driver add_gid function which in turn update the device. To support this in pvrdma device we need to hook into the create_bind and destroy_bind HW commands triggered by pvrdma driver in guest. Whenever a changed is made to the pvrdma device's GID table a special QMP messages is sent to be processed by libvirt to update the address of the backend Ethernet device. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_backend.c | 243 +++++++++++++++++++++++------------- hw/rdma/rdma_backend.h | 22 ++-- hw/rdma/rdma_backend_defs.h | 3 +- hw/rdma/rdma_rm.c | 104 ++++++++++++++- hw/rdma/rdma_rm.h | 17 ++- hw/rdma/rdma_rm_defs.h | 9 +- hw/rdma/rdma_utils.h | 15 +++ hw/rdma/vmw/pvrdma.h | 2 +- hw/rdma/vmw/pvrdma_cmd.c | 55 ++++---- hw/rdma/vmw/pvrdma_main.c | 25 +--- hw/rdma/vmw/pvrdma_qp_ops.c | 20 +++ 11 files changed, 370 insertions(+), 145 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 3eb0099f8d..5675504165 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -18,12 +18,14 @@ #include "qapi/error.h" #include "qapi/qmp/qlist.h" #include "qapi/qmp/qnum.h" +#include "qapi/qapi-events-rdma.h" =20 #include #include #include #include =20 +#include "contrib/rdmacm-mux/rdmacm-mux.h" #include "trace.h" #include "rdma_utils.h" #include "rdma_rm.h" @@ -300,11 +302,11 @@ static int build_host_sge_array(RdmaDeviceResources *= rdma_dev_res, return 0; } =20 -static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, - uint32_t num_sge) +static int mad_send(RdmaBackendDev *backend_dev, uint8_t sgid_idx, + union ibv_gid *sgid, struct ibv_sge *sge, uint32_t num= _sge) { - struct backend_umad umad =3D {0}; - char *hdr, *msg; + RdmaCmMuxMsg msg =3D {0}; + char *hdr, *data; int ret; =20 pr_dbg("num_sge=3D%d\n", num_sge); @@ -313,41 +315,50 @@ static int mad_send(RdmaBackendDev *backend_dev, stru= ct ibv_sge *sge, return -EINVAL; } =20 - umad.hdr.length =3D sge[0].length + sge[1].length; - pr_dbg("msg_len=3D%d\n", umad.hdr.length); + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_MAD; + memcpy(msg.hdr.sgid.raw, sgid->raw, sizeof(msg.hdr.sgid)); =20 - if (umad.hdr.length > sizeof(umad.mad)) { + msg.umad_len =3D sge[0].length + sge[1].length; + pr_dbg("umad_len=3D%d\n", msg.umad_len); + + if (msg.umad_len > sizeof(msg.umad.mad)) { return -ENOMEM; } =20 - umad.hdr.addr.qpn =3D htobe32(1); - umad.hdr.addr.grh_present =3D 1; - umad.hdr.addr.gid_index =3D backend_dev->backend_gid_idx; - memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); - umad.hdr.addr.hop_limit =3D 1; + msg.umad.hdr.addr.qpn =3D htobe32(1); + msg.umad.hdr.addr.grh_present =3D 1; + pr_dbg("sgid_idx=3D%d\n", sgid_idx); + pr_dbg("sgid=3D0x%llx\n", sgid->global.interface_id); + msg.umad.hdr.addr.gid_index =3D sgid_idx; + memcpy(msg.umad.hdr.addr.gid, sgid->raw, sizeof(msg.umad.hdr.addr.gid)= ); + msg.umad.hdr.addr.hop_limit =3D 1; =20 hdr =3D rdma_pci_dma_map(backend_dev->dev, sge[0].addr, sge[0].length); - msg =3D rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length); + data =3D rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length= ); + + pr_dbg_buf("mad_hdr", hdr, sge[0].length); + pr_dbg_buf("mad_data", data, sge[1].length); =20 - memcpy(&umad.mad[0], hdr, sge[0].length); - memcpy(&umad.mad[sge[0].length], msg, sge[1].length); + memcpy(&msg.umad.mad[0], hdr, sge[0].length); + memcpy(&msg.umad.mad[sge[0].length], data, sge[1].length); =20 - rdma_pci_dma_unmap(backend_dev->dev, msg, sge[1].length); + rdma_pci_dma_unmap(backend_dev->dev, data, sge[1].length); rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); =20 - ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, - sizeof(umad)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&m= sg, + sizeof(msg)); =20 pr_dbg("qemu_chr_fe_write=3D%d\n", ret); =20 - return (ret !=3D sizeof(umad)); + return (ret !=3D sizeof(msg)); } =20 void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, - union ibv_gid *dgid, uint32_t dqpn, - uint32_t dqkey, void *ctx) + uint8_t sgid_idx, union ibv_gid *sgid, + union ibv_gid *dgid, uint32_t dqpn, uint32_t d= qkey, + void *ctx) { BackendCtx *bctx; struct ibv_sge new_sge[MAX_SGE]; @@ -361,7 +372,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); - rc =3D mad_send(backend_dev, sge, num_sge); + rc =3D mad_send(backend_dev, sgid_idx, sgid, sge, num_sge); if (rc) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); } else { @@ -397,8 +408,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, } =20 if (qp_type =3D=3D IBV_QPT_UD) { - wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, - backend_dev->backend_gid_idx, dgid); + wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, sgid_idx, dgid); if (!wr.wr.ud.ah) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; @@ -703,9 +713,9 @@ int rdma_backend_qp_state_init(RdmaBackendDev *backend_= dev, RdmaBackendQP *qp, } =20 int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *= qp, - uint8_t qp_type, union ibv_gid *dgid, - uint32_t dqpn, uint32_t rq_psn, uint32_t qke= y, - bool use_qkey) + uint8_t qp_type, uint8_t sgid_idx, + union ibv_gid *dgid, uint32_t dqpn, + uint32_t rq_psn, uint32_t qkey, bool use_qke= y) { struct ibv_qp_attr attr =3D {0}; union ibv_gid ibv_gid =3D { @@ -717,13 +727,15 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend= _dev, RdmaBackendQP *qp, attr.qp_state =3D IBV_QPS_RTR; attr_mask =3D IBV_QP_STATE; =20 + qp->sgid_idx =3D sgid_idx; + switch (qp_type) { case IBV_QPT_RC: pr_dbg("dgid=3D0x%" PRIx64 ",%" PRIx64 "\n", be64_to_cpu(ibv_gid.global.subnet_prefix), be64_to_cpu(ibv_gid.global.interface_id)); pr_dbg("dqpn=3D0x%x\n", dqpn); - pr_dbg("sgid_idx=3D%d\n", backend_dev->backend_gid_idx); + pr_dbg("sgid_idx=3D%d\n", qp->sgid_idx); pr_dbg("sport_num=3D%d\n", backend_dev->port_num); pr_dbg("rq_psn=3D0x%x\n", rq_psn); =20 @@ -735,7 +747,7 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_d= ev, RdmaBackendQP *qp, attr.ah_attr.is_global =3D 1; attr.ah_attr.grh.hop_limit =3D 1; attr.ah_attr.grh.dgid =3D ibv_gid; - attr.ah_attr.grh.sgid_index =3D backend_dev->backend_gid_idx; + attr.ah_attr.grh.sgid_index =3D qp->sgid_idx; attr.rq_psn =3D rq_psn; =20 attr_mask |=3D IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN | @@ -744,8 +756,8 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_d= ev, RdmaBackendQP *qp, break; =20 case IBV_QPT_UD: + pr_dbg("qkey=3D0x%x\n", qkey); if (use_qkey) { - pr_dbg("qkey=3D0x%x\n", qkey); attr.qkey =3D qkey; attr_mask |=3D IBV_QP_QKEY; } @@ -861,13 +873,13 @@ static inline void build_mad_hdr(struct ibv_grh *grh,= union ibv_gid *sgid, grh->dgid =3D *my_gid; =20 pr_dbg("paylen=3D%d (net=3D0x%x)\n", paylen, grh->paylen); - pr_dbg("my_gid=3D0x%llx\n", my_gid->global.interface_id); - pr_dbg("gid=3D0x%llx\n", sgid->global.interface_id); + pr_dbg("dgid=3D0x%llx\n", my_gid->global.interface_id); + pr_dbg("sgid=3D0x%llx\n", sgid->global.interface_id); } =20 static inline int mad_can_receieve(void *opaque) { - return sizeof(struct backend_umad); + return sizeof(RdmaCmMuxMsg); } =20 static void mad_read(void *opaque, const uint8_t *buf, int size) @@ -877,13 +889,13 @@ static void mad_read(void *opaque, const uint8_t *buf= , int size) unsigned long cqe_ctx_id; BackendCtx *bctx; char *mad; - struct backend_umad *umad; + RdmaCmMuxMsg *msg; =20 - assert(size !=3D sizeof(umad)); - umad =3D (struct backend_umad *)buf; + assert(size !=3D sizeof(msg)); + msg =3D (RdmaCmMuxMsg *)buf; =20 pr_dbg("Got %d bytes\n", size); - pr_dbg("umad->hdr.length=3D%d\n", umad->hdr.length); + pr_dbg("umad_len=3D%d\n", msg->umad_len); =20 #ifdef PVRDMA_DEBUG struct umad_hdr *hdr =3D (struct umad_hdr *)&msg->umad.mad; @@ -913,15 +925,16 @@ static void mad_read(void *opaque, const uint8_t *buf= , int size) =20 mad =3D rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, bctx->sge.length); - if (!mad || bctx->sge.length < umad->hdr.length + MAD_HDR_SIZE) { + if (!mad || bctx->sge.length < msg->umad_len + MAD_HDR_SIZE) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, bctx->up_ctx); } else { + pr_dbg_buf("mad", msg->umad.mad, msg->umad_len); memset(mad, 0, bctx->sge.length); build_mad_hdr((struct ibv_grh *)mad, - (union ibv_gid *)&umad->hdr.addr.gid, - &backend_dev->gid, umad->hdr.length); - memcpy(&mad[MAD_HDR_SIZE], umad->mad, umad->hdr.length); + (union ibv_gid *)&msg->umad.hdr.addr.gid, &msg->hdr.= sgid, + msg->umad_len); + memcpy(&mad[MAD_HDR_SIZE], msg->umad.mad, msg->umad_len); rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); =20 comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); @@ -933,10 +946,10 @@ static void mad_read(void *opaque, const uint8_t *buf= , int size) =20 static int mad_init(RdmaBackendDev *backend_dev) { - struct backend_umad umad =3D {0}; int ret; =20 - if (!qemu_chr_fe_backend_connected(backend_dev->mad_chr_be)) { + ret =3D qemu_chr_fe_backend_connected(backend_dev->mad_chr_be); + if (!ret) { pr_dbg("Missing chardev for MAD multiplexer\n"); return -EIO; } @@ -944,14 +957,6 @@ static int mad_init(RdmaBackendDev *backend_dev) qemu_chr_fe_set_handlers(backend_dev->mad_chr_be, mad_can_receieve, mad_read, NULL, NULL, backend_dev, NULL, true= ); =20 - /* Register ourself */ - memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); - ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, - sizeof(umad.hdr)); - if (ret !=3D sizeof(umad.hdr)) { - pr_dbg("Fail to register to rdma_umadmux (%d)\n", ret); - } - qemu_mutex_init(&backend_dev->recv_mads_list.lock); backend_dev->recv_mads_list.list =3D qlist_new(); =20 @@ -988,23 +993,120 @@ static void mad_fini(RdmaBackendDev *backend_dev) qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); } =20 +int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, + union ibv_gid *gid) +{ + union ibv_gid sgid; + int ret; + int i =3D 0; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + do { + ret =3D ibv_query_gid(backend_dev->context, backend_dev->port_num,= i, + &sgid); + i++; + } while (!ret && (memcmp(&sgid, gid, sizeof(*gid)))); + + pr_dbg("gid_index=3D%d\n", i - 1); + + return ret ? ret : i - 1; +} + +int rdma_backend_add_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid) +{ + RdmaCmMuxMsg msg =3D {0}; + int ret; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_REG; + memcpy(msg.hdr.sgid.raw, gid->raw, sizeof(msg.hdr.sgid)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&m= sg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to register GID to rdma_umadmux (%d)\n", ret); + return -EIO; + } + + ret =3D qemu_chr_fe_read_all(backend_dev->mad_chr_be, (uint8_t *)&msg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to register GID to rdma_umadmux (%d)\n", ret); + return -EIO; + } + + if (msg.hdr.err_code !=3D RDMACM_MUX_ERR_CODE_OK) { + pr_dbg("Fail to register GID to rdma_umadmux (%d)\n", msg.hdr.err_= code); + return -EIO; + } + + qapi_event_send_rdma_gid_status_changed(ifname, true, + gid->global.subnet_prefix, + gid->global.interface_id); + + return ret; +} + +int rdma_backend_del_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid) +{ + RdmaCmMuxMsg msg =3D {0}; + int ret; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_UNREG; + memcpy(msg.hdr.sgid.raw, gid->raw, sizeof(msg.hdr.sgid)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&m= sg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to unregister GID from rdma_umadmux (%d)\n", ret); + return -EIO; + } + + ret =3D qemu_chr_fe_read_all(backend_dev->mad_chr_be, (uint8_t *)&msg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to unregister GID from rdma_umadmux (%d)\n", ret); + return -EIO; + } + + if (msg.hdr.err_code !=3D RDMACM_MUX_ERR_CODE_OK) { + pr_dbg("Fail to unregister GID from rdma_umadmux (%d)\n", + msg.hdr.err_code); + return -EIO; + } + + qapi_event_send_rdma_gid_status_changed(ifname, false, + gid->global.subnet_prefix, + gid->global.interface_id); + + return 0; +} + int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, - uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - CharBackend *mad_chr_be, Error **errp) + struct ibv_device_attr *dev_attr, CharBackend *mad_c= hr_be, + Error **errp) { int i; int ret =3D 0; int num_ibv_devices; struct ibv_device **dev_list; - struct ibv_port_attr port_attr; =20 memset(backend_dev, 0, sizeof(*backend_dev)); =20 backend_dev->dev =3D pdev; backend_dev->mad_chr_be =3D mad_chr_be; - backend_dev->backend_gid_idx =3D backend_gid_idx; backend_dev->port_num =3D port_num; backend_dev->rdma_dev_res =3D rdma_dev_res; =20 @@ -1041,9 +1143,8 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PC= IDevice *pdev, backend_dev->ib_dev =3D *dev_list; } =20 - pr_dbg("Using backend device %s, port %d, gid_idx %d\n", - ibv_get_device_name(backend_dev->ib_dev), - backend_dev->port_num, backend_dev->backend_gid_idx); + pr_dbg("Using backend device %s, port %d\n", + ibv_get_device_name(backend_dev->ib_dev), backend_dev->port_num= ); =20 backend_dev->context =3D ibv_open_device(backend_dev->ib_dev); if (!backend_dev->context) { @@ -1060,20 +1161,6 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, P= CIDevice *pdev, } pr_dbg("dev->backend_dev.channel=3D%p\n", backend_dev->channel); =20 - ret =3D ibv_query_port(backend_dev->context, backend_dev->port_num, - &port_attr); - if (ret) { - error_setg(errp, "Error %d from ibv_query_port", ret); - ret =3D -EIO; - goto out_destroy_comm_channel; - } - - if (backend_dev->backend_gid_idx >=3D port_attr.gid_tbl_len) { - error_setg(errp, "Invalid backend_gid_idx, should be less than %d", - port_attr.gid_tbl_len); - goto out_destroy_comm_channel; - } - ret =3D init_device_caps(backend_dev, dev_attr); if (ret) { error_setg(errp, "Failed to initialize device capabilities"); @@ -1081,18 +1168,6 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, P= CIDevice *pdev, goto out_destroy_comm_channel; } =20 - ret =3D ibv_query_gid(backend_dev->context, backend_dev->port_num, - backend_dev->backend_gid_idx, &backend_dev->gid); - if (ret) { - error_setg(errp, "Failed to query gid %d", - backend_dev->backend_gid_idx); - ret =3D -EIO; - goto out_destroy_comm_channel; - } - pr_dbg("subnet_prefix=3D0x%" PRIx64 "\n", - be64_to_cpu(backend_dev->gid.global.subnet_prefix)); - pr_dbg("interface_id=3D0x%" PRIx64 "\n", - be64_to_cpu(backend_dev->gid.global.interface_id)); =20 ret =3D mad_init(backend_dev); if (ret) { diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index fc83330251..59ad2b874b 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -28,11 +28,6 @@ enum ibv_special_qp_type { IBV_QPT_GSI =3D 1, }; =20 -static inline union ibv_gid *rdma_backend_gid(RdmaBackendDev *dev) -{ - return &dev->gid; -} - static inline uint32_t rdma_backend_qpn(const RdmaBackendQP *qp) { return qp->ibqp ? qp->ibqp->qp_num : 1; @@ -51,9 +46,15 @@ static inline uint32_t rdma_backend_mr_rkey(const RdmaBa= ckendMR *mr) int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, - uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - CharBackend *mad_chr_be, Error **errp); + struct ibv_device_attr *dev_attr, CharBackend *mad_c= hr_be, + Error **errp); void rdma_backend_fini(RdmaBackendDev *backend_dev); +int rdma_backend_add_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid); +int rdma_backend_del_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid); +int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, + union ibv_gid *gid); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); void rdma_backend_register_comp_handler(void (*handler)(int status, @@ -82,9 +83,9 @@ int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_= type, int rdma_backend_qp_state_init(RdmaBackendDev *backend_dev, RdmaBackendQP = *qp, uint8_t qp_type, uint32_t qkey); int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *= qp, - uint8_t qp_type, union ibv_gid *dgid, - uint32_t dqpn, uint32_t rq_psn, uint32_t qke= y, - bool use_qkey); + uint8_t qp_type, uint8_t sgid_idx, + union ibv_gid *dgid, uint32_t dqpn, + uint32_t rq_psn, uint32_t qkey, bool use_qke= y); int rdma_backend_qp_state_rts(RdmaBackendQP *qp, uint8_t qp_type, uint32_t sq_psn, uint32_t qkey, bool use_qke= y); int rdma_backend_query_qp(RdmaBackendQP *qp, struct ibv_qp_attr *attr, @@ -94,6 +95,7 @@ void rdma_backend_destroy_qp(RdmaBackendQP *qp); void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, + uint8_t sgid_idx, union ibv_gid *sgid, union ibv_gid *dgid, uint32_t dqpn, uint32_t d= qkey, void *ctx); void rdma_backend_post_recv(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 2a7e667075..ff8b2426a0 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -37,14 +37,12 @@ typedef struct RecvMadList { typedef struct RdmaBackendDev { struct ibv_device_attr dev_attr; RdmaBackendThread comp_thread; - union ibv_gid gid; PCIDevice *dev; RdmaDeviceResources *rdma_dev_res; struct ibv_device *ib_dev; struct ibv_context *context; struct ibv_comp_channel *channel; uint8_t port_num; - uint8_t backend_gid_idx; RecvMadList recv_mads_list; CharBackend *mad_chr_be; } RdmaBackendDev; @@ -66,6 +64,7 @@ typedef struct RdmaBackendCQ { typedef struct RdmaBackendQP { struct ibv_pd *ibpd; struct ibv_qp *ibqp; + uint8_t sgid_idx; } RdmaBackendQP; =20 #endif diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 4f10fcabcc..fe0979415d 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -391,7 +391,7 @@ out_dealloc_qp: } =20 int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backen= d_dev, - uint32_t qp_handle, uint32_t attr_mask, + uint32_t qp_handle, uint32_t attr_mask, uint8_t sgid= _idx, union ibv_gid *dgid, uint32_t dqpn, enum ibv_qp_state qp_state, uint32_t qkey, uint32_t rq_psn, uint32_t sq_psn) @@ -400,6 +400,7 @@ int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, Rdm= aBackendDev *backend_dev, int ret; =20 pr_dbg("qpn=3D0x%x\n", qp_handle); + pr_dbg("qkey=3D0x%x\n", qkey); =20 qp =3D rdma_rm_get_qp(dev_res, qp_handle); if (!qp) { @@ -430,9 +431,19 @@ int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, Rd= maBackendDev *backend_dev, } =20 if (qp->qp_state =3D=3D IBV_QPS_RTR) { + /* Get backend gid index */ + pr_dbg("Guest sgid_idx=3D%d\n", sgid_idx); + sgid_idx =3D rdma_rm_get_backend_gid_index(dev_res, backend_de= v, + sgid_idx); + if (sgid_idx <=3D 0) { /* TODO check also less than bk.max_sgi= d */ + pr_dbg("Fail to get bk sgid_idx for sgid_idx %d\n", sgid_i= dx); + return -EIO; + } + ret =3D rdma_backend_qp_state_rtr(backend_dev, &qp->backend_qp, - qp->qp_type, dgid, dqpn, rq_ps= n, - qkey, attr_mask & IBV_QP_QKEY); + qp->qp_type, sgid_idx, dgid, d= qpn, + rq_psn, qkey, + attr_mask & IBV_QP_QKEY); if (ret) { return -EIO; } @@ -523,11 +534,91 @@ void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev= _res, uint32_t cqe_ctx_id) res_tbl_dealloc(&dev_res->cqe_ctx_tbl, cqe_ctx_id); } =20 +int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, union ibv_gid *gid, int gid_idx) +{ + int rc; + + rc =3D rdma_backend_add_gid(backend_dev, ifname, gid); + if (rc <=3D 0) { + pr_dbg("Fail to add gid\n"); + return -EINVAL; + } + + memcpy(&dev_res->ports[0].gid_tbl[gid_idx].gid, gid, sizeof(*gid)); + + return 0; +} + +int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, int gid_idx) +{ + int rc; + + rc =3D rdma_backend_del_gid(backend_dev, ifname, + &dev_res->ports[0].gid_tbl[gid_idx].gid); + if (rc < 0) { + pr_dbg("Fail to delete gid\n"); + return -EINVAL; + } + + memset(dev_res->ports[0].gid_tbl[gid_idx].gid.raw, 0, + sizeof(dev_res->ports[0].gid_tbl[gid_idx].gid)); + dev_res->ports[0].gid_tbl[gid_idx].backend_gid_index =3D -1; + + return 0; +} + +int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, int sgid_id= x) +{ + if (unlikely(sgid_idx < 0 || sgid_idx > MAX_PORT_GIDS)) { + pr_dbg("Got invalid sgid_idx %d\n", sgid_idx); + return -EINVAL; + } + + if (unlikely(dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D= =3D -1)) { + dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D + rdma_backend_get_gid_index(backend_dev, + &dev_res->ports[0].gid_tbl[sgid_idx= ].gid); + } + + pr_dbg("backend_gid_index=3D%d\n", + dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index); + + return dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index; +} + static void destroy_qp_hash_key(gpointer data) { g_bytes_unref(data); } =20 +static void init_ports(RdmaDeviceResources *dev_res) +{ + int i, j; + + memset(dev_res->ports, 0, sizeof(dev_res->ports)); + + for (i =3D 0; i < MAX_PORTS; i++) { + dev_res->ports[i].state =3D IBV_PORT_DOWN; + for (j =3D 0; j < MAX_PORT_GIDS; j++) { + dev_res->ports[i].gid_tbl[j].backend_gid_index =3D -1; + } + } +} + +static void fini_ports(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, const char *ifname) +{ + int i; + + dev_res->ports[0].state =3D IBV_PORT_DOWN; + for (i =3D 0; i < MAX_PORT_GIDS; i++) { + rdma_rm_del_gid(dev_res, backend_dev, ifname, i); + } +} + int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev= _attr, Error **errp) { @@ -545,11 +636,16 @@ int rdma_rm_init(RdmaDeviceResources *dev_res, struct= ibv_device_attr *dev_attr, dev_attr->max_qp_wr, sizeof(void *)); res_tbl_init("UC", &dev_res->uc_tbl, MAX_UCS, sizeof(RdmaRmUC)); =20 + init_ports(dev_res); + return 0; } =20 -void rdma_rm_fini(RdmaDeviceResources *dev_res) +void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_de= v, + const char *ifname) { + fini_ports(dev_res, backend_dev, ifname); + res_tbl_free(&dev_res->uc_tbl); res_tbl_free(&dev_res->cqe_ctx_tbl); res_tbl_free(&dev_res->qp_tbl); diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index b4e04cc7b4..a7169b4e89 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -22,7 +22,8 @@ =20 int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev= _attr, Error **errp); -void rdma_rm_fini(RdmaDeviceResources *dev_res); +void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_de= v, + const char *ifname); =20 int rdma_rm_alloc_pd(RdmaDeviceResources *dev_res, RdmaBackendDev *backend= _dev, uint32_t *pd_handle, uint32_t ctx_handle); @@ -55,7 +56,7 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uint32= _t pd_handle, uint32_t recv_cq_handle, void *opaque, uint32_t *qpn); RdmaRmQP *rdma_rm_get_qp(RdmaDeviceResources *dev_res, uint32_t qpn); int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backen= d_dev, - uint32_t qp_handle, uint32_t attr_mask, + uint32_t qp_handle, uint32_t attr_mask, uint8_t sgid= _idx, union ibv_gid *dgid, uint32_t dqpn, enum ibv_qp_state qp_state, uint32_t qkey, uint32_t rq_psn, uint32_t sq_psn); @@ -69,4 +70,16 @@ int rdma_rm_alloc_cqe_ctx(RdmaDeviceResources *dev_res, = uint32_t *cqe_ctx_id, void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_i= d); void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ct= x_id); =20 +int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, union ibv_gid *gid, int gid_idx); +int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, int gid_idx); +int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, int sgid_id= x); +static inline union ibv_gid *rdma_rm_get_gid(RdmaDeviceResources *dev_res, + int sgid_idx) +{ + return &dev_res->ports[0].gid_tbl[sgid_idx].gid; +} + #endif diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 9b399063d3..7b3435f991 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -19,7 +19,7 @@ #include "rdma_backend_defs.h" =20 #define MAX_PORTS 1 -#define MAX_PORT_GIDS 1 +#define MAX_PORT_GIDS 255 #define MAX_GIDS MAX_PORT_GIDS #define MAX_PORT_PKEYS 1 #define MAX_PKEYS MAX_PORT_PKEYS @@ -86,8 +86,13 @@ typedef struct RdmaRmQP { enum ibv_qp_state qp_state; } RdmaRmQP; =20 +typedef struct RdmaRmGid { + union ibv_gid gid; + int backend_gid_index; +} RdmaRmGid; + typedef struct RdmaRmPort { - union ibv_gid gid_tbl[MAX_PORT_GIDS]; + RdmaRmGid gid_tbl[MAX_PORT_GIDS]; enum ibv_port_state state; } RdmaRmPort; =20 diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index 04c7c2ef5b..989db249ef 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -20,6 +20,7 @@ #include "qemu/osdep.h" #include "hw/pci/pci.h" #include "sysemu/dma.h" +#include "stdio.h" =20 #define pr_info(fmt, ...) \ fprintf(stdout, "%s: %-20s (%3d): " fmt, "rdma", __func__, __LINE__,\ @@ -40,9 +41,23 @@ extern unsigned long pr_dbg_cnt; #define pr_dbg(fmt, ...) \ fprintf(stdout, "%lx %ld: %-20s (%3d): " fmt, pthread_self(), pr_dbg_c= nt++, \ __func__, __LINE__, ## __VA_ARGS__) + +#define pr_dbg_buf(title, buf, len) \ +{ \ + char *b =3D g_malloc0(len * 3 + 1); \ + char b1[4]; \ + for (int i =3D 0; i < len; i++) { \ + sprintf(b1, "%.2X ", buf[i] & 0x000000FF); \ + strcat(b, b1); \ + } \ + pr_dbg("%s (%d): %s\n", title, len, b); \ + g_free(b); \ +} + #else #define init_pr_dbg(void) #define pr_dbg(fmt, ...) +#define pr_dbg_buf(title, buf, len) #endif =20 void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index 15c3f28b86..b019cb843a 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -79,8 +79,8 @@ typedef struct PVRDMADev { int interrupt_mask; struct ibv_device_attr dev_attr; uint64_t node_guid; + char *backend_eth_device_name; char *backend_device_name; - uint8_t backend_gid_idx; uint8_t backend_port_num; RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 57d6f41ae6..a334f6205e 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -504,13 +504,16 @@ static int modify_qp(PVRDMADev *dev, union pvrdma_cmd= _req *req, rsp->hdr.response =3D cmd->hdr.response; rsp->hdr.ack =3D PVRDMA_CMD_MODIFY_QP_RESP; =20 - rsp->hdr.err =3D rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_d= ev, - cmd->qp_handle, cmd->attr_mask, - (union ibv_gid *)&cmd->attrs.ah_attr.grh.= dgid, - cmd->attrs.dest_qp_num, - (enum ibv_qp_state)cmd->attrs.qp_state, - cmd->attrs.qkey, cmd->attrs.rq_psn, - cmd->attrs.sq_psn); + /* No need to verify sgid_index since it is u8 */ + + rsp->hdr.err =3D + rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_dev, cmd->qp_h= andle, + cmd->attr_mask, cmd->attrs.ah_attr.grh.sgid_inde= x, + (union ibv_gid *)&cmd->attrs.ah_attr.grh.dgid, + cmd->attrs.dest_qp_num, + (enum ibv_qp_state)cmd->attrs.qp_state, + cmd->attrs.qkey, cmd->attrs.rq_psn, + cmd->attrs.sq_psn); =20 pr_dbg("ret=3D%d\n", rsp->hdr.err); return rsp->hdr.err; @@ -570,10 +573,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cm= d_req *req, union pvrdma_cmd_resp *rsp) { struct pvrdma_cmd_create_bind *cmd =3D &req->create_bind; -#ifdef PVRDMA_DEBUG - __be64 *subnet =3D (__be64 *)&cmd->new_gid[0]; - __be64 *if_id =3D (__be64 *)&cmd->new_gid[8]; -#endif + int rc; + union ibv_gid *gid =3D (union ibv_gid *)&cmd->new_gid; =20 pr_dbg("index=3D%d\n", cmd->index); =20 @@ -582,19 +583,24 @@ static int create_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, } =20 pr_dbg("gid[%d]=3D0x%llx,0x%llx\n", cmd->index, - (long long unsigned int)be64_to_cpu(*subnet), - (long long unsigned int)be64_to_cpu(*if_id)); + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); =20 - /* Driver forces to one port only */ - memcpy(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, &cmd->new_g= id, - sizeof(cmd->new_gid)); + rc =3D rdma_rm_add_gid(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name, gid, cmd->index); + if (rc < 0) { + return -EINVAL; + } =20 /* TODO: Since drivers stores node_guid at load_dsr phase then this * assignment is not relevant, i need to figure out a way how to * retrieve MAC of our netdev */ - dev->node_guid =3D dev->rdma_dev_res.ports[0].gid_tbl[0].global.interf= ace_id; - pr_dbg("dev->node_guid=3D0x%llx\n", - (long long unsigned int)be64_to_cpu(dev->node_guid)); + if (!cmd->index) { + dev->node_guid =3D + dev->rdma_dev_res.ports[0].gid_tbl[0].gid.global.interface_id; + pr_dbg("dev->node_guid=3D0x%llx\n", + (long long unsigned int)be64_to_cpu(dev->node_guid)); + } =20 return 0; } @@ -602,6 +608,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd= _req *req, static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp) { + int rc; + struct pvrdma_cmd_destroy_bind *cmd =3D &req->destroy_bind; =20 pr_dbg("index=3D%d\n", cmd->index); @@ -610,8 +618,13 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, return -EINVAL; } =20 - memset(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, 0, - sizeof(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw)); + rc =3D rdma_rm_del_gid(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name, cmd->index); + + if (rc < 0) { + rsp->hdr.err =3D rc; + goto out; + } =20 return 0; } diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index fc2abd34af..ac8c092db0 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -36,9 +36,9 @@ #include "pvrdma_qp_ops.h" =20 static Property pvrdma_dev_properties[] =3D { - DEFINE_PROP_STRING("backend-dev", PVRDMADev, backend_device_name), - DEFINE_PROP_UINT8("backend-port", PVRDMADev, backend_port_num, 1), - DEFINE_PROP_UINT8("backend-gid-idx", PVRDMADev, backend_gid_idx, 0), + DEFINE_PROP_STRING("netdev", PVRDMADev, backend_eth_device_name), + DEFINE_PROP_STRING("ibdev", PVRDMADev, backend_device_name), + DEFINE_PROP_UINT8("ibport", PVRDMADev, backend_port_num, 1), DEFINE_PROP_UINT64("dev-caps-max-mr-size", PVRDMADev, dev_attr.max_mr_= size, MAX_MR_SIZE), DEFINE_PROP_INT32("dev-caps-max-qp", PVRDMADev, dev_attr.max_qp, MAX_Q= P), @@ -276,17 +276,6 @@ static void init_dsr_dev_caps(PVRDMADev *dev) pr_dbg("Initialized\n"); } =20 -static void init_ports(PVRDMADev *dev, Error **errp) -{ - int i; - - memset(dev->rdma_dev_res.ports, 0, sizeof(dev->rdma_dev_res.ports)); - - for (i =3D 0; i < MAX_PORTS; i++) { - dev->rdma_dev_res.ports[i].state =3D IBV_PORT_DOWN; - } -} - static void uninit_msix(PCIDevice *pdev, int used_vectors) { PVRDMADev *dev =3D PVRDMA_DEV(pdev); @@ -335,7 +324,8 @@ static void pvrdma_fini(PCIDevice *pdev) =20 pvrdma_qp_ops_fini(); =20 - rdma_rm_fini(&dev->rdma_dev_res); + rdma_rm_fini(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name); =20 rdma_backend_fini(&dev->backend_dev); =20 @@ -612,8 +602,7 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) =20 rc =3D rdma_backend_init(&dev->backend_dev, pdev, &dev->rdma_dev_res, dev->backend_device_name, dev->backend_port_num, - dev->backend_gid_idx, &dev->dev_attr, &dev->mad= _chr, - errp); + &dev->dev_attr, &dev->mad_chr, errp); if (rc) { goto out; } @@ -623,8 +612,6 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) goto out; } =20 - init_ports(dev, errp); - rc =3D pvrdma_qp_ops_init(); if (rc) { goto out; diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 3388be1926..2130824098 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -131,6 +131,8 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) RdmaRmQP *qp; PvrdmaSqWqe *wqe; PvrdmaRing *ring; + int sgid_idx; + union ibv_gid *sgid; =20 pr_dbg("qp_handle=3D0x%x\n", qp_handle); =20 @@ -156,8 +158,26 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) comp_ctx->cqe.qp =3D qp_handle; comp_ctx->cqe.opcode =3D IBV_WC_SEND; =20 + sgid =3D rdma_rm_get_gid(&dev->rdma_dev_res, wqe->hdr.wr.ud.av.gid= _index); + if (!sgid) { + pr_dbg("Fail to get gid for idx %d\n", wqe->hdr.wr.ud.av.gid_i= ndex); + return -EIO; + } + pr_dbg("sgid_id=3D%d, sgid=3D0x%llx\n", wqe->hdr.wr.ud.av.gid_inde= x, + sgid->global.interface_id); + + sgid_idx =3D rdma_rm_get_backend_gid_index(&dev->rdma_dev_res, + &dev->backend_dev, + wqe->hdr.wr.ud.av.gid_ind= ex); + if (sgid_idx <=3D 0) { + pr_dbg("Fail to get bk sgid_idx for sgid_idx %d\n", + wqe->hdr.wr.ud.av.gid_index); + return -EIO; + } + rdma_backend_post_send(&dev->backend_dev, &qp->backend_qp, qp->qp_= type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.nu= m_sge, + sgid_idx, sgid, (union ibv_gid *)wqe->hdr.wr.ud.av.dgid, wqe->hdr.wr.ud.remote_qpn, wqe->hdr.wr.ud.remote_qkey, comp_ctx); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 154209378558875.56001943061233; Mon, 12 Nov 2018 23:23:05 -0800 (PST) Received: from localhost ([::1]:52351 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT2O-0001kE-7J for importer@patchew.org; Tue, 13 Nov 2018 02:23:04 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51861) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuU-0004DK-GP for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuQ-0003Gp-IH for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:54 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:43696) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuP-0002xx-UN for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:50 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EENl079257; Tue, 13 Nov 2018 07:14:35 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nnwg18m84-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:34 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EX4G020468 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:33 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7EWb9017620; Tue, 13 Nov 2018 07:14:32 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:32 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=1X65f5TTbCC49qez/k0PtHaB1YEwFnmmC6uA5RcK8M0=; b=Q1GEUJm9gQSipLV53CUbvNcmOIMjMLqOAe4xdKGjvK4C3XOK3+nCOmN6ZUWCw9m1Q7S4 n+2TMlK871hwz7CDo5/a7NiEyfoqjikrY1/mJej11nQlKWywxQ7UJKvEi4zaoi4zgQbp S2aUNtrXKwxC/r7pc2SSZMPy//YK7+8qyX6XUQyYQ2v4jAw2UaTyyzwGDp1M5kdwZyDd DSHBplxVLkYgVvmxOxcRbEskd35Jy/ekZ8hzDFaGe6iTtLHgwfuzoguM9bmyaeMAGMBi PyXkcxmsS3sraPjad1HNigKPTXmBmYCMOtO9qwL0cGPOBGI22mvd7Cnwf7ny4WeamN4g Mw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:00 +0200 Message-Id: <20181113071336.6242-12-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 11/23] hw/pvrdma: Add support to allow guest to configure GID table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The control over the RDMA device's GID table is done by updating the device's Ethernet function addresses. Usually the first GID entry is determine by the MAC address, the second by the first IPv6 address and the third by the IPv4 address. Other entries can be added by adding more IP addresses. The opposite is the same, i.e. whenever an address is removed, the corresponding GID entry is removed. The process is done by the network and RDMA stacks. Whenever an address is added the ib_core driver is notified and calls the device driver add_gid function which in turn update the device. To support this in pvrdma device we need to hook into the create_bind and destroy_bind HW commands triggered by pvrdma driver in guest. Whenever a changed is made to the pvrdma device's GID table a special QMP messages is sent to be processed by libvirt to update the address of the backend Ethernet device. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_backend.c | 243 +++++++++++++++++++++++------------- hw/rdma/rdma_backend.h | 22 ++-- hw/rdma/rdma_backend_defs.h | 3 +- hw/rdma/rdma_rm.c | 104 ++++++++++++++- hw/rdma/rdma_rm.h | 17 ++- hw/rdma/rdma_rm_defs.h | 9 +- hw/rdma/rdma_utils.h | 15 +++ hw/rdma/vmw/pvrdma.h | 2 +- hw/rdma/vmw/pvrdma_cmd.c | 55 ++++---- hw/rdma/vmw/pvrdma_main.c | 25 +--- hw/rdma/vmw/pvrdma_qp_ops.c | 20 +++ 11 files changed, 370 insertions(+), 145 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 3eb0099f8d..5675504165 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -18,12 +18,14 @@ #include "qapi/error.h" #include "qapi/qmp/qlist.h" #include "qapi/qmp/qnum.h" +#include "qapi/qapi-events-rdma.h" =20 #include #include #include #include =20 +#include "contrib/rdmacm-mux/rdmacm-mux.h" #include "trace.h" #include "rdma_utils.h" #include "rdma_rm.h" @@ -300,11 +302,11 @@ static int build_host_sge_array(RdmaDeviceResources *= rdma_dev_res, return 0; } =20 -static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, - uint32_t num_sge) +static int mad_send(RdmaBackendDev *backend_dev, uint8_t sgid_idx, + union ibv_gid *sgid, struct ibv_sge *sge, uint32_t num= _sge) { - struct backend_umad umad =3D {0}; - char *hdr, *msg; + RdmaCmMuxMsg msg =3D {0}; + char *hdr, *data; int ret; =20 pr_dbg("num_sge=3D%d\n", num_sge); @@ -313,41 +315,50 @@ static int mad_send(RdmaBackendDev *backend_dev, stru= ct ibv_sge *sge, return -EINVAL; } =20 - umad.hdr.length =3D sge[0].length + sge[1].length; - pr_dbg("msg_len=3D%d\n", umad.hdr.length); + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_MAD; + memcpy(msg.hdr.sgid.raw, sgid->raw, sizeof(msg.hdr.sgid)); =20 - if (umad.hdr.length > sizeof(umad.mad)) { + msg.umad_len =3D sge[0].length + sge[1].length; + pr_dbg("umad_len=3D%d\n", msg.umad_len); + + if (msg.umad_len > sizeof(msg.umad.mad)) { return -ENOMEM; } =20 - umad.hdr.addr.qpn =3D htobe32(1); - umad.hdr.addr.grh_present =3D 1; - umad.hdr.addr.gid_index =3D backend_dev->backend_gid_idx; - memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); - umad.hdr.addr.hop_limit =3D 1; + msg.umad.hdr.addr.qpn =3D htobe32(1); + msg.umad.hdr.addr.grh_present =3D 1; + pr_dbg("sgid_idx=3D%d\n", sgid_idx); + pr_dbg("sgid=3D0x%llx\n", sgid->global.interface_id); + msg.umad.hdr.addr.gid_index =3D sgid_idx; + memcpy(msg.umad.hdr.addr.gid, sgid->raw, sizeof(msg.umad.hdr.addr.gid)= ); + msg.umad.hdr.addr.hop_limit =3D 1; =20 hdr =3D rdma_pci_dma_map(backend_dev->dev, sge[0].addr, sge[0].length); - msg =3D rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length); + data =3D rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length= ); + + pr_dbg_buf("mad_hdr", hdr, sge[0].length); + pr_dbg_buf("mad_data", data, sge[1].length); =20 - memcpy(&umad.mad[0], hdr, sge[0].length); - memcpy(&umad.mad[sge[0].length], msg, sge[1].length); + memcpy(&msg.umad.mad[0], hdr, sge[0].length); + memcpy(&msg.umad.mad[sge[0].length], data, sge[1].length); =20 - rdma_pci_dma_unmap(backend_dev->dev, msg, sge[1].length); + rdma_pci_dma_unmap(backend_dev->dev, data, sge[1].length); rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); =20 - ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, - sizeof(umad)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&m= sg, + sizeof(msg)); =20 pr_dbg("qemu_chr_fe_write=3D%d\n", ret); =20 - return (ret !=3D sizeof(umad)); + return (ret !=3D sizeof(msg)); } =20 void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, - union ibv_gid *dgid, uint32_t dqpn, - uint32_t dqkey, void *ctx) + uint8_t sgid_idx, union ibv_gid *sgid, + union ibv_gid *dgid, uint32_t dqpn, uint32_t d= qkey, + void *ctx) { BackendCtx *bctx; struct ibv_sge new_sge[MAX_SGE]; @@ -361,7 +372,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); - rc =3D mad_send(backend_dev, sge, num_sge); + rc =3D mad_send(backend_dev, sgid_idx, sgid, sge, num_sge); if (rc) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); } else { @@ -397,8 +408,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, } =20 if (qp_type =3D=3D IBV_QPT_UD) { - wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, - backend_dev->backend_gid_idx, dgid); + wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, sgid_idx, dgid); if (!wr.wr.ud.ah) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; @@ -703,9 +713,9 @@ int rdma_backend_qp_state_init(RdmaBackendDev *backend_= dev, RdmaBackendQP *qp, } =20 int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *= qp, - uint8_t qp_type, union ibv_gid *dgid, - uint32_t dqpn, uint32_t rq_psn, uint32_t qke= y, - bool use_qkey) + uint8_t qp_type, uint8_t sgid_idx, + union ibv_gid *dgid, uint32_t dqpn, + uint32_t rq_psn, uint32_t qkey, bool use_qke= y) { struct ibv_qp_attr attr =3D {0}; union ibv_gid ibv_gid =3D { @@ -717,13 +727,15 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend= _dev, RdmaBackendQP *qp, attr.qp_state =3D IBV_QPS_RTR; attr_mask =3D IBV_QP_STATE; =20 + qp->sgid_idx =3D sgid_idx; + switch (qp_type) { case IBV_QPT_RC: pr_dbg("dgid=3D0x%" PRIx64 ",%" PRIx64 "\n", be64_to_cpu(ibv_gid.global.subnet_prefix), be64_to_cpu(ibv_gid.global.interface_id)); pr_dbg("dqpn=3D0x%x\n", dqpn); - pr_dbg("sgid_idx=3D%d\n", backend_dev->backend_gid_idx); + pr_dbg("sgid_idx=3D%d\n", qp->sgid_idx); pr_dbg("sport_num=3D%d\n", backend_dev->port_num); pr_dbg("rq_psn=3D0x%x\n", rq_psn); =20 @@ -735,7 +747,7 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_d= ev, RdmaBackendQP *qp, attr.ah_attr.is_global =3D 1; attr.ah_attr.grh.hop_limit =3D 1; attr.ah_attr.grh.dgid =3D ibv_gid; - attr.ah_attr.grh.sgid_index =3D backend_dev->backend_gid_idx; + attr.ah_attr.grh.sgid_index =3D qp->sgid_idx; attr.rq_psn =3D rq_psn; =20 attr_mask |=3D IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN | @@ -744,8 +756,8 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_d= ev, RdmaBackendQP *qp, break; =20 case IBV_QPT_UD: + pr_dbg("qkey=3D0x%x\n", qkey); if (use_qkey) { - pr_dbg("qkey=3D0x%x\n", qkey); attr.qkey =3D qkey; attr_mask |=3D IBV_QP_QKEY; } @@ -861,13 +873,13 @@ static inline void build_mad_hdr(struct ibv_grh *grh,= union ibv_gid *sgid, grh->dgid =3D *my_gid; =20 pr_dbg("paylen=3D%d (net=3D0x%x)\n", paylen, grh->paylen); - pr_dbg("my_gid=3D0x%llx\n", my_gid->global.interface_id); - pr_dbg("gid=3D0x%llx\n", sgid->global.interface_id); + pr_dbg("dgid=3D0x%llx\n", my_gid->global.interface_id); + pr_dbg("sgid=3D0x%llx\n", sgid->global.interface_id); } =20 static inline int mad_can_receieve(void *opaque) { - return sizeof(struct backend_umad); + return sizeof(RdmaCmMuxMsg); } =20 static void mad_read(void *opaque, const uint8_t *buf, int size) @@ -877,13 +889,13 @@ static void mad_read(void *opaque, const uint8_t *buf= , int size) unsigned long cqe_ctx_id; BackendCtx *bctx; char *mad; - struct backend_umad *umad; + RdmaCmMuxMsg *msg; =20 - assert(size !=3D sizeof(umad)); - umad =3D (struct backend_umad *)buf; + assert(size !=3D sizeof(msg)); + msg =3D (RdmaCmMuxMsg *)buf; =20 pr_dbg("Got %d bytes\n", size); - pr_dbg("umad->hdr.length=3D%d\n", umad->hdr.length); + pr_dbg("umad_len=3D%d\n", msg->umad_len); =20 #ifdef PVRDMA_DEBUG struct umad_hdr *hdr =3D (struct umad_hdr *)&msg->umad.mad; @@ -913,15 +925,16 @@ static void mad_read(void *opaque, const uint8_t *buf= , int size) =20 mad =3D rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, bctx->sge.length); - if (!mad || bctx->sge.length < umad->hdr.length + MAD_HDR_SIZE) { + if (!mad || bctx->sge.length < msg->umad_len + MAD_HDR_SIZE) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, bctx->up_ctx); } else { + pr_dbg_buf("mad", msg->umad.mad, msg->umad_len); memset(mad, 0, bctx->sge.length); build_mad_hdr((struct ibv_grh *)mad, - (union ibv_gid *)&umad->hdr.addr.gid, - &backend_dev->gid, umad->hdr.length); - memcpy(&mad[MAD_HDR_SIZE], umad->mad, umad->hdr.length); + (union ibv_gid *)&msg->umad.hdr.addr.gid, &msg->hdr.= sgid, + msg->umad_len); + memcpy(&mad[MAD_HDR_SIZE], msg->umad.mad, msg->umad_len); rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); =20 comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); @@ -933,10 +946,10 @@ static void mad_read(void *opaque, const uint8_t *buf= , int size) =20 static int mad_init(RdmaBackendDev *backend_dev) { - struct backend_umad umad =3D {0}; int ret; =20 - if (!qemu_chr_fe_backend_connected(backend_dev->mad_chr_be)) { + ret =3D qemu_chr_fe_backend_connected(backend_dev->mad_chr_be); + if (!ret) { pr_dbg("Missing chardev for MAD multiplexer\n"); return -EIO; } @@ -944,14 +957,6 @@ static int mad_init(RdmaBackendDev *backend_dev) qemu_chr_fe_set_handlers(backend_dev->mad_chr_be, mad_can_receieve, mad_read, NULL, NULL, backend_dev, NULL, true= ); =20 - /* Register ourself */ - memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.g= id)); - ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&u= mad, - sizeof(umad.hdr)); - if (ret !=3D sizeof(umad.hdr)) { - pr_dbg("Fail to register to rdma_umadmux (%d)\n", ret); - } - qemu_mutex_init(&backend_dev->recv_mads_list.lock); backend_dev->recv_mads_list.list =3D qlist_new(); =20 @@ -988,23 +993,120 @@ static void mad_fini(RdmaBackendDev *backend_dev) qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); } =20 +int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, + union ibv_gid *gid) +{ + union ibv_gid sgid; + int ret; + int i =3D 0; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + do { + ret =3D ibv_query_gid(backend_dev->context, backend_dev->port_num,= i, + &sgid); + i++; + } while (!ret && (memcmp(&sgid, gid, sizeof(*gid)))); + + pr_dbg("gid_index=3D%d\n", i - 1); + + return ret ? ret : i - 1; +} + +int rdma_backend_add_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid) +{ + RdmaCmMuxMsg msg =3D {0}; + int ret; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_REG; + memcpy(msg.hdr.sgid.raw, gid->raw, sizeof(msg.hdr.sgid)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&m= sg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to register GID to rdma_umadmux (%d)\n", ret); + return -EIO; + } + + ret =3D qemu_chr_fe_read_all(backend_dev->mad_chr_be, (uint8_t *)&msg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to register GID to rdma_umadmux (%d)\n", ret); + return -EIO; + } + + if (msg.hdr.err_code !=3D RDMACM_MUX_ERR_CODE_OK) { + pr_dbg("Fail to register GID to rdma_umadmux (%d)\n", msg.hdr.err_= code); + return -EIO; + } + + qapi_event_send_rdma_gid_status_changed(ifname, true, + gid->global.subnet_prefix, + gid->global.interface_id); + + return ret; +} + +int rdma_backend_del_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid) +{ + RdmaCmMuxMsg msg =3D {0}; + int ret; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + msg.hdr.msg_type =3D RDMACM_MUX_MSG_TYPE_UNREG; + memcpy(msg.hdr.sgid.raw, gid->raw, sizeof(msg.hdr.sgid)); + ret =3D qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&m= sg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to unregister GID from rdma_umadmux (%d)\n", ret); + return -EIO; + } + + ret =3D qemu_chr_fe_read_all(backend_dev->mad_chr_be, (uint8_t *)&msg, + sizeof(msg)); + if (ret !=3D sizeof(msg)) { + pr_dbg("Fail to unregister GID from rdma_umadmux (%d)\n", ret); + return -EIO; + } + + if (msg.hdr.err_code !=3D RDMACM_MUX_ERR_CODE_OK) { + pr_dbg("Fail to unregister GID from rdma_umadmux (%d)\n", + msg.hdr.err_code); + return -EIO; + } + + qapi_event_send_rdma_gid_status_changed(ifname, false, + gid->global.subnet_prefix, + gid->global.interface_id); + + return 0; +} + int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, - uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - CharBackend *mad_chr_be, Error **errp) + struct ibv_device_attr *dev_attr, CharBackend *mad_c= hr_be, + Error **errp) { int i; int ret =3D 0; int num_ibv_devices; struct ibv_device **dev_list; - struct ibv_port_attr port_attr; =20 memset(backend_dev, 0, sizeof(*backend_dev)); =20 backend_dev->dev =3D pdev; backend_dev->mad_chr_be =3D mad_chr_be; - backend_dev->backend_gid_idx =3D backend_gid_idx; backend_dev->port_num =3D port_num; backend_dev->rdma_dev_res =3D rdma_dev_res; =20 @@ -1041,9 +1143,8 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PC= IDevice *pdev, backend_dev->ib_dev =3D *dev_list; } =20 - pr_dbg("Using backend device %s, port %d, gid_idx %d\n", - ibv_get_device_name(backend_dev->ib_dev), - backend_dev->port_num, backend_dev->backend_gid_idx); + pr_dbg("Using backend device %s, port %d\n", + ibv_get_device_name(backend_dev->ib_dev), backend_dev->port_num= ); =20 backend_dev->context =3D ibv_open_device(backend_dev->ib_dev); if (!backend_dev->context) { @@ -1060,20 +1161,6 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, P= CIDevice *pdev, } pr_dbg("dev->backend_dev.channel=3D%p\n", backend_dev->channel); =20 - ret =3D ibv_query_port(backend_dev->context, backend_dev->port_num, - &port_attr); - if (ret) { - error_setg(errp, "Error %d from ibv_query_port", ret); - ret =3D -EIO; - goto out_destroy_comm_channel; - } - - if (backend_dev->backend_gid_idx >=3D port_attr.gid_tbl_len) { - error_setg(errp, "Invalid backend_gid_idx, should be less than %d", - port_attr.gid_tbl_len); - goto out_destroy_comm_channel; - } - ret =3D init_device_caps(backend_dev, dev_attr); if (ret) { error_setg(errp, "Failed to initialize device capabilities"); @@ -1081,18 +1168,6 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, P= CIDevice *pdev, goto out_destroy_comm_channel; } =20 - ret =3D ibv_query_gid(backend_dev->context, backend_dev->port_num, - backend_dev->backend_gid_idx, &backend_dev->gid); - if (ret) { - error_setg(errp, "Failed to query gid %d", - backend_dev->backend_gid_idx); - ret =3D -EIO; - goto out_destroy_comm_channel; - } - pr_dbg("subnet_prefix=3D0x%" PRIx64 "\n", - be64_to_cpu(backend_dev->gid.global.subnet_prefix)); - pr_dbg("interface_id=3D0x%" PRIx64 "\n", - be64_to_cpu(backend_dev->gid.global.interface_id)); =20 ret =3D mad_init(backend_dev); if (ret) { diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index fc83330251..59ad2b874b 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -28,11 +28,6 @@ enum ibv_special_qp_type { IBV_QPT_GSI =3D 1, }; =20 -static inline union ibv_gid *rdma_backend_gid(RdmaBackendDev *dev) -{ - return &dev->gid; -} - static inline uint32_t rdma_backend_qpn(const RdmaBackendQP *qp) { return qp->ibqp ? qp->ibqp->qp_num : 1; @@ -51,9 +46,15 @@ static inline uint32_t rdma_backend_mr_rkey(const RdmaBa= ckendMR *mr) int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, - uint8_t backend_gid_idx, struct ibv_device_attr *dev= _attr, - CharBackend *mad_chr_be, Error **errp); + struct ibv_device_attr *dev_attr, CharBackend *mad_c= hr_be, + Error **errp); void rdma_backend_fini(RdmaBackendDev *backend_dev); +int rdma_backend_add_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid); +int rdma_backend_del_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid); +int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, + union ibv_gid *gid); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); void rdma_backend_register_comp_handler(void (*handler)(int status, @@ -82,9 +83,9 @@ int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_= type, int rdma_backend_qp_state_init(RdmaBackendDev *backend_dev, RdmaBackendQP = *qp, uint8_t qp_type, uint32_t qkey); int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *= qp, - uint8_t qp_type, union ibv_gid *dgid, - uint32_t dqpn, uint32_t rq_psn, uint32_t qke= y, - bool use_qkey); + uint8_t qp_type, uint8_t sgid_idx, + union ibv_gid *dgid, uint32_t dqpn, + uint32_t rq_psn, uint32_t qkey, bool use_qke= y); int rdma_backend_qp_state_rts(RdmaBackendQP *qp, uint8_t qp_type, uint32_t sq_psn, uint32_t qkey, bool use_qke= y); int rdma_backend_query_qp(RdmaBackendQP *qp, struct ibv_qp_attr *attr, @@ -94,6 +95,7 @@ void rdma_backend_destroy_qp(RdmaBackendQP *qp); void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, + uint8_t sgid_idx, union ibv_gid *sgid, union ibv_gid *dgid, uint32_t dqpn, uint32_t d= qkey, void *ctx); void rdma_backend_post_recv(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 2a7e667075..ff8b2426a0 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -37,14 +37,12 @@ typedef struct RecvMadList { typedef struct RdmaBackendDev { struct ibv_device_attr dev_attr; RdmaBackendThread comp_thread; - union ibv_gid gid; PCIDevice *dev; RdmaDeviceResources *rdma_dev_res; struct ibv_device *ib_dev; struct ibv_context *context; struct ibv_comp_channel *channel; uint8_t port_num; - uint8_t backend_gid_idx; RecvMadList recv_mads_list; CharBackend *mad_chr_be; } RdmaBackendDev; @@ -66,6 +64,7 @@ typedef struct RdmaBackendCQ { typedef struct RdmaBackendQP { struct ibv_pd *ibpd; struct ibv_qp *ibqp; + uint8_t sgid_idx; } RdmaBackendQP; =20 #endif diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 4f10fcabcc..fe0979415d 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -391,7 +391,7 @@ out_dealloc_qp: } =20 int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backen= d_dev, - uint32_t qp_handle, uint32_t attr_mask, + uint32_t qp_handle, uint32_t attr_mask, uint8_t sgid= _idx, union ibv_gid *dgid, uint32_t dqpn, enum ibv_qp_state qp_state, uint32_t qkey, uint32_t rq_psn, uint32_t sq_psn) @@ -400,6 +400,7 @@ int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, Rdm= aBackendDev *backend_dev, int ret; =20 pr_dbg("qpn=3D0x%x\n", qp_handle); + pr_dbg("qkey=3D0x%x\n", qkey); =20 qp =3D rdma_rm_get_qp(dev_res, qp_handle); if (!qp) { @@ -430,9 +431,19 @@ int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, Rd= maBackendDev *backend_dev, } =20 if (qp->qp_state =3D=3D IBV_QPS_RTR) { + /* Get backend gid index */ + pr_dbg("Guest sgid_idx=3D%d\n", sgid_idx); + sgid_idx =3D rdma_rm_get_backend_gid_index(dev_res, backend_de= v, + sgid_idx); + if (sgid_idx <=3D 0) { /* TODO check also less than bk.max_sgi= d */ + pr_dbg("Fail to get bk sgid_idx for sgid_idx %d\n", sgid_i= dx); + return -EIO; + } + ret =3D rdma_backend_qp_state_rtr(backend_dev, &qp->backend_qp, - qp->qp_type, dgid, dqpn, rq_ps= n, - qkey, attr_mask & IBV_QP_QKEY); + qp->qp_type, sgid_idx, dgid, d= qpn, + rq_psn, qkey, + attr_mask & IBV_QP_QKEY); if (ret) { return -EIO; } @@ -523,11 +534,91 @@ void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev= _res, uint32_t cqe_ctx_id) res_tbl_dealloc(&dev_res->cqe_ctx_tbl, cqe_ctx_id); } =20 +int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, union ibv_gid *gid, int gid_idx) +{ + int rc; + + rc =3D rdma_backend_add_gid(backend_dev, ifname, gid); + if (rc <=3D 0) { + pr_dbg("Fail to add gid\n"); + return -EINVAL; + } + + memcpy(&dev_res->ports[0].gid_tbl[gid_idx].gid, gid, sizeof(*gid)); + + return 0; +} + +int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, int gid_idx) +{ + int rc; + + rc =3D rdma_backend_del_gid(backend_dev, ifname, + &dev_res->ports[0].gid_tbl[gid_idx].gid); + if (rc < 0) { + pr_dbg("Fail to delete gid\n"); + return -EINVAL; + } + + memset(dev_res->ports[0].gid_tbl[gid_idx].gid.raw, 0, + sizeof(dev_res->ports[0].gid_tbl[gid_idx].gid)); + dev_res->ports[0].gid_tbl[gid_idx].backend_gid_index =3D -1; + + return 0; +} + +int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, int sgid_id= x) +{ + if (unlikely(sgid_idx < 0 || sgid_idx > MAX_PORT_GIDS)) { + pr_dbg("Got invalid sgid_idx %d\n", sgid_idx); + return -EINVAL; + } + + if (unlikely(dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D= =3D -1)) { + dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D + rdma_backend_get_gid_index(backend_dev, + &dev_res->ports[0].gid_tbl[sgid_idx= ].gid); + } + + pr_dbg("backend_gid_index=3D%d\n", + dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index); + + return dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index; +} + static void destroy_qp_hash_key(gpointer data) { g_bytes_unref(data); } =20 +static void init_ports(RdmaDeviceResources *dev_res) +{ + int i, j; + + memset(dev_res->ports, 0, sizeof(dev_res->ports)); + + for (i =3D 0; i < MAX_PORTS; i++) { + dev_res->ports[i].state =3D IBV_PORT_DOWN; + for (j =3D 0; j < MAX_PORT_GIDS; j++) { + dev_res->ports[i].gid_tbl[j].backend_gid_index =3D -1; + } + } +} + +static void fini_ports(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, const char *ifname) +{ + int i; + + dev_res->ports[0].state =3D IBV_PORT_DOWN; + for (i =3D 0; i < MAX_PORT_GIDS; i++) { + rdma_rm_del_gid(dev_res, backend_dev, ifname, i); + } +} + int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev= _attr, Error **errp) { @@ -545,11 +636,16 @@ int rdma_rm_init(RdmaDeviceResources *dev_res, struct= ibv_device_attr *dev_attr, dev_attr->max_qp_wr, sizeof(void *)); res_tbl_init("UC", &dev_res->uc_tbl, MAX_UCS, sizeof(RdmaRmUC)); =20 + init_ports(dev_res); + return 0; } =20 -void rdma_rm_fini(RdmaDeviceResources *dev_res) +void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_de= v, + const char *ifname) { + fini_ports(dev_res, backend_dev, ifname); + res_tbl_free(&dev_res->uc_tbl); res_tbl_free(&dev_res->cqe_ctx_tbl); res_tbl_free(&dev_res->qp_tbl); diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index b4e04cc7b4..a7169b4e89 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -22,7 +22,8 @@ =20 int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev= _attr, Error **errp); -void rdma_rm_fini(RdmaDeviceResources *dev_res); +void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_de= v, + const char *ifname); =20 int rdma_rm_alloc_pd(RdmaDeviceResources *dev_res, RdmaBackendDev *backend= _dev, uint32_t *pd_handle, uint32_t ctx_handle); @@ -55,7 +56,7 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uint32= _t pd_handle, uint32_t recv_cq_handle, void *opaque, uint32_t *qpn); RdmaRmQP *rdma_rm_get_qp(RdmaDeviceResources *dev_res, uint32_t qpn); int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backen= d_dev, - uint32_t qp_handle, uint32_t attr_mask, + uint32_t qp_handle, uint32_t attr_mask, uint8_t sgid= _idx, union ibv_gid *dgid, uint32_t dqpn, enum ibv_qp_state qp_state, uint32_t qkey, uint32_t rq_psn, uint32_t sq_psn); @@ -69,4 +70,16 @@ int rdma_rm_alloc_cqe_ctx(RdmaDeviceResources *dev_res, = uint32_t *cqe_ctx_id, void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_i= d); void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ct= x_id); =20 +int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, union ibv_gid *gid, int gid_idx); +int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_= dev, + const char *ifname, int gid_idx); +int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, int sgid_id= x); +static inline union ibv_gid *rdma_rm_get_gid(RdmaDeviceResources *dev_res, + int sgid_idx) +{ + return &dev_res->ports[0].gid_tbl[sgid_idx].gid; +} + #endif diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 9b399063d3..7b3435f991 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -19,7 +19,7 @@ #include "rdma_backend_defs.h" =20 #define MAX_PORTS 1 -#define MAX_PORT_GIDS 1 +#define MAX_PORT_GIDS 255 #define MAX_GIDS MAX_PORT_GIDS #define MAX_PORT_PKEYS 1 #define MAX_PKEYS MAX_PORT_PKEYS @@ -86,8 +86,13 @@ typedef struct RdmaRmQP { enum ibv_qp_state qp_state; } RdmaRmQP; =20 +typedef struct RdmaRmGid { + union ibv_gid gid; + int backend_gid_index; +} RdmaRmGid; + typedef struct RdmaRmPort { - union ibv_gid gid_tbl[MAX_PORT_GIDS]; + RdmaRmGid gid_tbl[MAX_PORT_GIDS]; enum ibv_port_state state; } RdmaRmPort; =20 diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index 04c7c2ef5b..989db249ef 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -20,6 +20,7 @@ #include "qemu/osdep.h" #include "hw/pci/pci.h" #include "sysemu/dma.h" +#include "stdio.h" =20 #define pr_info(fmt, ...) \ fprintf(stdout, "%s: %-20s (%3d): " fmt, "rdma", __func__, __LINE__,\ @@ -40,9 +41,23 @@ extern unsigned long pr_dbg_cnt; #define pr_dbg(fmt, ...) \ fprintf(stdout, "%lx %ld: %-20s (%3d): " fmt, pthread_self(), pr_dbg_c= nt++, \ __func__, __LINE__, ## __VA_ARGS__) + +#define pr_dbg_buf(title, buf, len) \ +{ \ + char *b =3D g_malloc0(len * 3 + 1); \ + char b1[4]; \ + for (int i =3D 0; i < len; i++) { \ + sprintf(b1, "%.2X ", buf[i] & 0x000000FF); \ + strcat(b, b1); \ + } \ + pr_dbg("%s (%d): %s\n", title, len, b); \ + g_free(b); \ +} + #else #define init_pr_dbg(void) #define pr_dbg(fmt, ...) +#define pr_dbg_buf(title, buf, len) #endif =20 void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index 15c3f28b86..b019cb843a 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -79,8 +79,8 @@ typedef struct PVRDMADev { int interrupt_mask; struct ibv_device_attr dev_attr; uint64_t node_guid; + char *backend_eth_device_name; char *backend_device_name; - uint8_t backend_gid_idx; uint8_t backend_port_num; RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 57d6f41ae6..a334f6205e 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -504,13 +504,16 @@ static int modify_qp(PVRDMADev *dev, union pvrdma_cmd= _req *req, rsp->hdr.response =3D cmd->hdr.response; rsp->hdr.ack =3D PVRDMA_CMD_MODIFY_QP_RESP; =20 - rsp->hdr.err =3D rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_d= ev, - cmd->qp_handle, cmd->attr_mask, - (union ibv_gid *)&cmd->attrs.ah_attr.grh.= dgid, - cmd->attrs.dest_qp_num, - (enum ibv_qp_state)cmd->attrs.qp_state, - cmd->attrs.qkey, cmd->attrs.rq_psn, - cmd->attrs.sq_psn); + /* No need to verify sgid_index since it is u8 */ + + rsp->hdr.err =3D + rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_dev, cmd->qp_h= andle, + cmd->attr_mask, cmd->attrs.ah_attr.grh.sgid_inde= x, + (union ibv_gid *)&cmd->attrs.ah_attr.grh.dgid, + cmd->attrs.dest_qp_num, + (enum ibv_qp_state)cmd->attrs.qp_state, + cmd->attrs.qkey, cmd->attrs.rq_psn, + cmd->attrs.sq_psn); =20 pr_dbg("ret=3D%d\n", rsp->hdr.err); return rsp->hdr.err; @@ -570,10 +573,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cm= d_req *req, union pvrdma_cmd_resp *rsp) { struct pvrdma_cmd_create_bind *cmd =3D &req->create_bind; -#ifdef PVRDMA_DEBUG - __be64 *subnet =3D (__be64 *)&cmd->new_gid[0]; - __be64 *if_id =3D (__be64 *)&cmd->new_gid[8]; -#endif + int rc; + union ibv_gid *gid =3D (union ibv_gid *)&cmd->new_gid; =20 pr_dbg("index=3D%d\n", cmd->index); =20 @@ -582,19 +583,24 @@ static int create_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, } =20 pr_dbg("gid[%d]=3D0x%llx,0x%llx\n", cmd->index, - (long long unsigned int)be64_to_cpu(*subnet), - (long long unsigned int)be64_to_cpu(*if_id)); + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); =20 - /* Driver forces to one port only */ - memcpy(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, &cmd->new_g= id, - sizeof(cmd->new_gid)); + rc =3D rdma_rm_add_gid(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name, gid, cmd->index); + if (rc < 0) { + return -EINVAL; + } =20 /* TODO: Since drivers stores node_guid at load_dsr phase then this * assignment is not relevant, i need to figure out a way how to * retrieve MAC of our netdev */ - dev->node_guid =3D dev->rdma_dev_res.ports[0].gid_tbl[0].global.interf= ace_id; - pr_dbg("dev->node_guid=3D0x%llx\n", - (long long unsigned int)be64_to_cpu(dev->node_guid)); + if (!cmd->index) { + dev->node_guid =3D + dev->rdma_dev_res.ports[0].gid_tbl[0].gid.global.interface_id; + pr_dbg("dev->node_guid=3D0x%llx\n", + (long long unsigned int)be64_to_cpu(dev->node_guid)); + } =20 return 0; } @@ -602,6 +608,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd= _req *req, static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp) { + int rc; + struct pvrdma_cmd_destroy_bind *cmd =3D &req->destroy_bind; =20 pr_dbg("index=3D%d\n", cmd->index); @@ -610,8 +618,13 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, return -EINVAL; } =20 - memset(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, 0, - sizeof(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw)); + rc =3D rdma_rm_del_gid(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name, cmd->index); + + if (rc < 0) { + rsp->hdr.err =3D rc; + goto out; + } =20 return 0; } diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index fc2abd34af..ac8c092db0 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -36,9 +36,9 @@ #include "pvrdma_qp_ops.h" =20 static Property pvrdma_dev_properties[] =3D { - DEFINE_PROP_STRING("backend-dev", PVRDMADev, backend_device_name), - DEFINE_PROP_UINT8("backend-port", PVRDMADev, backend_port_num, 1), - DEFINE_PROP_UINT8("backend-gid-idx", PVRDMADev, backend_gid_idx, 0), + DEFINE_PROP_STRING("netdev", PVRDMADev, backend_eth_device_name), + DEFINE_PROP_STRING("ibdev", PVRDMADev, backend_device_name), + DEFINE_PROP_UINT8("ibport", PVRDMADev, backend_port_num, 1), DEFINE_PROP_UINT64("dev-caps-max-mr-size", PVRDMADev, dev_attr.max_mr_= size, MAX_MR_SIZE), DEFINE_PROP_INT32("dev-caps-max-qp", PVRDMADev, dev_attr.max_qp, MAX_Q= P), @@ -276,17 +276,6 @@ static void init_dsr_dev_caps(PVRDMADev *dev) pr_dbg("Initialized\n"); } =20 -static void init_ports(PVRDMADev *dev, Error **errp) -{ - int i; - - memset(dev->rdma_dev_res.ports, 0, sizeof(dev->rdma_dev_res.ports)); - - for (i =3D 0; i < MAX_PORTS; i++) { - dev->rdma_dev_res.ports[i].state =3D IBV_PORT_DOWN; - } -} - static void uninit_msix(PCIDevice *pdev, int used_vectors) { PVRDMADev *dev =3D PVRDMA_DEV(pdev); @@ -335,7 +324,8 @@ static void pvrdma_fini(PCIDevice *pdev) =20 pvrdma_qp_ops_fini(); =20 - rdma_rm_fini(&dev->rdma_dev_res); + rdma_rm_fini(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name); =20 rdma_backend_fini(&dev->backend_dev); =20 @@ -612,8 +602,7 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) =20 rc =3D rdma_backend_init(&dev->backend_dev, pdev, &dev->rdma_dev_res, dev->backend_device_name, dev->backend_port_num, - dev->backend_gid_idx, &dev->dev_attr, &dev->mad= _chr, - errp); + &dev->dev_attr, &dev->mad_chr, errp); if (rc) { goto out; } @@ -623,8 +612,6 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) goto out; } =20 - init_ports(dev, errp); - rc =3D pvrdma_qp_ops_init(); if (rc) { goto out; diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 3388be1926..2130824098 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -131,6 +131,8 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) RdmaRmQP *qp; PvrdmaSqWqe *wqe; PvrdmaRing *ring; + int sgid_idx; + union ibv_gid *sgid; =20 pr_dbg("qp_handle=3D0x%x\n", qp_handle); =20 @@ -156,8 +158,26 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) comp_ctx->cqe.qp =3D qp_handle; comp_ctx->cqe.opcode =3D IBV_WC_SEND; =20 + sgid =3D rdma_rm_get_gid(&dev->rdma_dev_res, wqe->hdr.wr.ud.av.gid= _index); + if (!sgid) { + pr_dbg("Fail to get gid for idx %d\n", wqe->hdr.wr.ud.av.gid_i= ndex); + return -EIO; + } + pr_dbg("sgid_id=3D%d, sgid=3D0x%llx\n", wqe->hdr.wr.ud.av.gid_inde= x, + sgid->global.interface_id); + + sgid_idx =3D rdma_rm_get_backend_gid_index(&dev->rdma_dev_res, + &dev->backend_dev, + wqe->hdr.wr.ud.av.gid_ind= ex); + if (sgid_idx <=3D 0) { + pr_dbg("Fail to get bk sgid_idx for sgid_idx %d\n", + wqe->hdr.wr.ud.av.gid_index); + return -EIO; + } + rdma_backend_post_send(&dev->backend_dev, &qp->backend_qp, qp->qp_= type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.nu= m_sge, + sgid_idx, sgid, (union ibv_gid *)wqe->hdr.wr.ud.av.dgid, wqe->hdr.wr.ud.remote_qpn, wqe->hdr.wr.ud.remote_qkey, comp_ctx); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095198547885.8142787651396; Mon, 12 Nov 2018 23:46:38 -0800 (PST) Received: from localhost ([::1]:52475 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTPB-0004DF-2v for importer@patchew.org; Tue, 13 Nov 2018 02:46:37 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52317) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvS-000556-C6 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvQ-0004dC-Dc for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:54 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44644) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvP-0004bj-QG for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:52 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7E2ER078952; Tue, 13 Nov 2018 07:15:49 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nnwg18mcx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:49 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Fm6L018753 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:49 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FmRk028803; Tue, 13 Nov 2018 07:15:48 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:48 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=yQAKxSDm97kiD85kKOr1Hq5ffDMN3yaQPEwJjS+MWkg=; b=P3g3bY9Tx3ohY0llY9rgRiRzLFrrDKeeRuJmDS3+00jo0RQ08UsT7fcf8y5uxeNQuERI HJOB9CT8dqDy80prcbanNtQthw6k48pW4EFv7a3D04LPNZjhM4bQZq7SpBbbwvmk+kLr uyuRX9Za/m/T1FPVITorIpl5IjlFQXffSqrgV71gdMHIpMiIEgm3LJ+B5q2s9q/T+GP8 EwwlaGAnbdb5qFuH117A+rNWQZwiRq3emFEsNNez1MikJE+8Z2z/iafl/bjwA2RDGIvZ QFepjTNpz1FAL8phFO+BuL+9eHS8ihW01p16CXsmKan9OUaHJE3dmNj+CVcRlEGdpPPe YA== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:25 +0200 Message-Id: <20181113071336.6242-37-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 12/23] vmxnet3: Move some definitions to header file X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" pvrdma setup requires vmxnet3 device on PCI function 0 and PVRDMA device on PCI function 1. pvrdma device needs to access vmxnet3 device object for several reasons: 1. Make sure PCI function 0 is vmxnet3. 2. To monitor vmxnet3 device state. 3. To configure node_guid accoring to vmxnet3 device's MAC address. To be able to access vmxnet3 device the definition of VMXNET3State is moved to a new header file. Signed-off-by: Yuval Shaia Reviewed-by: Dmitry Fleytman --- hw/net/vmxnet3.c | 116 +----------------------------------- hw/net/vmxnet3_defs.h | 133 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 115 deletions(-) create mode 100644 hw/net/vmxnet3_defs.h diff --git a/hw/net/vmxnet3.c b/hw/net/vmxnet3.c index 3648630386..54746a4030 100644 --- a/hw/net/vmxnet3.c +++ b/hw/net/vmxnet3.c @@ -18,7 +18,6 @@ #include "qemu/osdep.h" #include "hw/hw.h" #include "hw/pci/pci.h" -#include "net/net.h" #include "net/tap.h" #include "net/checksum.h" #include "sysemu/sysemu.h" @@ -29,6 +28,7 @@ #include "migration/register.h" =20 #include "vmxnet3.h" +#include "vmxnet3_defs.h" #include "vmxnet_debug.h" #include "vmware_utils.h" #include "net_tx_pkt.h" @@ -131,23 +131,11 @@ typedef struct VMXNET3Class { DeviceRealize parent_dc_realize; } VMXNET3Class; =20 -#define TYPE_VMXNET3 "vmxnet3" -#define VMXNET3(obj) OBJECT_CHECK(VMXNET3State, (obj), TYPE_VMXNET3) - #define VMXNET3_DEVICE_CLASS(klass) \ OBJECT_CLASS_CHECK(VMXNET3Class, (klass), TYPE_VMXNET3) #define VMXNET3_DEVICE_GET_CLASS(obj) \ OBJECT_GET_CLASS(VMXNET3Class, (obj), TYPE_VMXNET3) =20 -/* Cyclic ring abstraction */ -typedef struct { - hwaddr pa; - uint32_t size; - uint32_t cell_size; - uint32_t next; - uint8_t gen; -} Vmxnet3Ring; - static inline void vmxnet3_ring_init(PCIDevice *d, Vmxnet3Ring *ring, hwaddr pa, @@ -245,108 +233,6 @@ vmxnet3_dump_rx_descr(struct Vmxnet3_RxDesc *descr) descr->rsvd, descr->dtype, descr->ext1, descr->btype); } =20 -/* Device state and helper functions */ -#define VMXNET3_RX_RINGS_PER_QUEUE (2) - -typedef struct { - Vmxnet3Ring tx_ring; - Vmxnet3Ring comp_ring; - - uint8_t intr_idx; - hwaddr tx_stats_pa; - struct UPT1_TxStats txq_stats; -} Vmxnet3TxqDescr; - -typedef struct { - Vmxnet3Ring rx_ring[VMXNET3_RX_RINGS_PER_QUEUE]; - Vmxnet3Ring comp_ring; - uint8_t intr_idx; - hwaddr rx_stats_pa; - struct UPT1_RxStats rxq_stats; -} Vmxnet3RxqDescr; - -typedef struct { - bool is_masked; - bool is_pending; - bool is_asserted; -} Vmxnet3IntState; - -typedef struct { - PCIDevice parent_obj; - NICState *nic; - NICConf conf; - MemoryRegion bar0; - MemoryRegion bar1; - MemoryRegion msix_bar; - - Vmxnet3RxqDescr rxq_descr[VMXNET3_DEVICE_MAX_RX_QUEUES]; - Vmxnet3TxqDescr txq_descr[VMXNET3_DEVICE_MAX_TX_QUEUES]; - - /* Whether MSI-X support was installed successfully */ - bool msix_used; - hwaddr drv_shmem; - hwaddr temp_shared_guest_driver_memory; - - uint8_t txq_num; - - /* This boolean tells whether RX packet being indicated has to */ - /* be split into head and body chunks from different RX rings */ - bool rx_packets_compound; - - bool rx_vlan_stripping; - bool lro_supported; - - uint8_t rxq_num; - - /* Network MTU */ - uint32_t mtu; - - /* Maximum number of fragments for indicated TX packets */ - uint32_t max_tx_frags; - - /* Maximum number of fragments for indicated RX packets */ - uint16_t max_rx_frags; - - /* Index for events interrupt */ - uint8_t event_int_idx; - - /* Whether automatic interrupts masking enabled */ - bool auto_int_masking; - - bool peer_has_vhdr; - - /* TX packets to QEMU interface */ - struct NetTxPkt *tx_pkt; - uint32_t offload_mode; - uint32_t cso_or_gso_size; - uint16_t tci; - bool needs_vlan; - - struct NetRxPkt *rx_pkt; - - bool tx_sop; - bool skip_current_tx_pkt; - - uint32_t device_active; - uint32_t last_command; - - uint32_t link_status_and_speed; - - Vmxnet3IntState interrupt_states[VMXNET3_MAX_INTRS]; - - uint32_t temp_mac; /* To store the low part first */ - - MACAddr perm_mac; - uint32_t vlan_table[VMXNET3_VFT_SIZE]; - uint32_t rx_mode; - MACAddr *mcast_list; - uint32_t mcast_list_len; - uint32_t mcast_list_buff_size; /* needed for live migration. */ - - /* Compatibility flags for migration */ - uint32_t compat_flags; -} VMXNET3State; - /* Interrupt management */ =20 /* diff --git a/hw/net/vmxnet3_defs.h b/hw/net/vmxnet3_defs.h new file mode 100644 index 0000000000..6c19d29b12 --- /dev/null +++ b/hw/net/vmxnet3_defs.h @@ -0,0 +1,133 @@ +/* + * QEMU VMWARE VMXNET3 paravirtual NIC + * + * Copyright (c) 2012 Ravello Systems LTD (http://ravellosystems.com) + * + * Developed by Daynix Computing LTD (http://www.daynix.com) + * + * Authors: + * Dmitry Fleytman + * Tamir Shomer + * Yan Vugenfirer + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + * + */ + +#include "net/net.h" +#include "hw/net/vmxnet3.h" + +#define TYPE_VMXNET3 "vmxnet3" +#define VMXNET3(obj) OBJECT_CHECK(VMXNET3State, (obj), TYPE_VMXNET3) + +/* Device state and helper functions */ +#define VMXNET3_RX_RINGS_PER_QUEUE (2) + +/* Cyclic ring abstraction */ +typedef struct { + hwaddr pa; + uint32_t size; + uint32_t cell_size; + uint32_t next; + uint8_t gen; +} Vmxnet3Ring; + +typedef struct { + Vmxnet3Ring tx_ring; + Vmxnet3Ring comp_ring; + + uint8_t intr_idx; + hwaddr tx_stats_pa; + struct UPT1_TxStats txq_stats; +} Vmxnet3TxqDescr; + +typedef struct { + Vmxnet3Ring rx_ring[VMXNET3_RX_RINGS_PER_QUEUE]; + Vmxnet3Ring comp_ring; + uint8_t intr_idx; + hwaddr rx_stats_pa; + struct UPT1_RxStats rxq_stats; +} Vmxnet3RxqDescr; + +typedef struct { + bool is_masked; + bool is_pending; + bool is_asserted; +} Vmxnet3IntState; + +typedef struct { + PCIDevice parent_obj; + NICState *nic; + NICConf conf; + MemoryRegion bar0; + MemoryRegion bar1; + MemoryRegion msix_bar; + + Vmxnet3RxqDescr rxq_descr[VMXNET3_DEVICE_MAX_RX_QUEUES]; + Vmxnet3TxqDescr txq_descr[VMXNET3_DEVICE_MAX_TX_QUEUES]; + + /* Whether MSI-X support was installed successfully */ + bool msix_used; + hwaddr drv_shmem; + hwaddr temp_shared_guest_driver_memory; + + uint8_t txq_num; + + /* This boolean tells whether RX packet being indicated has to */ + /* be split into head and body chunks from different RX rings */ + bool rx_packets_compound; + + bool rx_vlan_stripping; + bool lro_supported; + + uint8_t rxq_num; + + /* Network MTU */ + uint32_t mtu; + + /* Maximum number of fragments for indicated TX packets */ + uint32_t max_tx_frags; + + /* Maximum number of fragments for indicated RX packets */ + uint16_t max_rx_frags; + + /* Index for events interrupt */ + uint8_t event_int_idx; + + /* Whether automatic interrupts masking enabled */ + bool auto_int_masking; + + bool peer_has_vhdr; + + /* TX packets to QEMU interface */ + struct NetTxPkt *tx_pkt; + uint32_t offload_mode; + uint32_t cso_or_gso_size; + uint16_t tci; + bool needs_vlan; + + struct NetRxPkt *rx_pkt; + + bool tx_sop; + bool skip_current_tx_pkt; + + uint32_t device_active; + uint32_t last_command; + + uint32_t link_status_and_speed; + + Vmxnet3IntState interrupt_states[VMXNET3_MAX_INTRS]; + + uint32_t temp_mac; /* To store the low part first */ + + MACAddr perm_mac; + uint32_t vlan_table[VMXNET3_VFT_SIZE]; + uint32_t rx_mode; + MACAddr *mcast_list; + uint32_t mcast_list_len; + uint32_t mcast_list_buff_size; /* needed for live migration. */ + + /* Compatibility flags for migration */ + uint32_t compat_flags; +} VMXNET3State; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093575288175.2144159929128; Mon, 12 Nov 2018 23:19:35 -0800 (PST) Received: from localhost ([::1]:52330 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSyz-0007KY-Tb for importer@patchew.org; Tue, 13 Nov 2018 02:19:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51805) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuS-0004B3-8s for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuP-0003Ej-H1 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:52 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:59580) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuP-00031I-4h for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:49 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Dx1a046013; Tue, 13 Nov 2018 07:14:37 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp2120.oracle.com with ESMTP id 2nnw6egpp5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:37 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EakR010112 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:36 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7EZpb017630; Tue, 13 Nov 2018 07:14:35 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:35 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=yQAKxSDm97kiD85kKOr1Hq5ffDMN3yaQPEwJjS+MWkg=; b=V7WI/T2I1bOLly43TP4G7gpJwb7QR+OEkL0XCC/lJxm5QH35ffQe6gmoJCGTRDVxkKei ahkgw/dOUbpwp1roTSeGqDjc89wAX1l51NbeU4CFMLjAf0aQZUYpklxsDeKEN6h0nm68 U7QSLpP22tAeWBYd2cetL9/FUpuijqwatK1DHOhvuKZuJpgO6f9UaX+X/kq7hqbFLFhT VHdCzjgGCefAtkBa9vNPguQtGAPHMem4h3CvL8nbymbp9oYI5CP5AoH2y1rggj9acHDm Ciqrt9MSxB0pyMMYPabDiZyP18r8Z7XZYxA/DsfZ2BZPTPgy/kG208ooSsNUpTSb7Hox ng== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:01 +0200 Message-Id: <20181113071336.6242-13-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 12/23] vmxnet3: Move some definitions to header file X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" pvrdma setup requires vmxnet3 device on PCI function 0 and PVRDMA device on PCI function 1. pvrdma device needs to access vmxnet3 device object for several reasons: 1. Make sure PCI function 0 is vmxnet3. 2. To monitor vmxnet3 device state. 3. To configure node_guid accoring to vmxnet3 device's MAC address. To be able to access vmxnet3 device the definition of VMXNET3State is moved to a new header file. Signed-off-by: Yuval Shaia Reviewed-by: Dmitry Fleytman --- hw/net/vmxnet3.c | 116 +----------------------------------- hw/net/vmxnet3_defs.h | 133 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 115 deletions(-) create mode 100644 hw/net/vmxnet3_defs.h diff --git a/hw/net/vmxnet3.c b/hw/net/vmxnet3.c index 3648630386..54746a4030 100644 --- a/hw/net/vmxnet3.c +++ b/hw/net/vmxnet3.c @@ -18,7 +18,6 @@ #include "qemu/osdep.h" #include "hw/hw.h" #include "hw/pci/pci.h" -#include "net/net.h" #include "net/tap.h" #include "net/checksum.h" #include "sysemu/sysemu.h" @@ -29,6 +28,7 @@ #include "migration/register.h" =20 #include "vmxnet3.h" +#include "vmxnet3_defs.h" #include "vmxnet_debug.h" #include "vmware_utils.h" #include "net_tx_pkt.h" @@ -131,23 +131,11 @@ typedef struct VMXNET3Class { DeviceRealize parent_dc_realize; } VMXNET3Class; =20 -#define TYPE_VMXNET3 "vmxnet3" -#define VMXNET3(obj) OBJECT_CHECK(VMXNET3State, (obj), TYPE_VMXNET3) - #define VMXNET3_DEVICE_CLASS(klass) \ OBJECT_CLASS_CHECK(VMXNET3Class, (klass), TYPE_VMXNET3) #define VMXNET3_DEVICE_GET_CLASS(obj) \ OBJECT_GET_CLASS(VMXNET3Class, (obj), TYPE_VMXNET3) =20 -/* Cyclic ring abstraction */ -typedef struct { - hwaddr pa; - uint32_t size; - uint32_t cell_size; - uint32_t next; - uint8_t gen; -} Vmxnet3Ring; - static inline void vmxnet3_ring_init(PCIDevice *d, Vmxnet3Ring *ring, hwaddr pa, @@ -245,108 +233,6 @@ vmxnet3_dump_rx_descr(struct Vmxnet3_RxDesc *descr) descr->rsvd, descr->dtype, descr->ext1, descr->btype); } =20 -/* Device state and helper functions */ -#define VMXNET3_RX_RINGS_PER_QUEUE (2) - -typedef struct { - Vmxnet3Ring tx_ring; - Vmxnet3Ring comp_ring; - - uint8_t intr_idx; - hwaddr tx_stats_pa; - struct UPT1_TxStats txq_stats; -} Vmxnet3TxqDescr; - -typedef struct { - Vmxnet3Ring rx_ring[VMXNET3_RX_RINGS_PER_QUEUE]; - Vmxnet3Ring comp_ring; - uint8_t intr_idx; - hwaddr rx_stats_pa; - struct UPT1_RxStats rxq_stats; -} Vmxnet3RxqDescr; - -typedef struct { - bool is_masked; - bool is_pending; - bool is_asserted; -} Vmxnet3IntState; - -typedef struct { - PCIDevice parent_obj; - NICState *nic; - NICConf conf; - MemoryRegion bar0; - MemoryRegion bar1; - MemoryRegion msix_bar; - - Vmxnet3RxqDescr rxq_descr[VMXNET3_DEVICE_MAX_RX_QUEUES]; - Vmxnet3TxqDescr txq_descr[VMXNET3_DEVICE_MAX_TX_QUEUES]; - - /* Whether MSI-X support was installed successfully */ - bool msix_used; - hwaddr drv_shmem; - hwaddr temp_shared_guest_driver_memory; - - uint8_t txq_num; - - /* This boolean tells whether RX packet being indicated has to */ - /* be split into head and body chunks from different RX rings */ - bool rx_packets_compound; - - bool rx_vlan_stripping; - bool lro_supported; - - uint8_t rxq_num; - - /* Network MTU */ - uint32_t mtu; - - /* Maximum number of fragments for indicated TX packets */ - uint32_t max_tx_frags; - - /* Maximum number of fragments for indicated RX packets */ - uint16_t max_rx_frags; - - /* Index for events interrupt */ - uint8_t event_int_idx; - - /* Whether automatic interrupts masking enabled */ - bool auto_int_masking; - - bool peer_has_vhdr; - - /* TX packets to QEMU interface */ - struct NetTxPkt *tx_pkt; - uint32_t offload_mode; - uint32_t cso_or_gso_size; - uint16_t tci; - bool needs_vlan; - - struct NetRxPkt *rx_pkt; - - bool tx_sop; - bool skip_current_tx_pkt; - - uint32_t device_active; - uint32_t last_command; - - uint32_t link_status_and_speed; - - Vmxnet3IntState interrupt_states[VMXNET3_MAX_INTRS]; - - uint32_t temp_mac; /* To store the low part first */ - - MACAddr perm_mac; - uint32_t vlan_table[VMXNET3_VFT_SIZE]; - uint32_t rx_mode; - MACAddr *mcast_list; - uint32_t mcast_list_len; - uint32_t mcast_list_buff_size; /* needed for live migration. */ - - /* Compatibility flags for migration */ - uint32_t compat_flags; -} VMXNET3State; - /* Interrupt management */ =20 /* diff --git a/hw/net/vmxnet3_defs.h b/hw/net/vmxnet3_defs.h new file mode 100644 index 0000000000..6c19d29b12 --- /dev/null +++ b/hw/net/vmxnet3_defs.h @@ -0,0 +1,133 @@ +/* + * QEMU VMWARE VMXNET3 paravirtual NIC + * + * Copyright (c) 2012 Ravello Systems LTD (http://ravellosystems.com) + * + * Developed by Daynix Computing LTD (http://www.daynix.com) + * + * Authors: + * Dmitry Fleytman + * Tamir Shomer + * Yan Vugenfirer + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + * + */ + +#include "net/net.h" +#include "hw/net/vmxnet3.h" + +#define TYPE_VMXNET3 "vmxnet3" +#define VMXNET3(obj) OBJECT_CHECK(VMXNET3State, (obj), TYPE_VMXNET3) + +/* Device state and helper functions */ +#define VMXNET3_RX_RINGS_PER_QUEUE (2) + +/* Cyclic ring abstraction */ +typedef struct { + hwaddr pa; + uint32_t size; + uint32_t cell_size; + uint32_t next; + uint8_t gen; +} Vmxnet3Ring; + +typedef struct { + Vmxnet3Ring tx_ring; + Vmxnet3Ring comp_ring; + + uint8_t intr_idx; + hwaddr tx_stats_pa; + struct UPT1_TxStats txq_stats; +} Vmxnet3TxqDescr; + +typedef struct { + Vmxnet3Ring rx_ring[VMXNET3_RX_RINGS_PER_QUEUE]; + Vmxnet3Ring comp_ring; + uint8_t intr_idx; + hwaddr rx_stats_pa; + struct UPT1_RxStats rxq_stats; +} Vmxnet3RxqDescr; + +typedef struct { + bool is_masked; + bool is_pending; + bool is_asserted; +} Vmxnet3IntState; + +typedef struct { + PCIDevice parent_obj; + NICState *nic; + NICConf conf; + MemoryRegion bar0; + MemoryRegion bar1; + MemoryRegion msix_bar; + + Vmxnet3RxqDescr rxq_descr[VMXNET3_DEVICE_MAX_RX_QUEUES]; + Vmxnet3TxqDescr txq_descr[VMXNET3_DEVICE_MAX_TX_QUEUES]; + + /* Whether MSI-X support was installed successfully */ + bool msix_used; + hwaddr drv_shmem; + hwaddr temp_shared_guest_driver_memory; + + uint8_t txq_num; + + /* This boolean tells whether RX packet being indicated has to */ + /* be split into head and body chunks from different RX rings */ + bool rx_packets_compound; + + bool rx_vlan_stripping; + bool lro_supported; + + uint8_t rxq_num; + + /* Network MTU */ + uint32_t mtu; + + /* Maximum number of fragments for indicated TX packets */ + uint32_t max_tx_frags; + + /* Maximum number of fragments for indicated RX packets */ + uint16_t max_rx_frags; + + /* Index for events interrupt */ + uint8_t event_int_idx; + + /* Whether automatic interrupts masking enabled */ + bool auto_int_masking; + + bool peer_has_vhdr; + + /* TX packets to QEMU interface */ + struct NetTxPkt *tx_pkt; + uint32_t offload_mode; + uint32_t cso_or_gso_size; + uint16_t tci; + bool needs_vlan; + + struct NetRxPkt *rx_pkt; + + bool tx_sop; + bool skip_current_tx_pkt; + + uint32_t device_active; + uint32_t last_command; + + uint32_t link_status_and_speed; + + Vmxnet3IntState interrupt_states[VMXNET3_MAX_INTRS]; + + uint32_t temp_mac; /* To store the low part first */ + + MACAddr perm_mac; + uint32_t vlan_table[VMXNET3_VFT_SIZE]; + uint32_t rx_mode; + MACAddr *mcast_list; + uint32_t mcast_list_len; + uint32_t mcast_list_buff_size; /* needed for live migration. */ + + /* Compatibility flags for migration */ + uint32_t compat_flags; +} VMXNET3State; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095439697387.9247978087824; Mon, 12 Nov 2018 23:50:39 -0800 (PST) Received: from localhost ([::1]:52497 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTT4-000784-GU for importer@patchew.org; Tue, 13 Nov 2018 02:50:38 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52350) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvc-0005CX-1h for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvY-0004qi-AG for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:03 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44704) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvU-0004g9-AE for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:58 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EBl1078989; Tue, 13 Nov 2018 07:15:53 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nnwg18md4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:53 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FqBZ023949 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:52 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FpYP012407; Tue, 13 Nov 2018 07:15:52 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:51 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=GDZPjld51TPe5wjYbn7XLtT/O2pBEgsoWPYPPfvoI0I=; b=IH6/D/6QBHvKQtdR/k2rUq+7IG4RXSNq5EmpLFsRIc1V3BobwdiL2PpOKTzNtVt1f53M R0oW9YXnlIl66q6m/3WFrOjkZxjpSeR8lmqqQIgqLc++QhF5jTzPSyfznwjzlXDBpVcZ yU9HNK9KEPk509M0P+6TiffCINxabFMRrWMs64qG7YU0jAKdEN2TgxqKr1aeEe8iWGGv kelnzl/QiZYmb961slusnv4NLdhoT0K23701tbfKYkuysTIxcLpc/TIK4tXVFG86ODj8 9g+bQ5FM1jz3b08D2RufMPUhSwiQEpVQ9t9bS+N2XK0wryNIEEf9Zg6YH1mNgKRWfJ4G qw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:26 +0200 Message-Id: <20181113071336.6242-38-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=820 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 13/23] hw/pvrdma: Make sure PCI function 0 is vmxnet3 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Guest driver enforces it, we should also. Signed-off-by: Yuval Shaia --- hw/rdma/vmw/pvrdma.h | 2 ++ hw/rdma/vmw/pvrdma_main.c | 3 +++ 2 files changed, 5 insertions(+) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index b019cb843a..10a3c4fb7c 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -20,6 +20,7 @@ #include "hw/pci/pci.h" #include "hw/pci/msix.h" #include "chardev/char-fe.h" +#include "hw/net/vmxnet3_defs.h" =20 #include "../rdma_backend_defs.h" #include "../rdma_rm_defs.h" @@ -85,6 +86,7 @@ typedef struct PVRDMADev { RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; CharBackend mad_chr; + VMXNET3State *func0; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index ac8c092db0..fa6468d221 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -576,6 +576,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) return; } =20 + /* Break if not vmxnet3 device in slot 0 */ + dev->func0 =3D VMXNET3(pci_get_function_0(pdev)); + memdev_root =3D object_resolve_path("/objects", NULL); if (memdev_root) { object_child_foreach(memdev_root, pvrdma_check_ram_shared, &ram_sh= ared); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094302809358.42065345091896; Mon, 12 Nov 2018 23:31:42 -0800 (PST) Received: from localhost ([::1]:52398 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTAj-0000yH-IV for importer@patchew.org; Tue, 13 Nov 2018 02:31:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51824) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuS-0004BX-Ms for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuQ-0003Fo-4I for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:52 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:44964) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuP-0003BN-Qw for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:50 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DlPn037023; Tue, 13 Nov 2018 07:14:46 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2nnwc0gn0m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:45 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Ee8v010616 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:40 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EcZJ028189; Tue, 13 Nov 2018 07:14:38 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:38 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=GDZPjld51TPe5wjYbn7XLtT/O2pBEgsoWPYPPfvoI0I=; b=rf4ii5v/xS00n4TxcK+WY9hhZ6SgIlgJSN8Y+9e36h9J7UmQJKuCzZOWNsQ/FMlFcy57 /zMZRYBJNLpk3G4UgrKqZIZk4CrQcSucTojjFpbIyunVeq8nq/QghyMsv5RkI1ri6e8U gozmvm56DvCil2O2QckObhjn0VQ/1cd1Be6qPRkbzyIeQhukGaOS6/3qU+H1H2IM7qLB WpiQE7Ym6BWt1QjI5RajwdkJNp06c7OUr/8hxGrW7NJp2Qa4zcpstP0YSW0LAdeDC+0g 1gMVhcKMHy4i9bdF6CVv9UjuiaWeHRxAQLYSVIH55O9g3CnN5s5FLloxtLyIh0s0K3x4 1w== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:02 +0200 Message-Id: <20181113071336.6242-14-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=820 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 13/23] hw/pvrdma: Make sure PCI function 0 is vmxnet3 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Guest driver enforces it, we should also. Signed-off-by: Yuval Shaia --- hw/rdma/vmw/pvrdma.h | 2 ++ hw/rdma/vmw/pvrdma_main.c | 3 +++ 2 files changed, 5 insertions(+) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index b019cb843a..10a3c4fb7c 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -20,6 +20,7 @@ #include "hw/pci/pci.h" #include "hw/pci/msix.h" #include "chardev/char-fe.h" +#include "hw/net/vmxnet3_defs.h" =20 #include "../rdma_backend_defs.h" #include "../rdma_rm_defs.h" @@ -85,6 +86,7 @@ typedef struct PVRDMADev { RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; CharBackend mad_chr; + VMXNET3State *func0; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index ac8c092db0..fa6468d221 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -576,6 +576,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) return; } =20 + /* Break if not vmxnet3 device in slot 0 */ + dev->func0 =3D VMXNET3(pci_get_function_0(pdev)); + memdev_root =3D object_resolve_path("/objects", NULL); if (memdev_root) { object_child_foreach(memdev_root, pvrdma_check_ram_shared, &ram_sh= ared); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093771386397.063055941068; Mon, 12 Nov 2018 23:22:51 -0800 (PST) Received: from localhost ([::1]:52348 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT20-0001Ty-DA for importer@patchew.org; Tue, 13 Nov 2018 02:22:40 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51821) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuS-0004BT-MD for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuP-0003FT-UZ for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:52 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:43778) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuP-00037g-KE for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:49 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DskE078883; Tue, 13 Nov 2018 07:14:42 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2nnwg18m8e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:42 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EfNj010669 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:41 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EfaH028207; Tue, 13 Nov 2018 07:14:41 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:41 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=KhykC/+vqWNz2GvNfrMmhiLUkvic64U5uoAlUOxsEmk=; b=0Sc/Op7XzIZfmVS9QHthITQSOQz0vtwViYVoYly1XAKec9drr0sqqY0PjB2kC8IVtP11 E7KAiQ4Z/gnXc0KdmHfsBXND8EMvJg8Q7/265YIubqzNn5y95NTJOcHXh3CjZNbm+m4B B/79IQY05/Aj88GGcW0AkmLg04/fFD1Yo1uJmV7hdP46omj49yuX219TaF7I/fmzwGPc kLc/jKfyoRXxTpXMqe6/uat1TtHHsJtcfW8oRoUGsLgxAVrBgfkB7fn9211qvEHXEYJi yI6RB7YN0w0tKR2u6pkO6PBW1D/klzdLwIFWasHJmxNVkkb/CLAT3bFObMvTZUjNLCdK xQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:03 +0200 Message-Id: <20181113071336.6242-15-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 14/23] hw/rdma: Initialize node_guid from vmxnet3 mac address X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" node_guid should be set once device is load. Make node_guid be GID format (32 bit) of PCI function 0 vmxnet3 device's MAC. A new function was added to do the conversion. So for example the MAC 56:b6:44:e9:62:dc will be converted to GID 54b6:44ff:fee9:62dc. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_utils.h | 9 +++++++++ hw/rdma/vmw/pvrdma_cmd.c | 10 ---------- hw/rdma/vmw/pvrdma_main.c | 5 ++++- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index 989db249ef..202abb3366 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -63,4 +63,13 @@ extern unsigned long pr_dbg_cnt; void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len); =20 +static inline void addrconf_addr_eui48(uint8_t *eui, const char *addr) +{ + memcpy(eui, addr, 3); + eui[3] =3D 0xFF; + eui[4] =3D 0xFE; + memcpy(eui + 5, addr + 3, 3); + eui[0] ^=3D 2; +} + #endif diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index a334f6205e..2979582fac 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -592,16 +592,6 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cm= d_req *req, return -EINVAL; } =20 - /* TODO: Since drivers stores node_guid at load_dsr phase then this - * assignment is not relevant, i need to figure out a way how to - * retrieve MAC of our netdev */ - if (!cmd->index) { - dev->node_guid =3D - dev->rdma_dev_res.ports[0].gid_tbl[0].gid.global.interface_id; - pr_dbg("dev->node_guid=3D0x%llx\n", - (long long unsigned int)be64_to_cpu(dev->node_guid)); - } - return 0; } =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index fa6468d221..95e9322b7c 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -264,7 +264,7 @@ static void init_dsr_dev_caps(PVRDMADev *dev) dsr->caps.sys_image_guid =3D 0; pr_dbg("sys_image_guid=3D%" PRIx64 "\n", dsr->caps.sys_image_guid); =20 - dsr->caps.node_guid =3D cpu_to_be64(dev->node_guid); + dsr->caps.node_guid =3D dev->node_guid; pr_dbg("node_guid=3D%" PRIx64 "\n", be64_to_cpu(dsr->caps.node_guid)); =20 dsr->caps.phys_port_cnt =3D MAX_PORTS; @@ -579,6 +579,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) /* Break if not vmxnet3 device in slot 0 */ dev->func0 =3D VMXNET3(pci_get_function_0(pdev)); =20 + addrconf_addr_eui48((unsigned char *)&dev->node_guid, + (const char *)&dev->func0->conf.macaddr.a); + memdev_root =3D object_resolve_path("/objects", NULL); if (memdev_root) { object_child_foreach(memdev_root, pvrdma_check_ram_shared, &ram_sh= ared); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 154209560992620.372178505066472; Mon, 12 Nov 2018 23:53:29 -0800 (PST) Received: from localhost ([::1]:52521 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTVk-0000yJ-LX for importer@patchew.org; Tue, 13 Nov 2018 02:53:24 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52440) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvk-0005IM-Ip for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvh-0004yq-S9 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:12 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60746) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvf-0004rr-8m for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:07 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DmdA045961; Tue, 13 Nov 2018 07:16:01 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2nnw6egpv2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:01 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FtFU024083 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:55 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7FtUt012414; Tue, 13 Nov 2018 07:15:55 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:54 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=KhykC/+vqWNz2GvNfrMmhiLUkvic64U5uoAlUOxsEmk=; b=igL1/aTW/4KBQmr/jBqYnnKaYBrTPdZcVl4qAgPn38Fpi2cR0rEAEEzOY2Idc9X6EXy+ mW5lMX9sPKnULRrh248oLVUWzqLHZoyhPrXNJHA5sqGh76uKDDzQv/HtfEnVQglmIZpU Kd/iuYopquf8VJgp92fuvt7F6z6fmS4aLeUpR8xbBFz3b4cHmw6iVgpn8OTSZrEfqAIu WwVQVSJByaZqw5PBQsBTkDZJJo2cLYrn+qepSudpE6QCE3pLdPXXfH7dHRHGaqm4GTC4 hx95iRRyS5XoLIM8zaCWtTBEIARaI04r2RT7Stpv+gOh3elcX5+lEmZAXnIrm51ZD0jq ww== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:27 +0200 Message-Id: <20181113071336.6242-39-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 14/23] hw/rdma: Initialize node_guid from vmxnet3 mac address X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" node_guid should be set once device is load. Make node_guid be GID format (32 bit) of PCI function 0 vmxnet3 device's MAC. A new function was added to do the conversion. So for example the MAC 56:b6:44:e9:62:dc will be converted to GID 54b6:44ff:fee9:62dc. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_utils.h | 9 +++++++++ hw/rdma/vmw/pvrdma_cmd.c | 10 ---------- hw/rdma/vmw/pvrdma_main.c | 5 ++++- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index 989db249ef..202abb3366 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -63,4 +63,13 @@ extern unsigned long pr_dbg_cnt; void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len); =20 +static inline void addrconf_addr_eui48(uint8_t *eui, const char *addr) +{ + memcpy(eui, addr, 3); + eui[3] =3D 0xFF; + eui[4] =3D 0xFE; + memcpy(eui + 5, addr + 3, 3); + eui[0] ^=3D 2; +} + #endif diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index a334f6205e..2979582fac 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -592,16 +592,6 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cm= d_req *req, return -EINVAL; } =20 - /* TODO: Since drivers stores node_guid at load_dsr phase then this - * assignment is not relevant, i need to figure out a way how to - * retrieve MAC of our netdev */ - if (!cmd->index) { - dev->node_guid =3D - dev->rdma_dev_res.ports[0].gid_tbl[0].gid.global.interface_id; - pr_dbg("dev->node_guid=3D0x%llx\n", - (long long unsigned int)be64_to_cpu(dev->node_guid)); - } - return 0; } =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index fa6468d221..95e9322b7c 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -264,7 +264,7 @@ static void init_dsr_dev_caps(PVRDMADev *dev) dsr->caps.sys_image_guid =3D 0; pr_dbg("sys_image_guid=3D%" PRIx64 "\n", dsr->caps.sys_image_guid); =20 - dsr->caps.node_guid =3D cpu_to_be64(dev->node_guid); + dsr->caps.node_guid =3D dev->node_guid; pr_dbg("node_guid=3D%" PRIx64 "\n", be64_to_cpu(dsr->caps.node_guid)); =20 dsr->caps.phys_port_cnt =3D MAX_PORTS; @@ -579,6 +579,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) /* Break if not vmxnet3 device in slot 0 */ dev->func0 =3D VMXNET3(pci_get_function_0(pdev)); =20 + addrconf_addr_eui48((unsigned char *)&dev->node_guid, + (const char *)&dev->func0->conf.macaddr.a); + memdev_root =3D object_resolve_path("/objects", NULL); if (memdev_root) { object_child_foreach(memdev_root, pvrdma_check_ram_shared, &ram_sh= ared); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094128471924.6003273709867; Mon, 12 Nov 2018 23:28:48 -0800 (PST) Received: from localhost ([::1]:52381 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT7o-0006tO-O5 for importer@patchew.org; Tue, 13 Nov 2018 02:28:40 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51810) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuS-0004BB-DD for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuP-0003Ez-PA for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:52 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:59706) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuP-0003Bx-CC for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:49 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EBmO046135; Tue, 13 Nov 2018 07:14:46 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp2120.oracle.com with ESMTP id 2nnw6egppm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:46 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EjNN010486 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:45 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Ej19011902; Tue, 13 Nov 2018 07:14:45 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:44 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=l8vGF/W51A/3iW1mo+f2Z1ybpTNEpQ/j0asLFaGEpAU=; b=dl2OF1qViz/xOdGCStRmCce3F+Kpcln+jHSzC4fM4JPUGJOwEb0TVIZks/TJElfAgHZ3 tKq4sWkAvTiq1KebZlD/ToIjcLMltUa4u4N/ZU40mtrGoCTw5Vll08a8jZd8PXOTBQSB xQ0kxmyrWI18xGs1Puu89qFxeZb213lGNDuTid40qrSV/nrB12GQjbyl6zKtGB58w+s3 jxdUZrVKL3C7bDfNYtYR2Q5o4vfyTFCOsWiFg5iXnXdLJ5BWxHNyhd1dboKB+j9unINH oVuBJZaxtbLXLAi+pi3HE/r7JUs08YsQAOFuq6w083LeWGrVRnGWSJQ96tWVX50pnrBF /Q== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:04 +0200 Message-Id: <20181113071336.6242-16-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 15/23] hw/pvrdma: Make device state depend on Ethernet function state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" User should be able to control the device by changing Ethernet function state so if user runs 'ifconfig ens3 down' the PVRDMA function should be down as well. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_cmd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 2979582fac..0d3c818c20 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -139,7 +139,8 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_= req *req, resp->hdr.ack =3D PVRDMA_CMD_QUERY_PORT_RESP; resp->hdr.err =3D 0; =20 - resp->attrs.state =3D attrs.state; + resp->attrs.state =3D dev->func0->device_active ? attrs.state : + PVRDMA_PORT_DOWN; resp->attrs.max_mtu =3D attrs.max_mtu; resp->attrs.active_mtu =3D attrs.active_mtu; resp->attrs.phys_state =3D attrs.phys_state; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095230941314.75225448384276; Mon, 12 Nov 2018 23:47:10 -0800 (PST) Received: from localhost ([::1]:52476 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTPh-0004bg-OK for importer@patchew.org; Tue, 13 Nov 2018 02:47:09 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52436) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvk-0005IC-EU for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvh-0004yh-QX for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:12 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45974) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSve-0004qJ-0r for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:07 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EZsQ037533; Tue, 13 Nov 2018 07:15:59 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2nnwc0gn63-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:59 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Fw6u024187 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:58 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Fwn8028835; Tue, 13 Nov 2018 07:15:58 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:57 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=l8vGF/W51A/3iW1mo+f2Z1ybpTNEpQ/j0asLFaGEpAU=; b=GWbYsiFs5Z3eNPtjktxC2glXDCCrzwu3mKwpw1DwkPxZ+AhuwoEaNT4pHMevFzeqfp1V itZ4+Q0fPvtsE5hqqYqaRSs6mt8+Yf9PaZjL+beyCdVTfOTKByZlwJtbrulvLUUyIhpQ Ey/nxfuafUNgGhzplnRF4ncv+iiqp7oxb+T8wmiNSv6orqAvBI2wlPf8/31U8DJjJwHA jMVZqzOtBAZwpezXbfEMPcGjV6feN4QmUP6Di6B0TGX0xaD0pI+pZPWF8KMjxAzCDrjK 1LCavzhTEyIUv0f+80QHOzmsaGkkGOvJKE55HUHK4MK6lSwlyY9Cnmnqzt3z3NGVbTIi aw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:28 +0200 Message-Id: <20181113071336.6242-40-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 15/23] hw/pvrdma: Make device state depend on Ethernet function state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" User should be able to control the device by changing Ethernet function state so if user runs 'ifconfig ens3 down' the PVRDMA function should be down as well. Signed-off-by: Yuval Shaia --- hw/rdma/vmw/pvrdma_cmd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 2979582fac..0d3c818c20 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -139,7 +139,8 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_= req *req, resp->hdr.ack =3D PVRDMA_CMD_QUERY_PORT_RESP; resp->hdr.err =3D 0; =20 - resp->attrs.state =3D attrs.state; + resp->attrs.state =3D dev->func0->device_active ? attrs.state : + PVRDMA_PORT_DOWN; resp->attrs.max_mtu =3D attrs.max_mtu; resp->attrs.active_mtu =3D attrs.active_mtu; resp->attrs.phys_state =3D attrs.phys_state; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094724480747.4362805502642; Mon, 12 Nov 2018 23:38:44 -0800 (PST) Received: from localhost ([::1]:52431 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTHX-0006Pq-5a for importer@patchew.org; Tue, 13 Nov 2018 02:38:43 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52454) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvl-0005J9-GT for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvi-0004zV-Bn for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:13 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:46010) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvh-0004t1-Ud for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:10 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EHJM037457; Tue, 13 Nov 2018 07:16:02 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2nnwc0gn6b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:01 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7G1si019484 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:01 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7G1Gb012460; Tue, 13 Nov 2018 07:16:01 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:00 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=MdgNJ3zd8r65/2pfvDtcfBbyN5C5maVeU3+hWRiwSUU=; b=PfKoTAKbuOhcMx1kFiSBBE9+3PwGFLkRI38RZ89ev7VBGWWvFfxXGPCsEY/RDVoeEwBA cRHHy2E+JWjlOlWppPWSlVs57AliMu0tdyVNBywdOGGyUZWOOK7ieVQlRMOCsn8pza+2 rvHIVl4+2E5rtwFU5IDD/yyrkBcMAWlKLGgEky2mXLFmTj1Y1q41aYUbKe/7Jksr/FWK Ko4sYLpvJC3W9qncrTw727VCCGuklf1FGhJ6WHsYmWePlZeZkA+RGM+CkaXCGZzHtK9b x6rCvKII80br3pfDV9hZJCg6TJOzc6CFYvW44n65lX+soivmhO47rfz9sKsUZxG6h2KL aA== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:29 +0200 Message-Id: <20181113071336.6242-41-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 16/23] hw/pvrdma: Fill all CQE fields X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add ability to pass specific WC attributes to CQE such as GRH_BIT flag. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_backend.c | 59 +++++++++++++++++++++++-------------- hw/rdma/rdma_backend.h | 4 +-- hw/rdma/vmw/pvrdma_qp_ops.c | 31 +++++++++++-------- 3 files changed, 58 insertions(+), 36 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 5675504165..e453bda8f9 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -59,13 +59,24 @@ struct backend_umad { char mad[RDMA_MAX_PRIVATE_DATA]; }; =20 -static void (*comp_handler)(int status, unsigned int vendor_err, void *ctx= ); +static void (*comp_handler)(void *ctx, struct ibv_wc *wc); =20 -static void dummy_comp_handler(int status, unsigned int vendor_err, void *= ctx) +static void dummy_comp_handler(void *ctx, struct ibv_wc *wc) { pr_err("No completion handler is registered\n"); } =20 +static inline void complete_work(enum ibv_wc_status status, uint32_t vendo= r_err, + void *ctx) +{ + struct ibv_wc wc =3D {0}; + + wc.status =3D status; + wc.vendor_err =3D vendor_err; + + comp_handler(ctx, &wc); +} + static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq) { int i, ne; @@ -90,7 +101,7 @@ static void poll_cq(RdmaDeviceResources *rdma_dev_res, s= truct ibv_cq *ibcq) } pr_dbg("Processing %s CQE\n", bctx->is_tx_req ? "send" : "recv= "); =20 - comp_handler(wc[i].status, wc[i].vendor_err, bctx->up_ctx); + comp_handler(bctx->up_ctx, &wc[i]); =20 rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id); g_free(bctx); @@ -184,8 +195,8 @@ static void start_comp_thread(RdmaBackendDev *backend_d= ev) comp_handler_thread, backend_dev, QEMU_THREAD_DETAC= HED); } =20 -void rdma_backend_register_comp_handler(void (*handler)(int status, - unsigned int vendor_err, void *ctx= )) +void rdma_backend_register_comp_handler(void (*handler)(void *ctx, + struct ibv_wc *wc= )) { comp_handler =3D handler; } @@ -369,14 +380,14 @@ void rdma_backend_post_send(RdmaBackendDev *backend_d= ev, if (!qp->ibqp) { /* This field does not get initialized for QP0 and QP= 1 */ if (qp_type =3D=3D IBV_QPT_SMI) { pr_dbg("QP0 unsupported\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); rc =3D mad_send(backend_dev, sgid_idx, sgid, sge, num_sge); if (rc) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx= ); } else { - comp_handler(IBV_WC_SUCCESS, 0, ctx); + complete_work(IBV_WC_SUCCESS, 0, ctx); } } return; @@ -385,7 +396,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, pr_dbg("num_sge=3D%d\n", num_sge); if (!num_sge) { pr_dbg("num_sge=3D0\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); return; } =20 @@ -396,21 +407,21 @@ void rdma_backend_post_send(RdmaBackendDev *backend_d= ev, rc =3D rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx= ); if (unlikely(rc)) { pr_dbg("Failed to allocate cqe_ctx\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); goto out_free_bctx; } =20 rc =3D build_host_sge_array(backend_dev->rdma_dev_res, new_sge, sge, n= um_sge); if (rc) { pr_dbg("Error: Failed to build host SGE array\n"); - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); goto out_dealloc_cqe_ctx; } =20 if (qp_type =3D=3D IBV_QPT_UD) { wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, sgid_idx, dgid); if (!wr.wr.ud.ah) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx= ); goto out_dealloc_cqe_ctx; } wr.wr.ud.remote_qpn =3D dqpn; @@ -428,7 +439,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, if (rc) { pr_dbg("Fail (%d, %d) to post send WQE to qpn %d\n", rc, errno, qp->ibqp->qp_num); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; } =20 @@ -497,13 +508,13 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_d= ev, if (!qp->ibqp) { /* This field does not get initialized for QP0 and QP= 1 */ if (qp_type =3D=3D IBV_QPT_SMI) { pr_dbg("QP0 unsupported\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); rc =3D save_mad_recv_buffer(backend_dev, sge, num_sge, ctx); if (rc) { - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); } } return; @@ -512,7 +523,7 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, pr_dbg("num_sge=3D%d\n", num_sge); if (!num_sge) { pr_dbg("num_sge=3D0\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); return; } =20 @@ -523,14 +534,14 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_d= ev, rc =3D rdma_rm_alloc_cqe_ctx(rdma_dev_res, &bctx_id, bctx); if (unlikely(rc)) { pr_dbg("Failed to allocate cqe_ctx\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); goto out_free_bctx; } =20 rc =3D build_host_sge_array(rdma_dev_res, new_sge, sge, num_sge); if (rc) { pr_dbg("Error: Failed to build host SGE array\n"); - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); goto out_dealloc_cqe_ctx; } =20 @@ -542,7 +553,7 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, if (rc) { pr_dbg("Fail (%d, %d) to post recv WQE to qpn %d\n", rc, errno, qp->ibqp->qp_num); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; } =20 @@ -926,9 +937,10 @@ static void mad_read(void *opaque, const uint8_t *buf,= int size) mad =3D rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, bctx->sge.length); if (!mad || bctx->sge.length < msg->umad_len + MAD_HDR_SIZE) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, - bctx->up_ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, + bctx->up_ctx); } else { + struct ibv_wc wc =3D {0}; pr_dbg_buf("mad", msg->umad.mad, msg->umad_len); memset(mad, 0, bctx->sge.length); build_mad_hdr((struct ibv_grh *)mad, @@ -937,7 +949,10 @@ static void mad_read(void *opaque, const uint8_t *buf,= int size) memcpy(&mad[MAD_HDR_SIZE], msg->umad.mad, msg->umad_len); rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); =20 - comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); + wc.byte_len =3D msg->umad_len; + wc.status =3D IBV_WC_SUCCESS; + wc.wc_flags =3D IBV_WC_GRH; + comp_handler(bctx->up_ctx, &wc); } =20 g_free(bctx); diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 59ad2b874b..8cae40f827 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -57,8 +57,8 @@ int rdma_backend_get_gid_index(RdmaBackendDev *backend_de= v, union ibv_gid *gid); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); -void rdma_backend_register_comp_handler(void (*handler)(int status, - unsigned int vendor_err, void *ctx= )); +void rdma_backend_register_comp_handler(void (*handler)(void *ctx, + struct ibv_wc *wc)= ); void rdma_backend_unregister_comp_handler(void); =20 int rdma_backend_query_port(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 2130824098..300471a4c9 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -47,7 +47,7 @@ typedef struct PvrdmaRqWqe { * 3. Interrupt host */ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_handle, - struct pvrdma_cqe *cqe) + struct pvrdma_cqe *cqe, struct ibv_wc *wc) { struct pvrdma_cqe *cqe1; struct pvrdma_cqne *cqne; @@ -66,6 +66,7 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_ha= ndle, pr_dbg("Writing CQE\n"); cqe1 =3D pvrdma_ring_next_elem_write(ring); if (unlikely(!cqe1)) { + pr_dbg("No CQEs in ring\n"); return -EINVAL; } =20 @@ -73,8 +74,20 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_h= andle, cqe1->wr_id =3D cqe->wr_id; cqe1->qp =3D cqe->qp; cqe1->opcode =3D cqe->opcode; - cqe1->status =3D cqe->status; - cqe1->vendor_err =3D cqe->vendor_err; + cqe1->status =3D wc->status; + cqe1->byte_len =3D wc->byte_len; + cqe1->src_qp =3D wc->src_qp; + cqe1->wc_flags =3D wc->wc_flags; + cqe1->vendor_err =3D wc->vendor_err; + + pr_dbg("wr_id=3D%" PRIx64 "\n", cqe1->wr_id); + pr_dbg("qp=3D0x%lx\n", cqe1->qp); + pr_dbg("opcode=3D%d\n", cqe1->opcode); + pr_dbg("status=3D%d\n", cqe1->status); + pr_dbg("byte_len=3D%d\n", cqe1->byte_len); + pr_dbg("src_qp=3D%d\n", cqe1->src_qp); + pr_dbg("wc_flags=3D%d\n", cqe1->wc_flags); + pr_dbg("vendor_err=3D%d\n", cqe1->vendor_err); =20 pvrdma_ring_write_inc(ring); =20 @@ -99,18 +112,12 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq= _handle, return 0; } =20 -static void pvrdma_qp_ops_comp_handler(int status, unsigned int vendor_err, - void *ctx) +static void pvrdma_qp_ops_comp_handler(void *ctx, struct ibv_wc *wc) { CompHandlerCtx *comp_ctx =3D (CompHandlerCtx *)ctx; =20 - pr_dbg("cq_handle=3D%d\n", comp_ctx->cq_handle); - pr_dbg("wr_id=3D%" PRIx64 "\n", comp_ctx->cqe.wr_id); - pr_dbg("status=3D%d\n", status); - pr_dbg("vendor_err=3D0x%x\n", vendor_err); - comp_ctx->cqe.status =3D status; - comp_ctx->cqe.vendor_err =3D vendor_err; - pvrdma_post_cqe(comp_ctx->dev, comp_ctx->cq_handle, &comp_ctx->cqe); + pvrdma_post_cqe(comp_ctx->dev, comp_ctx->cq_handle, &comp_ctx->cqe, wc= ); + g_free(ctx); } =20 --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093936219817.3855010141867; Mon, 12 Nov 2018 23:25:36 -0800 (PST) Received: from localhost ([::1]:52361 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT4p-0004Fr-1g for importer@patchew.org; Tue, 13 Nov 2018 02:25:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51857) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuU-0004Cx-6K for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuR-0003Hp-9I for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:54 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45014) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuQ-0003GI-VN for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:51 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EZsL037533; Tue, 13 Nov 2018 07:14:49 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2nnwc0gn0q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:49 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7Emwx010841 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:48 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7Emai030347; Tue, 13 Nov 2018 07:14:48 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:47 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=MdgNJ3zd8r65/2pfvDtcfBbyN5C5maVeU3+hWRiwSUU=; b=r7s5ug64Fu7q/joq3MLK9gqPqn2iwbiJrMw/3sWwgCZ4gWO23bNTdyyaX7Sq+v2eeV84 93+gKVBhKaL+Pqr27s1YasbdhAZ76mvSu1vlQALR5ldu+voU4lWr/O2Ywsw7p9kTrV3d DCNZVt7J0OSZqIjE/h80/NKmC8cNGS67ZZ++0oWwBp/tF/6PjceOIFEmhHMiJmFR/ko/ efw+OFdfs6aYnkycCn4jxfJuHv+fsa8x3a8YZgUiGse0OuQudqofGYuzb1S24UymZo7U bv0LobOOFoEwpHua3Bb+1l19jhGaE5SynRUJ0nWVUD7hx/thHeuY1zVfqcRs+DuT/3i+ MA== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:05 +0200 Message-Id: <20181113071336.6242-17-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 16/23] hw/pvrdma: Fill all CQE fields X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add ability to pass specific WC attributes to CQE such as GRH_BIT flag. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 59 +++++++++++++++++++++++-------------- hw/rdma/rdma_backend.h | 4 +-- hw/rdma/vmw/pvrdma_qp_ops.c | 31 +++++++++++-------- 3 files changed, 58 insertions(+), 36 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 5675504165..e453bda8f9 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -59,13 +59,24 @@ struct backend_umad { char mad[RDMA_MAX_PRIVATE_DATA]; }; =20 -static void (*comp_handler)(int status, unsigned int vendor_err, void *ctx= ); +static void (*comp_handler)(void *ctx, struct ibv_wc *wc); =20 -static void dummy_comp_handler(int status, unsigned int vendor_err, void *= ctx) +static void dummy_comp_handler(void *ctx, struct ibv_wc *wc) { pr_err("No completion handler is registered\n"); } =20 +static inline void complete_work(enum ibv_wc_status status, uint32_t vendo= r_err, + void *ctx) +{ + struct ibv_wc wc =3D {0}; + + wc.status =3D status; + wc.vendor_err =3D vendor_err; + + comp_handler(ctx, &wc); +} + static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq) { int i, ne; @@ -90,7 +101,7 @@ static void poll_cq(RdmaDeviceResources *rdma_dev_res, s= truct ibv_cq *ibcq) } pr_dbg("Processing %s CQE\n", bctx->is_tx_req ? "send" : "recv= "); =20 - comp_handler(wc[i].status, wc[i].vendor_err, bctx->up_ctx); + comp_handler(bctx->up_ctx, &wc[i]); =20 rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id); g_free(bctx); @@ -184,8 +195,8 @@ static void start_comp_thread(RdmaBackendDev *backend_d= ev) comp_handler_thread, backend_dev, QEMU_THREAD_DETAC= HED); } =20 -void rdma_backend_register_comp_handler(void (*handler)(int status, - unsigned int vendor_err, void *ctx= )) +void rdma_backend_register_comp_handler(void (*handler)(void *ctx, + struct ibv_wc *wc= )) { comp_handler =3D handler; } @@ -369,14 +380,14 @@ void rdma_backend_post_send(RdmaBackendDev *backend_d= ev, if (!qp->ibqp) { /* This field does not get initialized for QP0 and QP= 1 */ if (qp_type =3D=3D IBV_QPT_SMI) { pr_dbg("QP0 unsupported\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); rc =3D mad_send(backend_dev, sgid_idx, sgid, sge, num_sge); if (rc) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx= ); } else { - comp_handler(IBV_WC_SUCCESS, 0, ctx); + complete_work(IBV_WC_SUCCESS, 0, ctx); } } return; @@ -385,7 +396,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, pr_dbg("num_sge=3D%d\n", num_sge); if (!num_sge) { pr_dbg("num_sge=3D0\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); return; } =20 @@ -396,21 +407,21 @@ void rdma_backend_post_send(RdmaBackendDev *backend_d= ev, rc =3D rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx= ); if (unlikely(rc)) { pr_dbg("Failed to allocate cqe_ctx\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); goto out_free_bctx; } =20 rc =3D build_host_sge_array(backend_dev->rdma_dev_res, new_sge, sge, n= um_sge); if (rc) { pr_dbg("Error: Failed to build host SGE array\n"); - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); goto out_dealloc_cqe_ctx; } =20 if (qp_type =3D=3D IBV_QPT_UD) { wr.wr.ud.ah =3D create_ah(backend_dev, qp->ibpd, sgid_idx, dgid); if (!wr.wr.ud.ah) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx= ); goto out_dealloc_cqe_ctx; } wr.wr.ud.remote_qpn =3D dqpn; @@ -428,7 +439,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, if (rc) { pr_dbg("Fail (%d, %d) to post send WQE to qpn %d\n", rc, errno, qp->ibqp->qp_num); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; } =20 @@ -497,13 +508,13 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_d= ev, if (!qp->ibqp) { /* This field does not get initialized for QP0 and QP= 1 */ if (qp_type =3D=3D IBV_QPT_SMI) { pr_dbg("QP0 unsupported\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } if (qp_type =3D=3D IBV_QPT_GSI) { pr_dbg("QP1\n"); rc =3D save_mad_recv_buffer(backend_dev, sge, num_sge, ctx); if (rc) { - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); } } return; @@ -512,7 +523,7 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, pr_dbg("num_sge=3D%d\n", num_sge); if (!num_sge) { pr_dbg("num_sge=3D0\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); return; } =20 @@ -523,14 +534,14 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_d= ev, rc =3D rdma_rm_alloc_cqe_ctx(rdma_dev_res, &bctx_id, bctx); if (unlikely(rc)) { pr_dbg("Failed to allocate cqe_ctx\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); goto out_free_bctx; } =20 rc =3D build_host_sge_array(rdma_dev_res, new_sge, sge, num_sge); if (rc) { pr_dbg("Error: Failed to build host SGE array\n"); - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); goto out_dealloc_cqe_ctx; } =20 @@ -542,7 +553,7 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, if (rc) { pr_dbg("Fail (%d, %d) to post recv WQE to qpn %d\n", rc, errno, qp->ibqp->qp_num); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; } =20 @@ -926,9 +937,10 @@ static void mad_read(void *opaque, const uint8_t *buf,= int size) mad =3D rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, bctx->sge.length); if (!mad || bctx->sge.length < msg->umad_len + MAD_HDR_SIZE) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, - bctx->up_ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, + bctx->up_ctx); } else { + struct ibv_wc wc =3D {0}; pr_dbg_buf("mad", msg->umad.mad, msg->umad_len); memset(mad, 0, bctx->sge.length); build_mad_hdr((struct ibv_grh *)mad, @@ -937,7 +949,10 @@ static void mad_read(void *opaque, const uint8_t *buf,= int size) memcpy(&mad[MAD_HDR_SIZE], msg->umad.mad, msg->umad_len); rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); =20 - comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); + wc.byte_len =3D msg->umad_len; + wc.status =3D IBV_WC_SUCCESS; + wc.wc_flags =3D IBV_WC_GRH; + comp_handler(bctx->up_ctx, &wc); } =20 g_free(bctx); diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 59ad2b874b..8cae40f827 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -57,8 +57,8 @@ int rdma_backend_get_gid_index(RdmaBackendDev *backend_de= v, union ibv_gid *gid); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); -void rdma_backend_register_comp_handler(void (*handler)(int status, - unsigned int vendor_err, void *ctx= )); +void rdma_backend_register_comp_handler(void (*handler)(void *ctx, + struct ibv_wc *wc)= ); void rdma_backend_unregister_comp_handler(void); =20 int rdma_backend_query_port(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 2130824098..300471a4c9 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -47,7 +47,7 @@ typedef struct PvrdmaRqWqe { * 3. Interrupt host */ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_handle, - struct pvrdma_cqe *cqe) + struct pvrdma_cqe *cqe, struct ibv_wc *wc) { struct pvrdma_cqe *cqe1; struct pvrdma_cqne *cqne; @@ -66,6 +66,7 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_ha= ndle, pr_dbg("Writing CQE\n"); cqe1 =3D pvrdma_ring_next_elem_write(ring); if (unlikely(!cqe1)) { + pr_dbg("No CQEs in ring\n"); return -EINVAL; } =20 @@ -73,8 +74,20 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_h= andle, cqe1->wr_id =3D cqe->wr_id; cqe1->qp =3D cqe->qp; cqe1->opcode =3D cqe->opcode; - cqe1->status =3D cqe->status; - cqe1->vendor_err =3D cqe->vendor_err; + cqe1->status =3D wc->status; + cqe1->byte_len =3D wc->byte_len; + cqe1->src_qp =3D wc->src_qp; + cqe1->wc_flags =3D wc->wc_flags; + cqe1->vendor_err =3D wc->vendor_err; + + pr_dbg("wr_id=3D%" PRIx64 "\n", cqe1->wr_id); + pr_dbg("qp=3D0x%lx\n", cqe1->qp); + pr_dbg("opcode=3D%d\n", cqe1->opcode); + pr_dbg("status=3D%d\n", cqe1->status); + pr_dbg("byte_len=3D%d\n", cqe1->byte_len); + pr_dbg("src_qp=3D%d\n", cqe1->src_qp); + pr_dbg("wc_flags=3D%d\n", cqe1->wc_flags); + pr_dbg("vendor_err=3D%d\n", cqe1->vendor_err); =20 pvrdma_ring_write_inc(ring); =20 @@ -99,18 +112,12 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq= _handle, return 0; } =20 -static void pvrdma_qp_ops_comp_handler(int status, unsigned int vendor_err, - void *ctx) +static void pvrdma_qp_ops_comp_handler(void *ctx, struct ibv_wc *wc) { CompHandlerCtx *comp_ctx =3D (CompHandlerCtx *)ctx; =20 - pr_dbg("cq_handle=3D%d\n", comp_ctx->cq_handle); - pr_dbg("wr_id=3D%" PRIx64 "\n", comp_ctx->cqe.wr_id); - pr_dbg("status=3D%d\n", status); - pr_dbg("vendor_err=3D0x%x\n", vendor_err); - comp_ctx->cqe.status =3D status; - comp_ctx->cqe.vendor_err =3D vendor_err; - pvrdma_post_cqe(comp_ctx->dev, comp_ctx->cq_handle, &comp_ctx->cqe); + pvrdma_post_cqe(comp_ctx->dev, comp_ctx->cq_handle, &comp_ctx->cqe, wc= ); + g_free(ctx); } =20 --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094909864486.2776773660048; Mon, 12 Nov 2018 23:41:49 -0800 (PST) Received: from localhost ([::1]:52448 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTKW-0000Lu-KN for importer@patchew.org; Tue, 13 Nov 2018 02:41:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52472) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvm-0005KT-GS for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvj-00050Z-DG for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:14 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60868) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvj-0004zv-2u for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:11 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Dxvb046016; Tue, 13 Nov 2018 07:16:10 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2120.oracle.com with ESMTP id 2nnw6egpvt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:10 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7G4dV015430 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:05 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7G4p8012487; Tue, 13 Nov 2018 07:16:04 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:03 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=zWVPitBAel5tOaDstYqdzfKofjMIXBmuS2lHaQgidv8=; b=rnGPb8Rve5wHkB5RIHD+950XqFKwBMpfFfxeGQ6Q4aOpSOAEPX1RPmjawtkXhJGmAJ76 JrbgPUMhqnpGO5Cf704MaAeo38YZRqNYAGUPXXzmJFBHGCbc3968/xx1SBQOWLFc7ght 5RzzcVfqRecmwChs69v6Hjus1RXQzbxakNuZ9QcvtkvOKnjG7PCf6IDpCwrV9DxhfuyH p0Lo5iWQAe78YC6/H6GfyoKIWDSqVkzJiGbqobdIaN1e6RMi239Cr1GwBHDjvDk6epT1 wPf0EK6UsTW4Ud3/EBxgEC/j96xEVGmvWwEpX3yrDguL24toHdu2EZmazVftqVoVDMnF xA== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:30 +0200 Message-Id: <20181113071336.6242-42-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=824 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 17/23] hw/pvrdma: Fill error code in command's response X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Driver checks error code let's set it. Signed-off-by: Yuval Shaia --- hw/rdma/vmw/pvrdma_cmd.c | 67 ++++++++++++++++++++++++++++------------ 1 file changed, 48 insertions(+), 19 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 0d3c818c20..a326c5d470 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -131,7 +131,8 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 if (rdma_backend_query_port(&dev->backend_dev, (struct ibv_port_attr *)&attrs)) { - return -ENOMEM; + resp->hdr.err =3D -ENOMEM; + goto out; } =20 memset(resp, 0, sizeof(*resp)); @@ -150,7 +151,9 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_= req *req, resp->attrs.active_width =3D 1; resp->attrs.active_speed =3D 1; =20 - return 0; +out: + pr_dbg("ret=3D%d\n", resp->hdr.err); + return resp->hdr.err; } =20 static int query_pkey(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -170,7 +173,7 @@ static int query_pkey(PVRDMADev *dev, union pvrdma_cmd_= req *req, resp->pkey =3D PVRDMA_PKEY; pr_dbg("pkey=3D0x%x\n", resp->pkey); =20 - return 0; + return resp->hdr.err; } =20 static int create_pd(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -200,7 +203,9 @@ static int destroy_pd(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 rdma_rm_dealloc_pd(&dev->rdma_dev_res, cmd->pd_handle); =20 - return 0; + rsp->hdr.err =3D 0; + + return rsp->hdr.err; } =20 static int create_mr(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -251,7 +256,9 @@ static int destroy_mr(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 rdma_rm_dealloc_mr(&dev->rdma_dev_res, cmd->mr_handle); =20 - return 0; + rsp->hdr.err =3D 0; + + return rsp->hdr.err; } =20 static int create_cq_ring(PCIDevice *pci_dev , PvrdmaRing **ring, @@ -353,7 +360,8 @@ static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd_= req *req, cq =3D rdma_rm_get_cq(&dev->rdma_dev_res, cmd->cq_handle); if (!cq) { pr_dbg("Invalid CQ handle\n"); - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 ring =3D (PvrdmaRing *)cq->opaque; @@ -364,7 +372,11 @@ static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd= _req *req, =20 rdma_rm_dealloc_cq(&dev->rdma_dev_res, cmd->cq_handle); =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int create_qp_rings(PCIDevice *pci_dev, uint64_t pdir_dma, @@ -553,7 +565,8 @@ static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd_= req *req, qp =3D rdma_rm_get_qp(&dev->rdma_dev_res, cmd->qp_handle); if (!qp) { pr_dbg("Invalid QP handle\n"); - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 rdma_rm_dealloc_qp(&dev->rdma_dev_res, cmd->qp_handle); @@ -567,7 +580,11 @@ static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd= _req *req, rdma_pci_dma_unmap(PCI_DEVICE(dev), ring->ring_state, TARGET_PAGE_SIZE= ); g_free(ring); =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -580,7 +597,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd= _req *req, pr_dbg("index=3D%d\n", cmd->index); =20 if (cmd->index >=3D MAX_PORT_GIDS) { - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 pr_dbg("gid[%d]=3D0x%llx,0x%llx\n", cmd->index, @@ -590,10 +608,15 @@ static int create_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, rc =3D rdma_rm_add_gid(&dev->rdma_dev_res, &dev->backend_dev, dev->backend_eth_device_name, gid, cmd->index); if (rc < 0) { - return -EINVAL; + rsp->hdr.err =3D rc; + goto out; } =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -606,7 +629,8 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_cm= d_req *req, pr_dbg("index=3D%d\n", cmd->index); =20 if (cmd->index >=3D MAX_PORT_GIDS) { - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 rc =3D rdma_rm_del_gid(&dev->rdma_dev_res, &dev->backend_dev, @@ -617,7 +641,11 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, goto out; } =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int create_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -634,9 +662,8 @@ static int create_uc(PVRDMADev *dev, union pvrdma_cmd_r= eq *req, resp->hdr.err =3D rdma_rm_alloc_uc(&dev->rdma_dev_res, cmd->pfn, &resp->ctx_handle); =20 - pr_dbg("ret=3D%d\n", resp->hdr.err); - - return 0; + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int destroy_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -648,7 +675,9 @@ static int destroy_uc(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 rdma_rm_dealloc_uc(&dev->rdma_dev_res, cmd->ctx_handle); =20 - return 0; + rsp->hdr.err =3D 0; + + return rsp->hdr.err; } struct cmd_handler { uint32_t cmd; @@ -696,7 +725,7 @@ int execute_command(PVRDMADev *dev) } =20 err =3D cmd_handlers[dsr_info->req->hdr.cmd].exec(dev, dsr_info->req, - dsr_info->rsp); + dsr_info->rsp); out: set_reg_val(dev, PVRDMA_REG_ERR, err); post_interrupt(dev, INTR_VEC_CMD_RING); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542093968158938.3494338600285; Mon, 12 Nov 2018 23:26:08 -0800 (PST) Received: from localhost ([::1]:52369 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT5K-0004jO-P8 for importer@patchew.org; Tue, 13 Nov 2018 02:26:06 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51879) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuV-0004Eo-Uk for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuU-0003Mp-Jw for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:55 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:43868) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuU-0003Li-Af for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:54 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DqgI078880; Tue, 13 Nov 2018 07:14:52 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2nnwg18m8x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:52 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EphO010992 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:51 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EpcW011936; Tue, 13 Nov 2018 07:14:51 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:50 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=zWVPitBAel5tOaDstYqdzfKofjMIXBmuS2lHaQgidv8=; b=eisDX6h398R6ouaNAjrujo/WSDHxPVTq9hh4XrpHYccRDdZWcygr+y2OhMpeoCoeR3mx igBKNLyhatgXDOMs2nR8Uow9WeLIQr5ZebnqMchjA9nCfz8psot1ffWh4ls/8xGacnWU t/sGRd5sULa14m8lkMbFFFtQr8psamPKZDwk1XlTOBdhYphSw2Ign0H3Rg0T5pL7x1x7 8pENyo3voRtWT0dekke9FtB6STZkSekBifwDSphWnM1hkaefPKPNSRo9K6nV5bgX4ZjA IPTgI9fFMRnDqZv9/kBwcF5SwHYHnDXLpglf0qvKZbdKoijO9XVKNCXxpVYzn2K8mRwc oQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:06 +0200 Message-Id: <20181113071336.6242-18-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=824 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 17/23] hw/pvrdma: Fill error code in command's response X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Driver checks error code let's set it. Signed-off-by: Yuval Shaia --- hw/rdma/vmw/pvrdma_cmd.c | 67 ++++++++++++++++++++++++++++------------ 1 file changed, 48 insertions(+), 19 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 0d3c818c20..a326c5d470 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -131,7 +131,8 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 if (rdma_backend_query_port(&dev->backend_dev, (struct ibv_port_attr *)&attrs)) { - return -ENOMEM; + resp->hdr.err =3D -ENOMEM; + goto out; } =20 memset(resp, 0, sizeof(*resp)); @@ -150,7 +151,9 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_= req *req, resp->attrs.active_width =3D 1; resp->attrs.active_speed =3D 1; =20 - return 0; +out: + pr_dbg("ret=3D%d\n", resp->hdr.err); + return resp->hdr.err; } =20 static int query_pkey(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -170,7 +173,7 @@ static int query_pkey(PVRDMADev *dev, union pvrdma_cmd_= req *req, resp->pkey =3D PVRDMA_PKEY; pr_dbg("pkey=3D0x%x\n", resp->pkey); =20 - return 0; + return resp->hdr.err; } =20 static int create_pd(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -200,7 +203,9 @@ static int destroy_pd(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 rdma_rm_dealloc_pd(&dev->rdma_dev_res, cmd->pd_handle); =20 - return 0; + rsp->hdr.err =3D 0; + + return rsp->hdr.err; } =20 static int create_mr(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -251,7 +256,9 @@ static int destroy_mr(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 rdma_rm_dealloc_mr(&dev->rdma_dev_res, cmd->mr_handle); =20 - return 0; + rsp->hdr.err =3D 0; + + return rsp->hdr.err; } =20 static int create_cq_ring(PCIDevice *pci_dev , PvrdmaRing **ring, @@ -353,7 +360,8 @@ static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd_= req *req, cq =3D rdma_rm_get_cq(&dev->rdma_dev_res, cmd->cq_handle); if (!cq) { pr_dbg("Invalid CQ handle\n"); - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 ring =3D (PvrdmaRing *)cq->opaque; @@ -364,7 +372,11 @@ static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd= _req *req, =20 rdma_rm_dealloc_cq(&dev->rdma_dev_res, cmd->cq_handle); =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int create_qp_rings(PCIDevice *pci_dev, uint64_t pdir_dma, @@ -553,7 +565,8 @@ static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd_= req *req, qp =3D rdma_rm_get_qp(&dev->rdma_dev_res, cmd->qp_handle); if (!qp) { pr_dbg("Invalid QP handle\n"); - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 rdma_rm_dealloc_qp(&dev->rdma_dev_res, cmd->qp_handle); @@ -567,7 +580,11 @@ static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd= _req *req, rdma_pci_dma_unmap(PCI_DEVICE(dev), ring->ring_state, TARGET_PAGE_SIZE= ); g_free(ring); =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -580,7 +597,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd= _req *req, pr_dbg("index=3D%d\n", cmd->index); =20 if (cmd->index >=3D MAX_PORT_GIDS) { - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 pr_dbg("gid[%d]=3D0x%llx,0x%llx\n", cmd->index, @@ -590,10 +608,15 @@ static int create_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, rc =3D rdma_rm_add_gid(&dev->rdma_dev_res, &dev->backend_dev, dev->backend_eth_device_name, gid, cmd->index); if (rc < 0) { - return -EINVAL; + rsp->hdr.err =3D rc; + goto out; } =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -606,7 +629,8 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_cm= d_req *req, pr_dbg("index=3D%d\n", cmd->index); =20 if (cmd->index >=3D MAX_PORT_GIDS) { - return -EINVAL; + rsp->hdr.err =3D -EINVAL; + goto out; } =20 rc =3D rdma_rm_del_gid(&dev->rdma_dev_res, &dev->backend_dev, @@ -617,7 +641,11 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_c= md_req *req, goto out; } =20 - return 0; + rsp->hdr.err =3D 0; + +out: + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int create_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -634,9 +662,8 @@ static int create_uc(PVRDMADev *dev, union pvrdma_cmd_r= eq *req, resp->hdr.err =3D rdma_rm_alloc_uc(&dev->rdma_dev_res, cmd->pfn, &resp->ctx_handle); =20 - pr_dbg("ret=3D%d\n", resp->hdr.err); - - return 0; + pr_dbg("ret=3D%d\n", rsp->hdr.err); + return rsp->hdr.err; } =20 static int destroy_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -648,7 +675,9 @@ static int destroy_uc(PVRDMADev *dev, union pvrdma_cmd_= req *req, =20 rdma_rm_dealloc_uc(&dev->rdma_dev_res, cmd->ctx_handle); =20 - return 0; + rsp->hdr.err =3D 0; + + return rsp->hdr.err; } struct cmd_handler { uint32_t cmd; @@ -696,7 +725,7 @@ int execute_command(PVRDMADev *dev) } =20 err =3D cmd_handlers[dsr_info->req->hdr.cmd].exec(dev, dsr_info->req, - dsr_info->rsp); + dsr_info->rsp); out: set_reg_val(dev, PVRDMA_REG_ERR, err); post_interrupt(dev, INTR_VEC_CMD_RING); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095050655216.71241101689145; Mon, 12 Nov 2018 23:44:10 -0800 (PST) Received: from localhost ([::1]:52458 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTMn-0002FF-Bg for importer@patchew.org; Tue, 13 Nov 2018 02:44:09 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52453) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvl-0005J8-GY for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvi-0004zP-BK for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:13 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60840) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvh-0004xq-Tt for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:10 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EShK046231; Tue, 13 Nov 2018 07:16:08 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2120.oracle.com with ESMTP id 2nnw6egpvq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:08 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7G7PD019746 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:07 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7G701018180; Tue, 13 Nov 2018 07:16:07 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:06 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=uAuMMDzBddedr6EFDuWspzX1l76aCl6ov7jAvUvUMbE=; b=2YIDzVP5kA9Bbq9BIh9GwJejQ9pn7NW3/1NLCIIxJbsGDTCbMbj6aqAENFsCV4miH0BW 0bN6pPz9U2YWkdk9ExsuOg8/FyJ18kg1ptdHJN21T0YcI32eIyXEqFwt8LdSRcyrBR3z 87CpPrXUqIZ+RYJskYFmLFQp0dqe4Zk5J2GNUEYorMBItHcOiDn/ly80sA86y7lP9tl3 3st1QrmjF2iRofXGvQU/rRNYmsRrmB8wZYd1pQ+NGZghE+GBv+TWljiqlltw/VmmFGIG sy9JO+T1WdPxj+BC7FKLh5R7Qx96VLx9LqXxOIWRZ3/y/4C4MRDvcX3m3GM5MINc4TRz YQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:31 +0200 Message-Id: <20181113071336.6242-43-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 18/23] hw/rdma: Remove unneeded code that handles more that one port X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Device supports only one port, let's remove a dead code that handles more than one port. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_rm.c | 34 ++++++++++++++++------------------ hw/rdma/rdma_rm.h | 2 +- hw/rdma/rdma_rm_defs.h | 4 ++-- 3 files changed, 19 insertions(+), 21 deletions(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index fe0979415d..0a5ab8935a 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -545,7 +545,7 @@ int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaB= ackendDev *backend_dev, return -EINVAL; } =20 - memcpy(&dev_res->ports[0].gid_tbl[gid_idx].gid, gid, sizeof(*gid)); + memcpy(&dev_res->port.gid_tbl[gid_idx].gid, gid, sizeof(*gid)); =20 return 0; } @@ -556,15 +556,15 @@ int rdma_rm_del_gid(RdmaDeviceResources *dev_res, Rdm= aBackendDev *backend_dev, int rc; =20 rc =3D rdma_backend_del_gid(backend_dev, ifname, - &dev_res->ports[0].gid_tbl[gid_idx].gid); + &dev_res->port.gid_tbl[gid_idx].gid); if (rc < 0) { pr_dbg("Fail to delete gid\n"); return -EINVAL; } =20 - memset(dev_res->ports[0].gid_tbl[gid_idx].gid.raw, 0, - sizeof(dev_res->ports[0].gid_tbl[gid_idx].gid)); - dev_res->ports[0].gid_tbl[gid_idx].backend_gid_index =3D -1; + memset(dev_res->port.gid_tbl[gid_idx].gid.raw, 0, + sizeof(dev_res->port.gid_tbl[gid_idx].gid)); + dev_res->port.gid_tbl[gid_idx].backend_gid_index =3D -1; =20 return 0; } @@ -577,16 +577,16 @@ int rdma_rm_get_backend_gid_index(RdmaDeviceResources= *dev_res, return -EINVAL; } =20 - if (unlikely(dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D= =3D -1)) { - dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D + if (unlikely(dev_res->port.gid_tbl[sgid_idx].backend_gid_index =3D=3D = -1)) { + dev_res->port.gid_tbl[sgid_idx].backend_gid_index =3D rdma_backend_get_gid_index(backend_dev, - &dev_res->ports[0].gid_tbl[sgid_idx= ].gid); + &dev_res->port.gid_tbl[sgid_idx].gid); } =20 pr_dbg("backend_gid_index=3D%d\n", - dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index); + dev_res->port.gid_tbl[sgid_idx].backend_gid_index); =20 - return dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index; + return dev_res->port.gid_tbl[sgid_idx].backend_gid_index; } =20 static void destroy_qp_hash_key(gpointer data) @@ -596,15 +596,13 @@ static void destroy_qp_hash_key(gpointer data) =20 static void init_ports(RdmaDeviceResources *dev_res) { - int i, j; + int i; =20 - memset(dev_res->ports, 0, sizeof(dev_res->ports)); + memset(&dev_res->port, 0, sizeof(dev_res->port)); =20 - for (i =3D 0; i < MAX_PORTS; i++) { - dev_res->ports[i].state =3D IBV_PORT_DOWN; - for (j =3D 0; j < MAX_PORT_GIDS; j++) { - dev_res->ports[i].gid_tbl[j].backend_gid_index =3D -1; - } + dev_res->port.state =3D IBV_PORT_DOWN; + for (i =3D 0; i < MAX_PORT_GIDS; i++) { + dev_res->port.gid_tbl[i].backend_gid_index =3D -1; } } =20 @@ -613,7 +611,7 @@ static void fini_ports(RdmaDeviceResources *dev_res, { int i; =20 - dev_res->ports[0].state =3D IBV_PORT_DOWN; + dev_res->port.state =3D IBV_PORT_DOWN; for (i =3D 0; i < MAX_PORT_GIDS; i++) { rdma_rm_del_gid(dev_res, backend_dev, ifname, i); } diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index a7169b4e89..3c602c04c0 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -79,7 +79,7 @@ int rdma_rm_get_backend_gid_index(RdmaDeviceResources *de= v_res, static inline union ibv_gid *rdma_rm_get_gid(RdmaDeviceResources *dev_res, int sgid_idx) { - return &dev_res->ports[0].gid_tbl[sgid_idx].gid; + return &dev_res->port.gid_tbl[sgid_idx].gid; } =20 #endif diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 7b3435f991..0ba61d1838 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -18,7 +18,7 @@ =20 #include "rdma_backend_defs.h" =20 -#define MAX_PORTS 1 +#define MAX_PORTS 1 /* Do not change - we support only one por= t */ #define MAX_PORT_GIDS 255 #define MAX_GIDS MAX_PORT_GIDS #define MAX_PORT_PKEYS 1 @@ -97,7 +97,7 @@ typedef struct RdmaRmPort { } RdmaRmPort; =20 typedef struct RdmaDeviceResources { - RdmaRmPort ports[MAX_PORTS]; + RdmaRmPort port; RdmaRmResTbl pd_tbl; RdmaRmResTbl mr_tbl; RdmaRmResTbl uc_tbl; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094136304455.84335516366843; Mon, 12 Nov 2018 23:28:56 -0800 (PST) Received: from localhost ([::1]:52380 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT7i-0006pL-EH for importer@patchew.org; Tue, 13 Nov 2018 02:28:34 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51954) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSue-0004Lr-LS for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuc-0003Xb-08 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:04 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:59900) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSub-0003Ws-P6 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:01 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EBmQ046135; Tue, 13 Nov 2018 07:15:00 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2120.oracle.com with ESMTP id 2nnw6egpqg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:00 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EsUq016222 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:54 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7EsmS030443; Tue, 13 Nov 2018 07:14:54 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:53 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=uAuMMDzBddedr6EFDuWspzX1l76aCl6ov7jAvUvUMbE=; b=S6kIhwpitdVYEKdTp6b0WF+UaC78z/xuuOVdKf3c6wwKp+AMhwUGx42NA45KdNEvRjYc JiEd9oqWjHrYQOvvucEi1xiyssmTAWGYtDmyRjGrUediwLbDbnBHTfi59kEzo7xb5gLJ d7Vv6SuilI9R9A+u7T/2ffAQ8gqBXSxuKYd4FHwUqc8quy+JcvmD8NDwE2Z/tAEoAXEQ 0jti+EsHJaTgLXRngX+6ochUnEJs8wXJa2fFEAdwpvf/AR5YUnvuGmUn8l6JjQyDEIYB 7pFZUfLRYHZSQWgK7/30kLAl2g8MadHn2KZcJjChLB1iCOJx2QAGNyhBLuJx35kMCakv Lw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:07 +0200 Message-Id: <20181113071336.6242-19-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 18/23] hw/rdma: Remove unneeded code that handles more that one port X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Device supports only one port, let's remove a dead code that handles more than one port. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 34 ++++++++++++++++------------------ hw/rdma/rdma_rm.h | 2 +- hw/rdma/rdma_rm_defs.h | 4 ++-- 3 files changed, 19 insertions(+), 21 deletions(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index fe0979415d..0a5ab8935a 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -545,7 +545,7 @@ int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaB= ackendDev *backend_dev, return -EINVAL; } =20 - memcpy(&dev_res->ports[0].gid_tbl[gid_idx].gid, gid, sizeof(*gid)); + memcpy(&dev_res->port.gid_tbl[gid_idx].gid, gid, sizeof(*gid)); =20 return 0; } @@ -556,15 +556,15 @@ int rdma_rm_del_gid(RdmaDeviceResources *dev_res, Rdm= aBackendDev *backend_dev, int rc; =20 rc =3D rdma_backend_del_gid(backend_dev, ifname, - &dev_res->ports[0].gid_tbl[gid_idx].gid); + &dev_res->port.gid_tbl[gid_idx].gid); if (rc < 0) { pr_dbg("Fail to delete gid\n"); return -EINVAL; } =20 - memset(dev_res->ports[0].gid_tbl[gid_idx].gid.raw, 0, - sizeof(dev_res->ports[0].gid_tbl[gid_idx].gid)); - dev_res->ports[0].gid_tbl[gid_idx].backend_gid_index =3D -1; + memset(dev_res->port.gid_tbl[gid_idx].gid.raw, 0, + sizeof(dev_res->port.gid_tbl[gid_idx].gid)); + dev_res->port.gid_tbl[gid_idx].backend_gid_index =3D -1; =20 return 0; } @@ -577,16 +577,16 @@ int rdma_rm_get_backend_gid_index(RdmaDeviceResources= *dev_res, return -EINVAL; } =20 - if (unlikely(dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D= =3D -1)) { - dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index =3D + if (unlikely(dev_res->port.gid_tbl[sgid_idx].backend_gid_index =3D=3D = -1)) { + dev_res->port.gid_tbl[sgid_idx].backend_gid_index =3D rdma_backend_get_gid_index(backend_dev, - &dev_res->ports[0].gid_tbl[sgid_idx= ].gid); + &dev_res->port.gid_tbl[sgid_idx].gid); } =20 pr_dbg("backend_gid_index=3D%d\n", - dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index); + dev_res->port.gid_tbl[sgid_idx].backend_gid_index); =20 - return dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index; + return dev_res->port.gid_tbl[sgid_idx].backend_gid_index; } =20 static void destroy_qp_hash_key(gpointer data) @@ -596,15 +596,13 @@ static void destroy_qp_hash_key(gpointer data) =20 static void init_ports(RdmaDeviceResources *dev_res) { - int i, j; + int i; =20 - memset(dev_res->ports, 0, sizeof(dev_res->ports)); + memset(&dev_res->port, 0, sizeof(dev_res->port)); =20 - for (i =3D 0; i < MAX_PORTS; i++) { - dev_res->ports[i].state =3D IBV_PORT_DOWN; - for (j =3D 0; j < MAX_PORT_GIDS; j++) { - dev_res->ports[i].gid_tbl[j].backend_gid_index =3D -1; - } + dev_res->port.state =3D IBV_PORT_DOWN; + for (i =3D 0; i < MAX_PORT_GIDS; i++) { + dev_res->port.gid_tbl[i].backend_gid_index =3D -1; } } =20 @@ -613,7 +611,7 @@ static void fini_ports(RdmaDeviceResources *dev_res, { int i; =20 - dev_res->ports[0].state =3D IBV_PORT_DOWN; + dev_res->port.state =3D IBV_PORT_DOWN; for (i =3D 0; i < MAX_PORT_GIDS; i++) { rdma_rm_del_gid(dev_res, backend_dev, ifname, i); } diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index a7169b4e89..3c602c04c0 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -79,7 +79,7 @@ int rdma_rm_get_backend_gid_index(RdmaDeviceResources *de= v_res, static inline union ibv_gid *rdma_rm_get_gid(RdmaDeviceResources *dev_res, int sgid_idx) { - return &dev_res->ports[0].gid_tbl[sgid_idx].gid; + return &dev_res->port.gid_tbl[sgid_idx].gid; } =20 #endif diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 7b3435f991..0ba61d1838 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -18,7 +18,7 @@ =20 #include "rdma_backend_defs.h" =20 -#define MAX_PORTS 1 +#define MAX_PORTS 1 /* Do not change - we support only one por= t */ #define MAX_PORT_GIDS 255 #define MAX_GIDS MAX_PORT_GIDS #define MAX_PORT_PKEYS 1 @@ -97,7 +97,7 @@ typedef struct RdmaRmPort { } RdmaRmPort; =20 typedef struct RdmaDeviceResources { - RdmaRmPort ports[MAX_PORTS]; + RdmaRmPort port; RdmaRmResTbl pd_tbl; RdmaRmResTbl mr_tbl; RdmaRmResTbl uc_tbl; --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094135969935.7028762128162; Mon, 12 Nov 2018 23:28:55 -0800 (PST) Received: from localhost ([::1]:52382 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMT7x-00076O-RM for importer@patchew.org; Tue, 13 Nov 2018 02:28:49 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51928) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSuc-0004Jo-Hi for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuZ-0003Uz-Tq for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:02 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45132) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuZ-0003Tr-Jb for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:14:59 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DktJ037018; Tue, 13 Nov 2018 07:14:58 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2nnwc0gn17-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:58 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EvGr011301 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:14:57 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7EvIf028290; Tue, 13 Nov 2018 07:14:57 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:56 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=eoefEC/OJV5MyfxvT9f/Fvlkhj7rI2tusr0W5VF7rY8=; b=XHS9D3ciQoGmGiVhLk4AXB2ue5qOSsMR/VuWcayPRfcbQeaWML5hReEkIY8JvQIa3/FB W7vWiqmoERpaF3KCqI3mHjOFDnKrJNWAdL9MrZI/ZNrk0baZOwXI1SnhUWX3wl0qjyWN VFlg4Kj2xkz4x49Z56HHD8UeFuOe1rRB/oe+qIlhDvRp3YB3kJDxq9BYlKoe7asDQYuD vSnUzCow3X+KRg3eOSlw4bQBs4wwWZejqPa85kOJh5gC+Gl2OkBmU8aSqkG9fx89VdYv yNvaUH1fCqCs5ms/FdrR9sbEOwfABSdtsn8oiztTWtw/uHHptKURbbYudbTNOxEuq9UO rg== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:08 +0200 Message-Id: <20181113071336.6242-20-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=863 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 19/23] vl: Introduce shutdown_notifiers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Notifier will be used for signaling shutdown event to inform system is shutdown. This will allow devices and other component to run some cleanup code needed before VM is shutdown. Signed-off-by: Yuval Shaia Reviewed-by: Cornelia Huck --- include/sysemu/sysemu.h | 1 + vl.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h index 8d6095d98b..0d15f16492 100644 --- a/include/sysemu/sysemu.h +++ b/include/sysemu/sysemu.h @@ -80,6 +80,7 @@ void qemu_register_wakeup_notifier(Notifier *notifier); void qemu_system_shutdown_request(ShutdownCause reason); void qemu_system_powerdown_request(void); void qemu_register_powerdown_notifier(Notifier *notifier); +void qemu_register_shutdown_notifier(Notifier *notifier); void qemu_system_debug_request(void); void qemu_system_vmstop_request(RunState reason); void qemu_system_vmstop_request_prepare(void); diff --git a/vl.c b/vl.c index 1fcacc5caa..d33d52522c 100644 --- a/vl.c +++ b/vl.c @@ -1578,6 +1578,8 @@ static NotifierList suspend_notifiers =3D NOTIFIER_LIST_INITIALIZER(suspend_notifiers); static NotifierList wakeup_notifiers =3D NOTIFIER_LIST_INITIALIZER(wakeup_notifiers); +static NotifierList shutdown_notifiers =3D + NOTIFIER_LIST_INITIALIZER(shutdown_notifiers); static uint32_t wakeup_reason_mask =3D ~(1 << QEMU_WAKEUP_REASON_NONE); =20 ShutdownCause qemu_shutdown_requested_get(void) @@ -1809,6 +1811,12 @@ static void qemu_system_powerdown(void) notifier_list_notify(&powerdown_notifiers, NULL); } =20 +static void qemu_system_shutdown(ShutdownCause cause) +{ + qapi_event_send_shutdown(shutdown_caused_by_guest(cause)); + notifier_list_notify(&shutdown_notifiers, &cause); +} + void qemu_system_powerdown_request(void) { trace_qemu_system_powerdown_request(); @@ -1821,6 +1829,11 @@ void qemu_register_powerdown_notifier(Notifier *noti= fier) notifier_list_add(&powerdown_notifiers, notifier); } =20 +void qemu_register_shutdown_notifier(Notifier *notifier) +{ + notifier_list_add(&shutdown_notifiers, notifier); +} + void qemu_system_debug_request(void) { debug_requested =3D 1; @@ -1848,7 +1861,7 @@ static bool main_loop_should_exit(void) request =3D qemu_shutdown_requested(); if (request) { qemu_kill_report(); - qapi_event_send_shutdown(shutdown_caused_by_guest(request)); + qemu_system_shutdown(request); if (no_shutdown) { vm_stop(RUN_STATE_SHUTDOWN); } else { --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095458519902.4877648541805; Mon, 12 Nov 2018 23:50:58 -0800 (PST) Received: from localhost ([::1]:52500 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTTN-0007OV-8n for importer@patchew.org; Tue, 13 Nov 2018 02:50:57 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52457) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvl-0005JC-IE for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvj-00051B-Q2 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:13 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60870) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvj-00050C-GW for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:11 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EShL046231; Tue, 13 Nov 2018 07:16:10 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2nnw6egpvu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:10 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7GAYD025066 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:10 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7GAUp031067; Tue, 13 Nov 2018 07:16:10 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:09 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=eoefEC/OJV5MyfxvT9f/Fvlkhj7rI2tusr0W5VF7rY8=; b=C+t+c547GFew9pssazigMFKr7cuua0FBkhoX1Km8iayEyXG+0nz8PHsnnjR+A3FKJd0p ApybrPE+V8MPyfVv0xbi0AqfbeYU/W+4ojr5myB1eBDWx10dQ0CmcCDtl2LtASS5mafJ k7Scpvg6AVNubhI8NqFv+wsUI/YO25Et1Zwo20VS7Hsm0f7/VUqUAOsS7LW9b3M9emqG mxPa+nO/dqEXbv9eeLiL9TAMZ/HA2Qb9c7gPnEF6z7nkt4MKF5N3EIdgv0nxa5bfzTJ1 Z+aqSfIo5JH/hnwbM30NKSAoKIyCzUb2RSvlCV5zOHdSCfs9neuqwnVjJ1EDTO1a4Y+8 sg== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:32 +0200 Message-Id: <20181113071336.6242-44-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=863 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 19/23] vl: Introduce shutdown_notifiers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Notifier will be used for signaling shutdown event to inform system is shutdown. This will allow devices and other component to run some cleanup code needed before VM is shutdown. Signed-off-by: Yuval Shaia --- include/sysemu/sysemu.h | 1 + vl.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h index 8d6095d98b..0d15f16492 100644 --- a/include/sysemu/sysemu.h +++ b/include/sysemu/sysemu.h @@ -80,6 +80,7 @@ void qemu_register_wakeup_notifier(Notifier *notifier); void qemu_system_shutdown_request(ShutdownCause reason); void qemu_system_powerdown_request(void); void qemu_register_powerdown_notifier(Notifier *notifier); +void qemu_register_shutdown_notifier(Notifier *notifier); void qemu_system_debug_request(void); void qemu_system_vmstop_request(RunState reason); void qemu_system_vmstop_request_prepare(void); diff --git a/vl.c b/vl.c index 1fcacc5caa..d33d52522c 100644 --- a/vl.c +++ b/vl.c @@ -1578,6 +1578,8 @@ static NotifierList suspend_notifiers =3D NOTIFIER_LIST_INITIALIZER(suspend_notifiers); static NotifierList wakeup_notifiers =3D NOTIFIER_LIST_INITIALIZER(wakeup_notifiers); +static NotifierList shutdown_notifiers =3D + NOTIFIER_LIST_INITIALIZER(shutdown_notifiers); static uint32_t wakeup_reason_mask =3D ~(1 << QEMU_WAKEUP_REASON_NONE); =20 ShutdownCause qemu_shutdown_requested_get(void) @@ -1809,6 +1811,12 @@ static void qemu_system_powerdown(void) notifier_list_notify(&powerdown_notifiers, NULL); } =20 +static void qemu_system_shutdown(ShutdownCause cause) +{ + qapi_event_send_shutdown(shutdown_caused_by_guest(cause)); + notifier_list_notify(&shutdown_notifiers, &cause); +} + void qemu_system_powerdown_request(void) { trace_qemu_system_powerdown_request(); @@ -1821,6 +1829,11 @@ void qemu_register_powerdown_notifier(Notifier *noti= fier) notifier_list_add(&powerdown_notifiers, notifier); } =20 +void qemu_register_shutdown_notifier(Notifier *notifier) +{ + notifier_list_add(&shutdown_notifiers, notifier); +} + void qemu_system_debug_request(void) { debug_requested =3D 1; @@ -1848,7 +1861,7 @@ static bool main_loop_should_exit(void) request =3D qemu_shutdown_requested(); if (request) { qemu_kill_report(); - qapi_event_send_shutdown(shutdown_caused_by_guest(request)); + qemu_system_shutdown(request); if (no_shutdown) { vm_stop(RUN_STATE_SHUTDOWN); } else { --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094319365539.4535016073481; Mon, 12 Nov 2018 23:31:59 -0800 (PST) Received: from localhost ([::1]:52399 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTAy-00015m-3z for importer@patchew.org; Tue, 13 Nov 2018 02:31:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51972) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSug-0004Pg-Jj for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSud-0003a9-Sd for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:06 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45212) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSud-0003Z0-Gz for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:03 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DoGJ037287; Tue, 13 Nov 2018 07:15:02 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2nnwc0gn1w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:01 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7F0dL011501 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:01 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7F0Fq017724; Tue, 13 Nov 2018 07:15:00 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:14:59 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=9eRXkQbyyMZ5WgMskyD2f5Oi5/sCXopRvEqHTguzMkk=; b=rSoc+iiGqT133QwB5Mo/mHCCU4J71IGt7as8REwwi3KJlukYYE/gHxp0GCBK4NcSft9H jlp2cwSEGpL3Rr4fiaJEDdYQcb6n3os6UUKRK96BntsK6hFlt+5pOhrX0/DWe/eoZvwA 8I5ujeI6JGysijR11Jy+fYYGOTr2QJi6km1W3MhGVtprbUUIrpuVJ1Ag3s1Hytj08GgS aTGRccLCO2d6DYJIAMgjGxP2CQwoK+FGcg3BY05nlq+mhQbPS/88+FsuqXsGgRk9sON8 5AX7Q5k07m6qISMlH88Z0VS94S3q0c/bTMTjDMwSQhi4TlFgNKHQC+/p46PhSSVE/AEk aA== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:09 +0200 Message-Id: <20181113071336.6242-21-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 20/23] hw/pvrdma: Clean device's resource when system is shutdown X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" In order to clean some external resources such as GIDs, QPs etc, register to receive notification when VM is shutdown. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma.h | 2 ++ hw/rdma/vmw/pvrdma_main.c | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index 10a3c4fb7c..ffae36986e 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -17,6 +17,7 @@ #define PVRDMA_PVRDMA_H =20 #include "qemu/units.h" +#include "qemu/notify.h" #include "hw/pci/pci.h" #include "hw/pci/msix.h" #include "chardev/char-fe.h" @@ -87,6 +88,7 @@ typedef struct PVRDMADev { RdmaDeviceResources rdma_dev_res; CharBackend mad_chr; VMXNET3State *func0; + Notifier shutdown_notifier; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 95e9322b7c..45a59cddf9 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -24,6 +24,7 @@ #include "hw/qdev-properties.h" #include "cpu.h" #include "trace.h" +#include "sysemu/sysemu.h" =20 #include "../rdma_rm.h" #include "../rdma_backend.h" @@ -559,6 +560,14 @@ static int pvrdma_check_ram_shared(Object *obj, void *= opaque) return 0; } =20 +static void pvrdma_shutdown_notifier(Notifier *n, void *opaque) +{ + PVRDMADev *dev =3D container_of(n, PVRDMADev, shutdown_notifier); + PCIDevice *pci_dev =3D PCI_DEVICE(dev); + + pvrdma_fini(pci_dev); +} + static void pvrdma_realize(PCIDevice *pdev, Error **errp) { int rc; @@ -623,6 +632,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) goto out; } =20 + dev->shutdown_notifier.notify =3D pvrdma_shutdown_notifier; + qemu_register_shutdown_notifier(&dev->shutdown_notifier); + out: if (rc) { error_append_hint(errp, "Device fail to load\n"); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095418932271.0422161287688; Mon, 12 Nov 2018 23:50:18 -0800 (PST) Received: from localhost ([::1]:52496 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTSj-0006s5-KJ for importer@patchew.org; Tue, 13 Nov 2018 02:50:17 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52514) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvq-0005Op-Px for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:22 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvo-000551-61 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:18 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44968) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvn-00054G-TM for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:16 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Dlmh078853; Tue, 13 Nov 2018 07:16:14 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nnwg18mej-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:13 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7GDLA019975 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:13 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7GD0G028996; Tue, 13 Nov 2018 07:16:13 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:12 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=9eRXkQbyyMZ5WgMskyD2f5Oi5/sCXopRvEqHTguzMkk=; b=wVOVC6uiK3BeZzQtIH/8ejucmqsxXOHlvwgUUTIRe8FRRtC4TeMlXshOr5kMvChmexxc pv+qoLUurkTHbgvVYr+1UvpgC9lDbt/tpSWyBzBkpoiqUwD6zYBTEZGfKAoBBxpy2K6D eVHCH53wnDLCvoXdmiIp6ruObQj6FZcBHMZiCSnJuXVG2VnAoURVM8vWLAydgzUcIZrS oTBw4bkHjBRbch97gooRNsQSXXUtvzUleBMo+EwwqfPgxcbIXN1HxjplyJF3OqfkxEe3 Os6n0vZY+0EBR4Zz6kbdSch2k2lEnkhZVGCIkLezR+bT+MHEsLvVpVJt4wdNJcKy128Q Fw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:33 +0200 Message-Id: <20181113071336.6242-45-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 20/23] hw/pvrdma: Clean device's resource when system is shutdown X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" In order to clean some external resources such as GIDs, QPs etc, register to receive notification when VM is shutdown. Signed-off-by: Yuval Shaia --- hw/rdma/vmw/pvrdma.h | 2 ++ hw/rdma/vmw/pvrdma_main.c | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index 10a3c4fb7c..ffae36986e 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -17,6 +17,7 @@ #define PVRDMA_PVRDMA_H =20 #include "qemu/units.h" +#include "qemu/notify.h" #include "hw/pci/pci.h" #include "hw/pci/msix.h" #include "chardev/char-fe.h" @@ -87,6 +88,7 @@ typedef struct PVRDMADev { RdmaDeviceResources rdma_dev_res; CharBackend mad_chr; VMXNET3State *func0; + Notifier shutdown_notifier; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) =20 diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 95e9322b7c..45a59cddf9 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -24,6 +24,7 @@ #include "hw/qdev-properties.h" #include "cpu.h" #include "trace.h" +#include "sysemu/sysemu.h" =20 #include "../rdma_rm.h" #include "../rdma_backend.h" @@ -559,6 +560,14 @@ static int pvrdma_check_ram_shared(Object *obj, void *= opaque) return 0; } =20 +static void pvrdma_shutdown_notifier(Notifier *n, void *opaque) +{ + PVRDMADev *dev =3D container_of(n, PVRDMADev, shutdown_notifier); + PCIDevice *pci_dev =3D PCI_DEVICE(dev); + + pvrdma_fini(pci_dev); +} + static void pvrdma_realize(PCIDevice *pdev, Error **errp) { int rc; @@ -623,6 +632,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **err= p) goto out; } =20 + dev->shutdown_notifier.notify =3D pvrdma_shutdown_notifier; + qemu_register_shutdown_notifier(&dev->shutdown_notifier); + out: if (rc) { error_append_hint(errp, "Device fail to load\n"); --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095622920893.6528159101659; Mon, 12 Nov 2018 23:53:42 -0800 (PST) Received: from localhost ([::1]:52526 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTW1-0001Ay-NM for importer@patchew.org; Tue, 13 Nov 2018 02:53:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52529) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvu-0005Qk-Ow for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvr-00057P-0V for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:22 -0500 Received: from aserp2120.oracle.com ([141.146.126.78]:60938) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvq-00056l-N0 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:18 -0500 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7E2Gq046085; Tue, 13 Nov 2018 07:16:16 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2120.oracle.com with ESMTP id 2nnw6egpw4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:16 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7GGAo025301 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:16 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7GGlR031091; Tue, 13 Nov 2018 07:16:16 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:15 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=jYlRlxzcZYsxJzt7HWUCpfcwnJ3VpNl/Fn0/X3fP85w=; b=HUkqklPYggR0b//iNpfhwhfL9vdr3X5s+qCgxR7EEG0M9r0sLfbBmonGYu4PU6XFvhP0 D12ucsjKiTL5lyCbfcjDKy8k/8oMO5RqBBysuPDTpkXYqx8UQyaCarU2yJ/EZuPKlQ/u Zrtj5SKfQSTffuzrDWAU/jQ3XFoY5AhYKtHsDuGqneAu7mFCZp1lGv1GiLpilqIEt/pD dNRJxecInwaS+jdLMHWOiNyimuc/Ekq9+KocqAd6+gVk/wNNkGYVG5XVhmaY0JNCKHF0 TrDgUP5oa3AO3UGxuPhj7Hy2iBT+FG6BBORp8A7URNAQUtf0Id9KlmvdgGAy8dVfgXh7 Tw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:34 +0200 Message-Id: <20181113071336.6242-46-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=719 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.78 Subject: [Qemu-devel] [PATCH v3 21/23] hw/rdma: Do not use bitmap_zero_extend to free bitmap X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" bitmap_zero_extend is designed to work for extending, not for shrinking. Using g_free instead. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_rm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 0a5ab8935a..35a96d9a64 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -43,7 +43,7 @@ static inline void res_tbl_free(RdmaRmResTbl *tbl) { qemu_mutex_destroy(&tbl->lock); g_free(tbl->tbl); - bitmap_zero_extend(tbl->bitmap, tbl->tbl_sz, 0); + g_free(tbl->bitmap); } =20 static inline void *res_tbl_get(RdmaRmResTbl *tbl, uint32_t handle) --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: temperror (zoho.com: Error in retrieving data from DNS) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 154209430222678.31477333752048; Mon, 12 Nov 2018 23:31:42 -0800 (PST) Received: from localhost ([::1]:52397 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTAj-0000uk-4t for importer@patchew.org; Tue, 13 Nov 2018 02:31:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51982) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSui-0004Rg-J7 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSuf-0003cK-MY for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:08 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45264) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSuf-0003bM-E0 for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:05 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EHJG037457; Tue, 13 Nov 2018 07:15:04 GMT Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2nnwc0gn2d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:03 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7F3OJ011472 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:03 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7F3Mi028424; Tue, 13 Nov 2018 07:15:03 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:02 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=jYlRlxzcZYsxJzt7HWUCpfcwnJ3VpNl/Fn0/X3fP85w=; b=TiSvn0lz7Ji32tsrAByD6FYD3O3Si/5gnk/I/x9jpkJuxYg9+C2QP6iW3PpAe6HtrEa7 7vuRTJ/AMlDAlS0gk1Ls6q0OS40shKAIefyIy262b4WWgoNSxhoX0x/OPCJKFB6vgyrg mM5Yr/q5nssWMvKFf0zjszU1ipIKp4n8W+tdvg/C+aokCeKhN7sGJ5gEXqbb6JeeOzob 4CuPBSOBalZd5X3mCr/gEqOLr7dCYCsCIGz1AIt0Fplbc0+FrRo0mc3ECDtT5plkjNMo 7VByt97jJkNPvoj3t6kJXBrYZLJPhrnrb/AcLLEAN+23OsHQYYh03tYNZPPLiG9zk15B ig== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:10 +0200 Message-Id: <20181113071336.6242-22-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=719 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 21/23] hw/rdma: Do not use bitmap_zero_extend to free bitmap X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" bitmap_zero_extend is designed to work for extending, not for shrinking. Using g_free instead. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 0a5ab8935a..35a96d9a64 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -43,7 +43,7 @@ static inline void res_tbl_free(RdmaRmResTbl *tbl) { qemu_mutex_destroy(&tbl->lock); g_free(tbl->tbl); - bitmap_zero_extend(tbl->bitmap, tbl->tbl_sz, 0); + g_free(tbl->bitmap); } =20 static inline void *res_tbl_get(RdmaRmResTbl *tbl, uint32_t handle) --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094517387859.9247215763912; Mon, 12 Nov 2018 23:35:17 -0800 (PST) Received: from localhost ([::1]:52411 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTE9-0003mr-3p for importer@patchew.org; Tue, 13 Nov 2018 02:35:13 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52008) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSux-0004hp-GI for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSup-0003tG-UL for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:20 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:44062) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSul-0003gl-Nk for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:12 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Do7x078867; Tue, 13 Nov 2018 07:15:08 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nnwg18ma6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:08 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7F6Aa022087 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:07 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7F6k4012065; Tue, 13 Nov 2018 07:15:06 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:05 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=OH4Ki7gnTgYSrH1EXkvCgitZAsyYgUeuJh3rpCpFR+Q=; b=ZoNdzx4c8BMsKgHVguq1Ygw0uQ2TLHMaDIbX5Wo1GcX6sdhxQZaOyxhQ3BDPNBRuDQVu RUiqggZ8PgVzAQRdXw/Ksk9zsHmz/6UoUAvRzcgoenb8HE/RFQFw3egF//VE/L6crYWL 04iZ4NIY68uKfaWj7NVEQoAsZCWxTDzLACIopf6vKTWJr9NHPdcRrMT5NceenCqupGm9 Sp+70SB+sM5lUlcyz5oqPxIM5M3edOGkrjr2H5TDMjCHljtTDLyHZ/yRva1qUxN00I0I gUlRAziYWIDeRo3o+cqoSaiAxyQ3lLKkH8wXXTLq3JUqKI5RsfV1SUETE6QJzhPyEcQj BA== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:11 +0200 Message-Id: <20181113071336.6242-23-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=802 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 22/23] hw/rdma: Do not call rdma_backend_del_gid on an empty gid X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When device goes down the function fini_ports loops over all entries in gid table regardless of the fact whether entry is valid or not. In case that entry is not valid we'd like to skip from any further processing in backend device. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 35a96d9a64..e3f6b2f6ea 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -555,6 +555,10 @@ int rdma_rm_del_gid(RdmaDeviceResources *dev_res, Rdma= BackendDev *backend_dev, { int rc; =20 + if (!dev_res->port.gid_tbl[gid_idx].gid.global.interface_id) { + return 0; + } + rc =3D rdma_backend_del_gid(backend_dev, ifname, &dev_res->port.gid_tbl[gid_idx].gid); if (rc < 0) { --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095079923883.0576447517611; Mon, 12 Nov 2018 23:44:39 -0800 (PST) Received: from localhost ([::1]:52460 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTNG-0002dr-OU for importer@patchew.org; Tue, 13 Nov 2018 02:44:38 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52547) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSvz-0005Sv-3u for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSvv-0005AJ-CK for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:27 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:46294) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvu-00059E-TE for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:23 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7DmhN037161; Tue, 13 Nov 2018 07:16:21 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2nnwc0gn7q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:20 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7GJxj025421 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:20 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7GJu3012609; Tue, 13 Nov 2018 07:16:19 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:18 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=OH4Ki7gnTgYSrH1EXkvCgitZAsyYgUeuJh3rpCpFR+Q=; b=SXSKV7EGrEtnvORu4C6Je1R4DGigICUbJUyfVKiVb8OsBAdz7Um3QxzAcLsl4mxpfTOH WR0yQ0aqY4vI51A/jYX1o5eyLFiTugAo2EUuXojshH+VL3le0rcSnQ3BUwPTBJ2fZ+S9 R10GkFRgLayR5ED+XuilXfJN03GWWMF1CqIp/sGgvD6/XCCWio9yURThzhZAaPHyROBo 1HRrwmfRRFq0ftd4PyTWwj5JSn66O/leCni8Bk8DAFTnEhXX5F2wDXU60eJnCmll7DFW 2pdsPdtByffJsSC4J82emvG19acU4DwtSpjS03FoaFKrfKm2TyjOkYX7yTDC1Ob/d9us kw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:35 +0200 Message-Id: <20181113071336.6242-47-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=802 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 22/23] hw/rdma: Do not call rdma_backend_del_gid on an empty gid X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When device goes down the function fini_ports loops over all entries in gid table regardless of the fact whether entry is valid or not. In case that entry is not valid we'd like to skip from any further processing in backend device. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_rm.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 35a96d9a64..e3f6b2f6ea 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -555,6 +555,10 @@ int rdma_rm_del_gid(RdmaDeviceResources *dev_res, Rdma= BackendDev *backend_dev, { int rc; =20 + if (!dev_res->port.gid_tbl[gid_idx].gid.global.interface_id) { + return 0; + } + rc =3D rdma_backend_del_gid(backend_dev, ifname, &dev_res->port.gid_tbl[gid_idx].gid); if (rc < 0) { --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542094515483923.0502833651834; Mon, 12 Nov 2018 23:35:15 -0800 (PST) Received: from localhost ([::1]:52410 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTDz-0003e5-ON for importer@patchew.org; Tue, 13 Nov 2018 02:35:03 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52038) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSv3-0004mp-2G for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSux-00041C-4t for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:27 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:45350) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSur-0003kA-Cz for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:15:19 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7Dokq037290; Tue, 13 Nov 2018 07:15:10 GMT Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2nnwc0gn2r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:10 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7F9jF012238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:15:09 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7F90u028469; Tue, 13 Nov 2018 07:15:09 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:15:08 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=Mw/y2lwgfEa7jCpGfWJaFRrwAfCgYeubThGvsG0oSGg=; b=tyFcfMEz6yQ3H3OLot2iz3jRgerOxBc+OxNGFFz88mXuwBrExX+qsnXE45cQeDSJVmwL 8vF+BsiZVRnQr9uL51ROctJQnVX1tVRNOxTi8eqruZ1Qjo1FBqUmgf89VGlMnp8a8232 J8G0Rfg1N90EfUJo0JNXCwMSjdaXnLrnYa5zBfK/NTY9ogvUtuVJV7Gqp2g2S3yP3Et2 efwwgcwM1b2zTDdu9KzNTX1281Mx/BjyGMrHHOw7iGWcvldVbsnc3u8ST07Oc+JBrAh7 ABaj5knC6Sst+p0+8BBTYW5W1u2o34uE50DxFZzCfw8XjP3iYDPwUH0Qlnk1yf0hw1RO vQ== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:12 +0200 Message-Id: <20181113071336.6242-24-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] [PATCH v3 23/23] docs: Update pvrdma device documentation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Interface with the device is changed with the addition of support for MAD packets. Adjust documentation accordingly. While there fix a minor mistake which may lead to think that there is a relation between using RXE on host and the compatibility with bare-metal peers. Signed-off-by: Yuval Shaia --- docs/pvrdma.txt | 103 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 84 insertions(+), 19 deletions(-) diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt index 5599318159..9e8d1674b7 100644 --- a/docs/pvrdma.txt +++ b/docs/pvrdma.txt @@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for an= y special guest modifications. =20 While it complies with the VMware device, it can also communicate with bare -metal RDMA-enabled machines and does not require an RDMA HCA in the host, = it -can work with Soft-RoCE (rxe). +metal RDMA-enabled machines as peers. + +It does not require an RDMA HCA in the host, it can work with Soft-RoCE (r= xe). =20 It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be @@ -78,29 +79,93 @@ the required RDMA libraries. =20 3. Usage =3D=3D=3D=3D=3D=3D=3D=3D + + +3.1 VM Memory settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Currently the device is working only with memory backed RAM and it must be mark as "shared": -m 1G \ -object memory-backend-ram,id=3Dmb1,size=3D1G,share \ -numa node,memdev=3Dmb1 \ =20 -The pvrdma device is composed of two functions: - - Function 0 is a vmxnet Ethernet Device which is redundant in Guest - but is required to pass the ibdevice GID using its MAC. - Examples: - For an rxe backend using eth0 interface it will use its mac: - -device vmxnet3,addr=3D.0,multifunction=3Don,mac=3D - For an SRIOV VF, we take the Ethernet Interface exposed by it: - -device vmxnet3,multifunction=3Don,mac=3D - - Function 1 is the actual device: - -device pvrdma,addr=3D.1,backend-dev=3D,backend-gid= -idx=3D,backend-port=3D - where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) - Note: Pay special attention that the GID at backend-gid-idx matches vmxne= t's MAC. - The rules of conversion are part of the RoCE spec, but since manual conve= rsion - is not required, spotting problems is not hard: - Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a - MAC: 7c:fe:90:cb:74:3a - Note the difference between the first byte of the MAC and the GID. + +3.2 MAD Multiplexer +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +MAD Multiplexer is a service that exposes MAD-like interface for VMs in +order to overcome the limitation where only single entity can register with +MAD layer to send and receive RDMA-CM MAD packets. + +To build rdmacm-mux run +# make rdmacm-mux + +The program accepts 3 command line arguments and exposes a UNIX socket to +be used to relay control and data messages to and from the service. +-s unix-socket-path Path to unix socket to listen on + (default /var/run/rdmacm-mux) +-d rdma-device-name Name of RDMA device to register with + (default rxe0) +-p rdma-device-port Port number of RDMA device to register with + (default 1) +The final UNIX socket file name is a concatenation of the 3 arguments so +for example for device name mlx5_0 and port 2 the file +/var/run/rdmacm-mux-mlx5_0-2 will be created. + +Please refer to contrib/rdmacm-mux for more details. + + +3.3 PCI devices settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +RoCE device exposes two functions - Ethernet and RDMA. +To support it, pvrdma device is composed of two PCI functions, an Ethernet +device of type vmxnet3 on PCI slot 0 and a pvrdma device on PCI slot 1. The +Ethernet function can be used for other Ethernet purposes such as IP. + + +3.4 Device parameters +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +- netdev: Specifies the Ethernet device on host. For Soft-RoCE (rxe) this + would be the Ethernet device used to create it. For any other physical + RoCE device this would be the netdev name of the device. +- ibdev: The IB device name on host for example rxe0, mlx5_0 etc. +- mad-chardev: The name of the MAD multiplexer char device. +- ibport: In case of multi-port device (such as Mellanox's HCA) this + specify the port to use. If not set 1 will be used. +- dev-caps-max-mr-size: The maximum size of MR. +- dev-caps-max-qp: Maximum number of QPs. +- dev-caps-max-sge: Maximum number of SGE elements in WR. +- dev-caps-max-cq: Maximum number of CQs. +- dev-caps-max-mr: Maximum number of MRs. +- dev-caps-max-pd: Maximum number of PDs. +- dev-caps-max-ah: Maximum number of AHs. + +Notes: +- The first 3 parameters are mandatory settings, the rest have their + defaults. +- The last 8 parameters (the ones that prefixed by dev-caps) defines the t= op + limits but the final values are adjusted by the backend device limitatio= ns. + +3.5 Example +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Define bridge device with vmxnet3 network backend: + + + + +
+ + +Define pvrdma device: + + + + + + + + + + =20 =20 =20 --=20 2.17.2 From nobody Sun Apr 28 22:39:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=oracle.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1542095263523314.8021259444647; Mon, 12 Nov 2018 23:47:43 -0800 (PST) Received: from localhost ([::1]:52478 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMTQE-0004yW-6Q for importer@patchew.org; Tue, 13 Nov 2018 02:47:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52570) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gMSwA-0005ae-8f for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gMSw3-0005Ib-2B for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:38 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:45080) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gMSvz-0005CV-4K for qemu-devel@nongnu.org; Tue, 13 Nov 2018 02:16:29 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wAD7EL1N079286; Tue, 13 Nov 2018 07:16:24 GMT Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nnwg18mey-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:23 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wAD7GMEP025572 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Nov 2018 07:16:23 GMT Received: from abhmp0014.oracle.com (abhmp0014.oracle.com [141.146.116.20]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wAD7GM7o018349; Tue, 13 Nov 2018 07:16:22 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 12 Nov 2018 23:16:21 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=Mw/y2lwgfEa7jCpGfWJaFRrwAfCgYeubThGvsG0oSGg=; b=D7i9cBDh8NO06KRm4gkJFnFJhHq0y11vCpihrL6fdbplWDb6bxrUK3gMlJjzV1IAlDa9 45JDqL60a9+wAX3HQuu+097+nPeVG0gzCNK98iktcJucRBGxqvRO436R3mlx9e8m1ETk BT2NBHFHgauZftwJH5YPgMRb4Upa/VTOVIMTWLnqQtWtPt1Ex/P9uD1YVspVSQR1tMRP Y9GFEr69I0/JpMCILWcDpHbQZpUNOTIAGw9RIWbx/NVNrBTsTboEdOdMfZ2BVtK1PM4C eijyjGrl4/8XUS6pSQrac4Byjjr/KD5yVAMT7UjeGGHhcqrtqsXcz8flslsUQKDdkqq4 nw== From: Yuval Shaia To: yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, dmitry.fleytman@gmail.com, jasowang@redhat.com, eblake@redhat.com, armbru@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, shamir.rabinovitch@oracle.com, cohuck@redhat.com Date: Tue, 13 Nov 2018 09:13:36 +0200 Message-Id: <20181113071336.6242-48-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181113071336.6242-1-yuval.shaia@oracle.com> References: <20181113071336.6242-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9075 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v3 23/23] docs: Update pvrdma device documentation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Interface with the device is changed with the addition of support for MAD packets. Adjust documentation accordingly. While there fix a minor mistake which may lead to think that there is a relation between using RXE on host and the compatibility with bare-metal peers. Signed-off-by: Yuval Shaia --- docs/pvrdma.txt | 103 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 84 insertions(+), 19 deletions(-) diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt index 5599318159..9e8d1674b7 100644 --- a/docs/pvrdma.txt +++ b/docs/pvrdma.txt @@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for an= y special guest modifications. =20 While it complies with the VMware device, it can also communicate with bare -metal RDMA-enabled machines and does not require an RDMA HCA in the host, = it -can work with Soft-RoCE (rxe). +metal RDMA-enabled machines as peers. + +It does not require an RDMA HCA in the host, it can work with Soft-RoCE (r= xe). =20 It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be @@ -78,29 +79,93 @@ the required RDMA libraries. =20 3. Usage =3D=3D=3D=3D=3D=3D=3D=3D + + +3.1 VM Memory settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Currently the device is working only with memory backed RAM and it must be mark as "shared": -m 1G \ -object memory-backend-ram,id=3Dmb1,size=3D1G,share \ -numa node,memdev=3Dmb1 \ =20 -The pvrdma device is composed of two functions: - - Function 0 is a vmxnet Ethernet Device which is redundant in Guest - but is required to pass the ibdevice GID using its MAC. - Examples: - For an rxe backend using eth0 interface it will use its mac: - -device vmxnet3,addr=3D.0,multifunction=3Don,mac=3D - For an SRIOV VF, we take the Ethernet Interface exposed by it: - -device vmxnet3,multifunction=3Don,mac=3D - - Function 1 is the actual device: - -device pvrdma,addr=3D.1,backend-dev=3D,backend-gid= -idx=3D,backend-port=3D - where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) - Note: Pay special attention that the GID at backend-gid-idx matches vmxne= t's MAC. - The rules of conversion are part of the RoCE spec, but since manual conve= rsion - is not required, spotting problems is not hard: - Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a - MAC: 7c:fe:90:cb:74:3a - Note the difference between the first byte of the MAC and the GID. + +3.2 MAD Multiplexer +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +MAD Multiplexer is a service that exposes MAD-like interface for VMs in +order to overcome the limitation where only single entity can register with +MAD layer to send and receive RDMA-CM MAD packets. + +To build rdmacm-mux run +# make rdmacm-mux + +The program accepts 3 command line arguments and exposes a UNIX socket to +be used to relay control and data messages to and from the service. +-s unix-socket-path Path to unix socket to listen on + (default /var/run/rdmacm-mux) +-d rdma-device-name Name of RDMA device to register with + (default rxe0) +-p rdma-device-port Port number of RDMA device to register with + (default 1) +The final UNIX socket file name is a concatenation of the 3 arguments so +for example for device name mlx5_0 and port 2 the file +/var/run/rdmacm-mux-mlx5_0-2 will be created. + +Please refer to contrib/rdmacm-mux for more details. + + +3.3 PCI devices settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +RoCE device exposes two functions - Ethernet and RDMA. +To support it, pvrdma device is composed of two PCI functions, an Ethernet +device of type vmxnet3 on PCI slot 0 and a pvrdma device on PCI slot 1. The +Ethernet function can be used for other Ethernet purposes such as IP. + + +3.4 Device parameters +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +- netdev: Specifies the Ethernet device on host. For Soft-RoCE (rxe) this + would be the Ethernet device used to create it. For any other physical + RoCE device this would be the netdev name of the device. +- ibdev: The IB device name on host for example rxe0, mlx5_0 etc. +- mad-chardev: The name of the MAD multiplexer char device. +- ibport: In case of multi-port device (such as Mellanox's HCA) this + specify the port to use. If not set 1 will be used. +- dev-caps-max-mr-size: The maximum size of MR. +- dev-caps-max-qp: Maximum number of QPs. +- dev-caps-max-sge: Maximum number of SGE elements in WR. +- dev-caps-max-cq: Maximum number of CQs. +- dev-caps-max-mr: Maximum number of MRs. +- dev-caps-max-pd: Maximum number of PDs. +- dev-caps-max-ah: Maximum number of AHs. + +Notes: +- The first 3 parameters are mandatory settings, the rest have their + defaults. +- The last 8 parameters (the ones that prefixed by dev-caps) defines the t= op + limits but the final values are adjusted by the backend device limitatio= ns. + +3.5 Example +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Define bridge device with vmxnet3 network backend: + + + + +
+ + +Define pvrdma device: + + + + + + + + + + =20 =20 =20 --=20 2.17.2