From nobody Wed May 8 04:27:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1550485848827809.3731122409608; Mon, 18 Feb 2019 02:30:48 -0800 (PST) Received: from localhost ([127.0.0.1]:55988 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgC7-00017c-Fp for importer@patchew.org; Mon, 18 Feb 2019 05:30:39 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43303) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgAB-0008WS-Cy for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gvgA4-0006Ao-CO for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:37 -0500 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:37480) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gvg9q-00061E-RT for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:23 -0500 Received: by mail-pl1-x643.google.com with SMTP id q3so1215614pll.4 for ; Mon, 18 Feb 2019 02:28:16 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id 33sm9969413pgs.81.2019.02.18.02.28.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Feb 2019 02:28:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NN0WPyqMrX+JT9Tv1ZeuHZKvlukuMvcn+PiOCkzZoiM=; b=NOm4aYHSI0VkvNtu2dKkUk/biSvGqyrBUELrLuvu8VEeW/zYSWTVsULHbm6/XzIx6m IhbKBjEIeO2TZG4BBy+rgW9zRtS9vw6CtjRUAUwBuUFGmU44BbgYHC7+zC2gXTX+EXTp VaKLCh3n/7q+RFA+6JjZg0Q7vougstgQ2BluN4ftn1VIqGL1nlE62ki1+y5qpoXtwXtg 9gmAT1tbv2w4F1TZmfGNZaTmHfEHIKXJCpJn8QS2b59prM+BbuG9WlJC9RzHOljkuYq1 N01n2UoA2dMqIzSH5l9lphEb7kIST5ax6LiapT/VuOLkCp7qPoon+nsCyx3Eh6ySnv2D kStQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NN0WPyqMrX+JT9Tv1ZeuHZKvlukuMvcn+PiOCkzZoiM=; b=o3UpL8AKrlakY04v05/TK9KaQN/TrvvOq+zPKYb3tIVbEky6cpIec2d+VLx8kiRUeW M7TRmu0fQz22ZvmvwERo2wtFfskSRhjhI5UbGYpSa2uAWv+9Sr2oFCSpVIOY/+t+Bvc+ nntsfwTFYBlTYCkGJoHjo6R0fuwCimPzuZC8faIzss8gcKOC4DqQtN4otpDaGK4DI6go 82j1qA4XY+nULFDlChb/RpDl91jwYrxN3k5E8YHpqPiibQ96e3yPxO21OAGgQ+7V4chi hL8IB8IyTg4rGD/yw2MjGMPH08I03bwp4xh6U/KDb/UUFgN2Wtr6VifX3FKcY+8jWhWP jklw== X-Gm-Message-State: AHQUAuZeN5sAyTpp3oRkkTTTuGPM3MDp1f2ViK4mgmK8/2fsIG+oADjz PQ13g6hlOYync3reEGPlzaQ= X-Google-Smtp-Source: AHgI3IY3zX0mAmfupedlmPoVxbiuHyrSyAFiLIZFml6eMR+9K3rVqPB5perw4nPZIvDtBAOMvIaDzA== X-Received: by 2002:a17:902:a40d:: with SMTP id p13mr14788098plq.144.1550485695617; Mon, 18 Feb 2019 02:28:15 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, stefanha@gmail.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Mon, 18 Feb 2019 18:27:42 +0800 Message-Id: <20190218102748.2242-2-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190218102748.2242-1-xieyongji@baidu.com> References: <20190218102748.2242-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 Subject: [Qemu-devel] [PATCH v6 1/7] vhost-user: Support transferring inflight buffer between qemu and backend X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xie Yongji This patch introduces two new messages VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD to support transferring a shared buffer between qemu and backend. Firstly, qemu uses VHOST_USER_GET_INFLIGHT_FD to get the shared buffer from backend. Then qemu should send it back through VHOST_USER_SET_INFLIGHT_FD each time we start vhost-user. This shared buffer is used to track inflight I/O by backend. Qemu should retrieve a new one when vm reset. Signed-off-by: Xie Yongji Signed-off-by: Chai Wen Signed-off-by: Zhang Yu --- docs/interop/vhost-user.txt | 264 ++++++++++++++++++++++++++++++ hw/virtio/vhost-user.c | 107 ++++++++++++ hw/virtio/vhost.c | 96 +++++++++++ include/hw/virtio/vhost-backend.h | 10 ++ include/hw/virtio/vhost.h | 18 ++ 5 files changed, 495 insertions(+) diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index c2194711d9..61c6d0e415 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -142,6 +142,17 @@ Depending on the request type, payload can be: Offset: a 64-bit offset of this area from the start of the supplied file descriptor =20 + * Inflight description + ----------------------------------------------------- + | mmap size | mmap offset | num queues | queue size | + ----------------------------------------------------- + + mmap size: a 64-bit size of area to track inflight I/O + mmap offset: a 64-bit offset of this area from the start + of the supplied file descriptor + num queues: a 16-bit number of virtqueues + queue size: a 16-bit size of virtqueues + In QEMU the vhost-user message is implemented with the following struct: =20 typedef struct VhostUserMsg { @@ -157,6 +168,7 @@ typedef struct VhostUserMsg { struct vhost_iotlb_msg iotlb; VhostUserConfig config; VhostUserVringArea area; + VhostUserInflight inflight; }; } QEMU_PACKED VhostUserMsg; =20 @@ -175,6 +187,7 @@ the ones that do: * VHOST_USER_GET_PROTOCOL_FEATURES * VHOST_USER_GET_VRING_BASE * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD) + * VHOST_USER_GET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD) =20 [ Also see the section on REPLY_ACK protocol extension. ] =20 @@ -188,6 +201,7 @@ in the ancillary data: * VHOST_USER_SET_VRING_CALL * VHOST_USER_SET_VRING_ERR * VHOST_USER_SET_SLAVE_REQ_FD + * VHOST_USER_SET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD) =20 If Master is unable to send the full message or receives a wrong reply it = will close the connection. An optional reconnection mechanism can be implemente= d. @@ -382,6 +396,235 @@ If VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD protocol featu= re is negotiated, slave can send file descriptors (at most 8 descriptors in each message) to master via ancillary data using this fd communication channel. =20 +Inflight I/O tracking +--------------------- + +To support reconnecting after restart or crash, slave may need to resubmit +inflight I/Os. If virtqueue is processed in order, we can easily achieve +that by getting the inflight descriptors from descriptor table (split virt= queue) +or descriptor ring (packed virtqueue). However, it can't work when we proc= ess +descriptors out-of-order because some entries which store the information = of +inflight descriptors in available ring (split virtqueue) or descriptor +ring (packed virtqueue) might be overrided by new entries. To solve this +problem, slave need to allocate an extra buffer to store this information = of inflight +descriptors and share it with master for persistent. VHOST_USER_GET_INFLIG= HT_FD and +VHOST_USER_SET_INFLIGHT_FD are used to transfer this buffer between master +and slave. And the format of this buffer is described below: + +------------------------------------------------------- +| queue0 region | queue1 region | ... | queueN region | +------------------------------------------------------- + +N is the number of available virtqueues. Slave could get it from num queues +field of VhostUserInflight. + +For split virtqueue, queue region can be implemented as: + +typedef struct DescStateSplit { + /* Indicate whether this descriptor is inflight or not. + * Only available for head-descriptor. */ + uint8_t inflight; + + /* Padding */ + uint8_t padding; + + /* Link to the last processed entry */ + uint16_t next; +} DescStateSplit; + +typedef struct QueueRegionSplit { + /* The feature flags of this region. Now it's initialized to 0. */ + uint64_t features; + + /* The version of this region. It's 1 currently. + * Zero value indicates an uninitialized buffer */ + uint16_t version; + + /* The size of DescStateSplit array. It's equal to the virtqueue + * size. Slave could get it from queue size field of VhostUserInflight= . */ + uint16_t desc_num; + + /* The head of processed DescStateSplit entry list */ + uint16_t process_head; + + /* Storing the idx value of used ring */ + uint16_t used_idx; + + /* Used to track the state of each descriptor in descriptor table */ + DescStateSplit desc[0]; +} QueueRegionSplit; + +To track inflight I/O, the queue region should be processed as follows: + +When receiving available buffers from the driver: + + 1. Get the next available head-descriptor index from available ring, i + + 2. Set desc[i].inflight to 1 + +When supplying used buffers to the driver: + + 1. Get corresponding used head-descriptor index, i + + 2. Set desc[i].next to process_head + + 3. Set process_head to i + + 4. Steps 1,2,3 may be performed repeatedly if batching is possible + + 5. Increase the idx value of used ring by the size of the batch + + 6. Set the inflight field of each DescStateSplit entry in the batch to= 0 + + 7. Set used_idx to the idx value of used ring + +When reconnecting: + + 1. If the value of used_idx does not match the idx value of used ring, + + (a) Subtract the value of used_idx from the idx value of used ring= to get + the number of in-progress DescStateSplit entries + + (b) Set the inflight field of the in-progress DescStateSplit entri= es which + start from process_head to 0 + + (c) Set used_idx to the idx value of used ring + + 2. Resubmit each inflight DescStateSplit entry + +For packed virtqueue, queue region can be implemented as: + +typedef struct DescStatePacked { + /* Indicate whether this descriptor is inflight or not. + * Only available for head-descriptor. */ + uint8_t inflight; + + /* Padding */ + uint8_t padding; + + /* Link to the next free entry */ + uint16_t next; + + /* Link to the last entry of descriptor list. + * Only available for head-descriptor. */ + uint16_t last; + + /* The length of descriptor list. + * Only available for head-descriptor. */ + uint16_t num; + + /* The buffer id */ + uint16_t id; + + /* The descriptor flags */ + uint16_t flags; + + /* The buffer length */ + uint32_t len; + + /* The buffer address */ + uint64_t addr; +} DescStatePacked; + +typedef struct QueueRegionPacked { + /* The feature flags of this region. Now it's initialized to 0. */ + uint64_t features; + + /* The version of this region. It's 1 currently. + * Zero value indicates an uninitialized buffer */ + uint16_t version; + + /* The size of DescStatePacked array. It's equal to the virtqueue + * size. Slave could get it from queue size field of VhostUserInflight= . */ + uint16_t desc_num; + + /* The head of free DescStatePacked entry list */ + uint16_t free_head; + + /* The old head of free DescStatePacked entry list */ + uint16_t old_free_head; + + /* The used index of descriptor ring */ + uint16_t used_idx; + + /* The old used index of descriptor ring */ + uint16_t old_used_idx; + + /* Device ring wrap counter */ + uint8_t used_wrap_counter; + + /* The old device ring wrap counter */ + uint8_t old_used_wrap_counter; + + /* Padding */ + uint8_t padding[7]; + + /* Used to track the state of each descriptor fetched from descriptor = ring */ + DescStatePacked desc[0]; +} QueueRegionPacked; + +To track inflight I/O, the queue region should be processed as follows: + +When receiving available buffers from the driver: + + 1. Get the next available descriptor entry from descriptor ring, d + + 2. If d is head descriptor, + + (a) Set desc[old_free_head].num to 0 + + (b) Set desc[old_free_head].inflight to 1 + + 3. If d is last descriptor, set desc[old_free_head].last to free_head + + 4. Increase desc[old_free_head].num by 1 + + 5. Set desc[free_head].addr, desc[free_head].len, desc[free_head].flag= s, + desc[free_head].id to d.addr, d.len, d.flags, d.id + + 6. Set free_head to desc[free_head].next + + 7. If d is last descriptor, set old_free_head to free_head + +When supplying used buffers to the driver: + + 1. Get corresponding used head-descriptor entry from descriptor ring, d + + 2. Get corresponding DescStatePacked entry, e + + 3. Set desc[e.last].next to free_head + + 4. Set free_head to the index of e + + 5. Steps 1,2,3,4 may be performed repeatedly if batching is possible + + 6. Increase used_idx by the size of the batch and update used_wrap_cou= nter if needed + + 7. Update d.flags + + 8. Set the inflight field of each head DescStatePacked entry in the ba= tch to 0 + + 9. Set old_free_head, old_used_idx, old_used_wrap_counter to free_head= , used_idx, + used_wrap_counter + +When reconnecting: + + 1. If used_idx does not match old_used_idx, + + (a) Get the next descriptor ring entry through old_used_idx, d + + (b) Use old_used_wrap_counter to calculate the available flags + + (c) If d.flags is not equal to the calculated flags value, set old= _free_head, + old_used_idx, old_used_wrap_counter to free_head, used_idx, used_w= rap_counter + + 2. Set free_head, used_idx, used_wrap_counter to old_free_head, old_us= ed_idx, + old_used_wrap_counter + + 3. Set the inflight field of each free DescStatePacked entry to 0 + + 4. Resubmit each inflight DescStatePacked entry + Protocol features ----------------- =20 @@ -397,6 +640,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_CONFIG 9 #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10 #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11 +#define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12 =20 Master message types -------------------- @@ -761,6 +1005,26 @@ Master message types was previously sent. The value returned is an error indication; 0 is success. =20 + * VHOST_USER_GET_INFLIGHT_FD + Id: 31 + Equivalent ioctl: N/A + Master payload: inflight description + + When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been + successfully negotiated, this message is submitted by master to get + a shared buffer from slave. The shared buffer will be used to track + inflight I/O by slave. QEMU should retrieve a new one when vm reset. + + * VHOST_USER_SET_INFLIGHT_FD + Id: 32 + Equivalent ioctl: N/A + Master payload: inflight description + + When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been + successfully negotiated, this message is submitted by master to send + the shared inflight buffer back to slave so that slave could get + inflight I/O after a crash or restart. + Slave message types ------------------- =20 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 564a31d12c..21a81998ba 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -52,6 +52,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_CONFIG =3D 9, VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD =3D 10, VHOST_USER_PROTOCOL_F_HOST_NOTIFIER =3D 11, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD =3D 12, VHOST_USER_PROTOCOL_F_MAX }; =20 @@ -89,6 +90,8 @@ typedef enum VhostUserRequest { VHOST_USER_POSTCOPY_ADVISE =3D 28, VHOST_USER_POSTCOPY_LISTEN =3D 29, VHOST_USER_POSTCOPY_END =3D 30, + VHOST_USER_GET_INFLIGHT_FD =3D 31, + VHOST_USER_SET_INFLIGHT_FD =3D 32, VHOST_USER_MAX } VhostUserRequest; =20 @@ -147,6 +150,13 @@ typedef struct VhostUserVringArea { uint64_t offset; } VhostUserVringArea; =20 +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint16_t num_queues; + uint16_t queue_size; +} VhostUserInflight; + typedef struct { VhostUserRequest request; =20 @@ -169,6 +179,7 @@ typedef union { VhostUserConfig config; VhostUserCryptoSession session; VhostUserVringArea area; + VhostUserInflight inflight; } VhostUserPayload; =20 typedef struct VhostUserMsg { @@ -1739,6 +1750,100 @@ static bool vhost_user_mem_section_filter(struct vh= ost_dev *dev, return result; } =20 +static int vhost_user_get_inflight_fd(struct vhost_dev *dev, + uint16_t queue_size, + struct vhost_inflight *inflight) +{ + void *addr; + int fd; + struct vhost_user *u =3D dev->opaque; + CharBackend *chr =3D u->user->chr; + VhostUserMsg msg =3D { + .hdr.request =3D VHOST_USER_GET_INFLIGHT_FD, + .hdr.flags =3D VHOST_USER_VERSION, + .payload.inflight.num_queues =3D dev->nvqs, + .payload.inflight.queue_size =3D queue_size, + .hdr.size =3D sizeof(msg.payload.inflight), + }; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + return -1; + } + + if (vhost_user_read(dev, &msg) < 0) { + return -1; + } + + if (msg.hdr.request !=3D VHOST_USER_GET_INFLIGHT_FD) { + error_report("Received unexpected msg type. " + "Expected %d received %d", + VHOST_USER_GET_INFLIGHT_FD, msg.hdr.request); + return -1; + } + + if (msg.hdr.size !=3D sizeof(msg.payload.inflight)) { + error_report("Received bad msg size."); + return -1; + } + + if (!msg.payload.inflight.mmap_size) { + return 0; + } + + fd =3D qemu_chr_fe_get_msgfd(chr); + if (fd < 0) { + error_report("Failed to get mem fd"); + return -1; + } + + addr =3D mmap(0, msg.payload.inflight.mmap_size, PROT_READ | PROT_WRIT= E, + MAP_SHARED, fd, msg.payload.inflight.mmap_offset); + + if (addr =3D=3D MAP_FAILED) { + error_report("Failed to mmap mem fd"); + close(fd); + return -1; + } + + inflight->addr =3D addr; + inflight->fd =3D fd; + inflight->size =3D msg.payload.inflight.mmap_size; + inflight->offset =3D msg.payload.inflight.mmap_offset; + inflight->queue_size =3D queue_size; + + return 0; +} + +static int vhost_user_set_inflight_fd(struct vhost_dev *dev, + struct vhost_inflight *inflight) +{ + VhostUserMsg msg =3D { + .hdr.request =3D VHOST_USER_SET_INFLIGHT_FD, + .hdr.flags =3D VHOST_USER_VERSION, + .payload.inflight.mmap_size =3D inflight->size, + .payload.inflight.mmap_offset =3D inflight->offset, + .payload.inflight.num_queues =3D dev->nvqs, + .payload.inflight.queue_size =3D inflight->queue_size, + .hdr.size =3D sizeof(msg.payload.inflight), + }; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (vhost_user_write(dev, &msg, &inflight->fd, 1) < 0) { + return -1; + } + + return 0; +} + VhostUserState *vhost_user_init(void) { VhostUserState *user =3D g_new0(struct VhostUserState, 1); @@ -1790,4 +1895,6 @@ const VhostOps user_ops =3D { .vhost_crypto_create_session =3D vhost_user_crypto_create_session, .vhost_crypto_close_session =3D vhost_user_crypto_close_session, .vhost_backend_mem_section_filter =3D vhost_user_mem_section_filte= r, + .vhost_get_inflight_fd =3D vhost_user_get_inflight_fd, + .vhost_set_inflight_fd =3D vhost_user_set_inflight_fd, }; diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index 569c4053ea..8db1a855eb 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1481,6 +1481,102 @@ void vhost_dev_set_config_notifier(struct vhost_dev= *hdev, hdev->config_ops =3D ops; } =20 +void vhost_dev_free_inflight(struct vhost_inflight *inflight) +{ + if (inflight->addr) { + qemu_memfd_free(inflight->addr, inflight->size, inflight->fd); + inflight->addr =3D NULL; + inflight->fd =3D -1; + } +} + +static int vhost_dev_resize_inflight(struct vhost_inflight *inflight, + uint64_t new_size) +{ + Error *err =3D NULL; + int fd =3D -1; + void *addr =3D qemu_memfd_alloc("vhost-inflight", new_size, + F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEA= L, + &fd, &err); + + if (err) { + error_report_err(err); + return -1; + } + + vhost_dev_free_inflight(inflight); + inflight->offset =3D 0; + inflight->addr =3D addr; + inflight->fd =3D fd; + inflight->size =3D new_size; + + return 0; +} + +void vhost_dev_save_inflight(struct vhost_inflight *inflight, QEMUFile *f) +{ + if (inflight->addr) { + qemu_put_be64(f, inflight->size); + qemu_put_be16(f, inflight->queue_size); + qemu_put_buffer(f, inflight->addr, inflight->size); + } else { + qemu_put_be64(f, 0); + } +} + +int vhost_dev_load_inflight(struct vhost_inflight *inflight, QEMUFile *f) +{ + uint64_t size; + + size =3D qemu_get_be64(f); + if (!size) { + return 0; + } + + if (inflight->size !=3D size) { + if (vhost_dev_resize_inflight(inflight, size)) { + return -1; + } + } + inflight->queue_size =3D qemu_get_be16(f); + + qemu_get_buffer(f, inflight->addr, size); + + return 0; +} + +int vhost_dev_set_inflight(struct vhost_dev *dev, + struct vhost_inflight *inflight) +{ + int r; + + if (dev->vhost_ops->vhost_set_inflight_fd && inflight->addr) { + r =3D dev->vhost_ops->vhost_set_inflight_fd(dev, inflight); + if (r) { + VHOST_OPS_DEBUG("vhost_set_inflight_fd failed"); + return -errno; + } + } + + return 0; +} + +int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size, + struct vhost_inflight *inflight) +{ + int r; + + if (dev->vhost_ops->vhost_get_inflight_fd) { + r =3D dev->vhost_ops->vhost_get_inflight_fd(dev, queue_size, infli= ght); + if (r) { + VHOST_OPS_DEBUG("vhost_get_inflight_fd failed"); + return -errno; + } + } + + return 0; +} + /* Host notifiers must be enabled at this point. */ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev) { diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-ba= ckend.h index 81283ec50f..d6632a18e6 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -25,6 +25,7 @@ typedef enum VhostSetConfigType { VHOST_SET_CONFIG_TYPE_MIGRATION =3D 1, } VhostSetConfigType; =20 +struct vhost_inflight; struct vhost_dev; struct vhost_log; struct vhost_memory; @@ -104,6 +105,13 @@ typedef int (*vhost_crypto_close_session_op)(struct vh= ost_dev *dev, typedef bool (*vhost_backend_mem_section_filter_op)(struct vhost_dev *dev, MemoryRegionSection *secti= on); =20 +typedef int (*vhost_get_inflight_fd_op)(struct vhost_dev *dev, + uint16_t queue_size, + struct vhost_inflight *inflight); + +typedef int (*vhost_set_inflight_fd_op)(struct vhost_dev *dev, + struct vhost_inflight *inflight); + typedef struct VhostOps { VhostBackendType backend_type; vhost_backend_init vhost_backend_init; @@ -142,6 +150,8 @@ typedef struct VhostOps { vhost_crypto_create_session_op vhost_crypto_create_session; vhost_crypto_close_session_op vhost_crypto_close_session; vhost_backend_mem_section_filter_op vhost_backend_mem_section_filter; + vhost_get_inflight_fd_op vhost_get_inflight_fd; + vhost_set_inflight_fd_op vhost_set_inflight_fd; } VhostOps; =20 extern const VhostOps user_ops; diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h index a7f449fa87..619498c8f4 100644 --- a/include/hw/virtio/vhost.h +++ b/include/hw/virtio/vhost.h @@ -7,6 +7,15 @@ #include "exec/memory.h" =20 /* Generic structures common for any vhost based device. */ + +struct vhost_inflight { + int fd; + void *addr; + uint64_t size; + uint64_t offset; + uint16_t queue_size; +}; + struct vhost_virtqueue { int kick; int call; @@ -120,4 +129,13 @@ int vhost_dev_set_config(struct vhost_dev *dev, const = uint8_t *data, */ void vhost_dev_set_config_notifier(struct vhost_dev *dev, const VhostDevConfigOps *ops); + +void vhost_dev_reset_inflight(struct vhost_inflight *inflight); +void vhost_dev_free_inflight(struct vhost_inflight *inflight); +void vhost_dev_save_inflight(struct vhost_inflight *inflight, QEMUFile *f); +int vhost_dev_load_inflight(struct vhost_inflight *inflight, QEMUFile *f); +int vhost_dev_set_inflight(struct vhost_dev *dev, + struct vhost_inflight *inflight); +int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size, + struct vhost_inflight *inflight); #endif --=20 2.17.1 From nobody Wed May 8 04:27:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1550486277408566.2109450545071; Mon, 18 Feb 2019 02:37:57 -0800 (PST) Received: from localhost ([127.0.0.1]:56120 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgJ7-0006Rv-Fx for importer@patchew.org; Mon, 18 Feb 2019 05:37:53 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43475) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgAd-0000NC-LP for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gvgAb-0006Vr-Nl for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:07 -0500 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:38214) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gvgAZ-000635-Q3 for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:05 -0500 Received: by mail-pl1-x643.google.com with SMTP id e5so8589403plb.5 for ; Mon, 18 Feb 2019 02:28:19 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id i74sm25764291pfi.33.2019.02.18.02.28.18 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Feb 2019 02:28:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fiHcsz3JHZWGauGOQSLwZCLApSBpe9h6n5kwPgJaYQs=; b=fLm/BH/x9EPjut9dCwJW/vRxrX0oTbjmSZMDcQDuCqP38VxAvdE3YpShwUUr7cGaBt /q64v51VUqV0Xz1FTwoasdC2WwaWHkkCZKBaNskNk4hYAmhbVuAD/0etPkYpgKbyEJR5 XHx0rhot428S0C5trIB3YDDNKLBWcHPB0SeyPnKwRPaI2u20xniwpCzNWWQWlVVEg7IT yCnsSo0rJ/PpezViuvrbXK3r+ZXo5Wz6kQmItEEfiHyruOYu1JvQMdZY9ZmZSruJJ+u1 0fGJHh7nNHRCaH1J9Wx3sfvapMdJm0tcn7/xbPWIcSV8qGvj5qaspSCKOZ7nisXp3RS2 XSuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=fiHcsz3JHZWGauGOQSLwZCLApSBpe9h6n5kwPgJaYQs=; b=b36zlHz3b2BLkiswjeSZke51aFKNvFw0icmsn9l1gnvO1XyhlMbueWS98aXzfxOZ3P 5y7J2U8h3n9+dCkltCWgXh4owHQnje0+DKnpgFgmTjOzDcaFODfpvkEYo5S1DOBciaB0 gkoy/r8B09g9rvKobJ5ZX1wiY7FGkWNUeA3W5ro7bqjo8iJKiaQ0EGQxBH8fuzE8IiXn fBueVVmCa9eWZBkMVsOYFXv71rld85XN/GJULCom5vBpalHt7naSDJPqtyYrYLI11Cr7 fZI8voxZ+Z6mYCb7JPQRfdnV0gYD6LsWt8Z/g3Fm+/iFJHDFtSOY5+jD6gKw2PO4Q9dZ uj6Q== X-Gm-Message-State: AHQUAubgCi0BcYumjNohlZhPjkLUZbjchHar8qPW/BML1U8Gaontt/uU +a4qf+C1S3NlpBn+plq3fX0= X-Google-Smtp-Source: AHgI3IalkO2zr1G0wLDaWvcmkZIscQDxZsbEQme8UnjXJek4gTp65MTX49w7aBfxSaTFHj/7VQX7MA== X-Received: by 2002:a17:902:8b82:: with SMTP id ay2mr24325656plb.64.1550485699023; Mon, 18 Feb 2019 02:28:19 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, stefanha@gmail.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Mon, 18 Feb 2019 18:27:43 +0800 Message-Id: <20190218102748.2242-3-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190218102748.2242-1-xieyongji@baidu.com> References: <20190218102748.2242-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 Subject: [Qemu-devel] [PATCH v6 2/7] libvhost-user: Remove unnecessary FD flag check for event file descriptors X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xie Yongji The vu_check_queue_msg_file() has checked the FD flag. So let's delete the redundant check after it. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- contrib/libvhost-user/libvhost-user.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 3f14b4138b..16fec3a3fd 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -907,10 +907,8 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) dev->vq[index].kick_fd =3D -1; } =20 - if (!(vmsg->payload.u64 & VHOST_USER_VRING_NOFD_MASK)) { - dev->vq[index].kick_fd =3D vmsg->fds[0]; - DPRINT("Got kick_fd: %d for vq: %d\n", vmsg->fds[0], index); - } + dev->vq[index].kick_fd =3D vmsg->fds[0]; + DPRINT("Got kick_fd: %d for vq: %d\n", vmsg->fds[0], index); =20 dev->vq[index].started =3D true; if (dev->iface->queue_set_started) { @@ -995,9 +993,7 @@ vu_set_vring_call_exec(VuDev *dev, VhostUserMsg *vmsg) dev->vq[index].call_fd =3D -1; } =20 - if (!(vmsg->payload.u64 & VHOST_USER_VRING_NOFD_MASK)) { - dev->vq[index].call_fd =3D vmsg->fds[0]; - } + dev->vq[index].call_fd =3D vmsg->fds[0]; =20 DPRINT("Got call_fd: %d for vq: %d\n", vmsg->fds[0], index); =20 @@ -1020,9 +1016,7 @@ vu_set_vring_err_exec(VuDev *dev, VhostUserMsg *vmsg) dev->vq[index].err_fd =3D -1; } =20 - if (!(vmsg->payload.u64 & VHOST_USER_VRING_NOFD_MASK)) { - dev->vq[index].err_fd =3D vmsg->fds[0]; - } + dev->vq[index].err_fd =3D vmsg->fds[0]; =20 return false; } --=20 2.17.1 From nobody Wed May 8 04:27:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1550486045633614.3115695144107; Mon, 18 Feb 2019 02:34:05 -0800 (PST) Received: from localhost ([127.0.0.1]:56050 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgFN-0003l7-HL for importer@patchew.org; Mon, 18 Feb 2019 05:34:01 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43540) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgAf-0000Ow-Bt for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gvgAd-0006Xr-Vt for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:09 -0500 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:33529) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gvgAd-000653-LO for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:07 -0500 Received: by mail-pl1-x644.google.com with SMTP id y10so8609856plp.0 for ; Mon, 18 Feb 2019 02:28:23 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id e6sm23502582pfe.59.2019.02.18.02.28.21 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Feb 2019 02:28:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Re0HyIiq6TcSY2HsM8WeDdKG2mWgNhg6qEZuo5AbfkM=; b=mdgQLrLiZYDbaWReiBuXsSmp1kw5pPUZWC0bioeRbDNGCYAVXWy5t0JAgY1UXR9SRJ TBBqp5NkyrKuDCHNr22WJVSsIwXJLn/k7JAN0uFcoX1PEixHKtkr8EEz3zn8o2OP/76+ w1OjJW5KkxLZP9CLlwX9/Sdvt+/8O3JrJfTIGyKfvGMM7qb87Zgle/0OD/tdrBEgT16v DwgXZfnIY/Mp+n8t97Z0plK/jpypRQgJCrsyFLAABBORX9Lta4RqSXjjy3A0aBVultSo OCvTHnYPQOOo6voYynAms/k1sL6tZNTR/lJAqk6nDj+ufDXtRT4XtjzvqhBGfklVojOp Guwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Re0HyIiq6TcSY2HsM8WeDdKG2mWgNhg6qEZuo5AbfkM=; b=OqLncUb2rhVx6nNFPkm9uuiIXkdVBLLzy/tcSQd4aqSatRG67K1WpDYEUBndHyO84Z eu2+fH3pfnWS80TEwp7tMdQPgAmB8bNOdOmIOCOqtvCDbnoxt/18mfdo4Y6VPW5/9YUc L8F26Tw5x6WdwC1ljdhRE8HrfErpfdmI4SANc0oJwpNXVkSXCqWdus0iuUahvBgG/J6J EXwMmb5uLU2TpvR2ckGcsi/ABBEO+FeW4uX2UDA+3pews7WpHkZLYjBJuydX00Gk5PCr WGDWHpNHjwmYMoSXFooRpb+Jumpve8XkodMsGSzuXGyyVd3pgSVihWzB6A/3rlgi7331 D6rg== X-Gm-Message-State: AHQUAuay3JNytymJEaWndC/tvBs9DgxIwNXb8AeBW4rNsY1uhg6m+Z3b oaTypkSMpU8f8ZEUNvOQGPxvdPUm X-Google-Smtp-Source: AHgI3Ia5uhW+qcJ2YsTTPWjBXdugipI+c3yVJIlr6NoCS0EorjxvJD/qtvDxEdO5m26PFF8SAnc+WA== X-Received: by 2002:a17:902:33c1:: with SMTP id b59mr24397574plc.220.1550485702426; Mon, 18 Feb 2019 02:28:22 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, stefanha@gmail.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Mon, 18 Feb 2019 18:27:44 +0800 Message-Id: <20190218102748.2242-4-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190218102748.2242-1-xieyongji@baidu.com> References: <20190218102748.2242-1-xieyongji@baidu.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 Subject: [Qemu-devel] [PATCH v6 3/7] libvhost-user: Introduce vu_queue_map_desc() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: Xie Yongji Introduce vu_queue_map_desc() which should be independent with vu_queue_pop(); Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 88 ++++++++++++++++----------- 1 file changed, 51 insertions(+), 37 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 16fec3a3fd..ea0f414b6d 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -1847,49 +1847,20 @@ virtqueue_alloc_element(size_t sz, return elem; } =20 -void * -vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) +static void * +vu_queue_map_desc(VuDev *dev, VuVirtq *vq, unsigned int idx, size_t sz) { - unsigned int i, head, max, desc_len; + struct vring_desc *desc =3D vq->vring.desc; uint64_t desc_addr, read_len; + unsigned int desc_len; + unsigned int max =3D vq->vring.num; + unsigned int i =3D idx; VuVirtqElement *elem; - unsigned out_num, in_num; + unsigned int out_num =3D 0, in_num =3D 0; struct iovec iov[VIRTQUEUE_MAX_SIZE]; struct vring_desc desc_buf[VIRTQUEUE_MAX_SIZE]; - struct vring_desc *desc; int rc; =20 - if (unlikely(dev->broken) || - unlikely(!vq->vring.avail)) { - return NULL; - } - - if (vu_queue_empty(dev, vq)) { - return NULL; - } - /* Needed after virtio_queue_empty(), see comment in - * virtqueue_num_heads(). */ - smp_rmb(); - - /* When we start there are none of either input nor output. */ - out_num =3D in_num =3D 0; - - max =3D vq->vring.num; - if (vq->inuse >=3D vq->vring.num) { - vu_panic(dev, "Virtqueue size exceeded"); - return NULL; - } - - if (!virtqueue_get_head(dev, vq, vq->last_avail_idx++, &head)) { - return NULL; - } - - if (vu_has_feature(dev, VIRTIO_RING_F_EVENT_IDX)) { - vring_set_avail_event(vq, vq->last_avail_idx); - } - - i =3D head; - desc =3D vq->vring.desc; if (desc[i].flags & VRING_DESC_F_INDIRECT) { if (desc[i].len % sizeof(struct vring_desc)) { vu_panic(dev, "Invalid size for indirect buffer table"); @@ -1941,12 +1912,13 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) } while (rc =3D=3D VIRTQUEUE_READ_DESC_MORE); =20 if (rc =3D=3D VIRTQUEUE_READ_DESC_ERROR) { + vu_panic(dev, "read descriptor error"); return NULL; } =20 /* Now copy what we have collected and mapped */ elem =3D virtqueue_alloc_element(sz, out_num, in_num); - elem->index =3D head; + elem->index =3D idx; for (i =3D 0; i < out_num; i++) { elem->out_sg[i] =3D iov[i]; } @@ -1954,6 +1926,48 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) elem->in_sg[i] =3D iov[out_num + i]; } =20 + return elem; +} + +void * +vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) +{ + unsigned int head; + VuVirtqElement *elem; + + if (unlikely(dev->broken) || + unlikely(!vq->vring.avail)) { + return NULL; + } + + if (vu_queue_empty(dev, vq)) { + return NULL; + } + /* + * Needed after virtio_queue_empty(), see comment in + * virtqueue_num_heads(). + */ + smp_rmb(); + + if (vq->inuse >=3D vq->vring.num) { + vu_panic(dev, "Virtqueue size exceeded"); + return NULL; + } + + if (!virtqueue_get_head(dev, vq, vq->last_avail_idx++, &head)) { + return NULL; + } + + if (vu_has_feature(dev, VIRTIO_RING_F_EVENT_IDX)) { + vring_set_avail_event(vq, vq->last_avail_idx); + } + + elem =3D vu_queue_map_desc(dev, vq, head, sz); + + if (!elem) { + return NULL; + } + vq->inuse++; =20 return elem; --=20 2.17.1 From nobody Wed May 8 04:27:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1550486150010497.07773853828576; Mon, 18 Feb 2019 02:35:50 -0800 (PST) Received: from localhost ([127.0.0.1]:56096 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgH4-00054J-JQ for importer@patchew.org; Mon, 18 Feb 2019 05:35:46 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43344) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgAM-0000AP-5f for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gvgAI-0006Ik-Dl for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:48 -0500 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]:39302) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gvgAF-00067s-23 for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:44 -0500 Received: by mail-pf1-x443.google.com with SMTP id f132so8346486pfa.6 for ; Mon, 18 Feb 2019 02:28:26 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id y133sm16720841pfb.107.2019.02.18.02.28.24 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Feb 2019 02:28:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=L+qRV69voumnYZYH52MXTK68aVZZdLWEpmD431v+uaQ=; b=EMkLqOzwq0Mab6+bMuVLc0s2hbSutuCnlTxQM2Wo/RrAWlHmkkky7StpCiSdWgS3bz MDtkpBT4oV/MUDVGg/xu3aBmuFbX2xPRzjjEBg9jYbfOlydm2HWfYZY8BGmwEXKPX/aw ocbY3S+g0Dgywcl2iVZPq9xZYH3TGqUzQVb8tdeJru0LRYiWDAdipzn8P0/oZzk4R5cd Kor9IrDLHufVHvI4BxXjsG1XtyEELotNJnbMlHgzC2dkycyFvBYrXWqogkmh9BDQQxIi XYVdL04XL92inbQA/FvDxhX6r186nochEUGcu3HFQql7qNAJWJJ6bwVIb2P+Gmt5WkOf tgiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=L+qRV69voumnYZYH52MXTK68aVZZdLWEpmD431v+uaQ=; b=H+o63O5iyPRgrgPzrQ+HrYStqq951OoK286sVWLoUvCPcsTGdB/ydnQWyR1HkTw1cR ObbPaQOzQcFbeaLp9CxjXaXvDHYhQ9CS8hkYm2McATuzMNb9JII8FDqJSVsLsqtap2Im PbjOrFHE16CGdiu/ATXkSCLFeaF5hIJs/mWbfsuxKsji80sepSmlb9kMULQbYW4mYloM 5fuf4q4P18v3onaWbqkyI7whK5EjbPYnMf8Zrwu3u/F6+PCkctDLdsh6XryOkc+L8f5v vRHNqpjbV8NZ4acZPx78daMmc549JDI4ZSlwy1ad4wFSrxI+ROA0iNKCJINGHpmpGBv+ xY2Q== X-Gm-Message-State: AHQUAuaJ1YCDz/MMiZpyChLxg8bnnpCQdxHhPPFYkW2eakyGFJpihAce a95F0aAjXGMEVw8AYdKFtnA= X-Google-Smtp-Source: AHgI3IYWdTgl2opKEV54L+E7I86ZRg0EYIqPK8RB4BfLVnlT304k92sZaStZBvisUjFFlOvJUJMBBg== X-Received: by 2002:a62:e082:: with SMTP id d2mr24192061pfm.240.1550485705838; Mon, 18 Feb 2019 02:28:25 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, stefanha@gmail.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Mon, 18 Feb 2019 18:27:45 +0800 Message-Id: <20190218102748.2242-5-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190218102748.2242-1-xieyongji@baidu.com> References: <20190218102748.2242-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::443 Subject: [Qemu-devel] [PATCH v6 4/7] libvhost-user: Support tracking inflight I/O in shared memory X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xie Yongji This patch adds support for VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD message to set/get shared buffer to/from qemu. Then backend can track inflight I/O in this buffer. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- Makefile | 2 +- contrib/libvhost-user/libvhost-user.c | 300 ++++++++++++++++++++++++-- contrib/libvhost-user/libvhost-user.h | 58 +++++ 3 files changed, 339 insertions(+), 21 deletions(-) diff --git a/Makefile b/Makefile index 3658310b95..8469bd94fb 100644 --- a/Makefile +++ b/Makefile @@ -477,7 +477,7 @@ Makefile: $(version-obj-y) # Build libraries =20 libqemuutil.a: $(util-obj-y) $(trace-obj-y) $(stub-obj-y) -libvhost-user.a: $(libvhost-user-obj-y) +libvhost-user.a: $(libvhost-user-obj-y) $(util-obj-y) $(stub-obj-y) =20 ###################################################################### =20 diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index ea0f414b6d..c20850e890 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -41,6 +41,8 @@ #endif =20 #include "qemu/atomic.h" +#include "qemu/osdep.h" +#include "qemu/memfd.h" =20 #include "libvhost-user.h" =20 @@ -53,6 +55,18 @@ _min1 < _min2 ? _min1 : _min2; }) #endif =20 +/* Round number down to multiple */ +#define ALIGN_DOWN(n, m) ((n) / (m) * (m)) + +/* Round number up to multiple */ +#define ALIGN_UP(n, m) ALIGN_DOWN((n) + (m) - 1, (m)) + +/* Align each region to cache line size in inflight buffer */ +#define INFLIGHT_ALIGNMENT 64 + +/* The version of inflight buffer */ +#define INFLIGHT_VERSION 1 + #define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) =20 /* The version of the protocol we support */ @@ -66,6 +80,20 @@ } \ } while (0) =20 +static inline +bool has_feature(uint64_t features, unsigned int fbit) +{ + assert(fbit < 64); + return !!(features & (1ULL << fbit)); +} + +static inline +bool vu_has_feature(VuDev *dev, + unsigned int fbit) +{ + return has_feature(dev->features, fbit); +} + static const char * vu_request_to_string(unsigned int req) { @@ -100,6 +128,8 @@ vu_request_to_string(unsigned int req) REQ(VHOST_USER_POSTCOPY_ADVISE), REQ(VHOST_USER_POSTCOPY_LISTEN), REQ(VHOST_USER_POSTCOPY_END), + REQ(VHOST_USER_GET_INFLIGHT_FD), + REQ(VHOST_USER_SET_INFLIGHT_FD), REQ(VHOST_USER_MAX), }; #undef REQ @@ -890,6 +920,55 @@ vu_check_queue_msg_file(VuDev *dev, VhostUserMsg *vmsg) return true; } =20 +static int +vu_check_queue_inflights(VuDev *dev, VuVirtq *vq) +{ + int i =3D 0; + + if (!has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (unlikely(!vq->inflight)) { + return -1; + } + + if (unlikely(!vq->inflight->version)) { + /* initialize the buffer */ + vq->inflight->version =3D INFLIGHT_VERSION; + return 0; + } + + vq->used_idx =3D vq->vring.used->idx; + vq->inflight_num =3D 0; + + if (unlikely(vq->inflight->used_idx !=3D vq->used_idx)) { + vq->inflight->desc[vq->inflight->process_head].inflight =3D 0; + + barrier(); + + vq->inflight->used_idx =3D vq->used_idx; + } + + for (i =3D 0; i < vq->inflight->desc_num; i++) { + if (vq->inflight->desc[i].inflight =3D=3D 0) { + continue; + } + + vq->inflight_desc[vq->inflight_num++] =3D i; + vq->inuse++; + } + vq->shadow_avail_idx =3D vq->last_avail_idx =3D vq->inuse + vq->used_i= dx; + + /* in case of I/O hang after reconnecting */ + if (eventfd_write(vq->kick_fd, 1)) { + return -1; + } + + return 0; +} + static bool vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) { @@ -923,6 +1002,10 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg) dev->vq[index].kick_fd, index); } =20 + if (vu_check_queue_inflights(dev, &dev->vq[index])) { + vu_panic(dev, "Failed to check inflights for vq: %d\n", index); + } + return false; } =20 @@ -995,6 +1078,11 @@ vu_set_vring_call_exec(VuDev *dev, VhostUserMsg *vmsg) =20 dev->vq[index].call_fd =3D vmsg->fds[0]; =20 + /* in case of I/O hang after reconnecting */ + if (eventfd_write(vmsg->fds[0], 1)) { + return -1; + } + DPRINT("Got call_fd: %d for vq: %d\n", vmsg->fds[0], index); =20 return false; @@ -1209,6 +1297,116 @@ vu_set_postcopy_end(VuDev *dev, VhostUserMsg *vmsg) return true; } =20 +static inline uint64_t +vu_inflight_queue_size(uint16_t queue_size) +{ + return ALIGN_UP(sizeof(VuDescStateSplit) * queue_size + + sizeof(uint16_t), INFLIGHT_ALIGNMENT); +} + +static bool +vu_get_inflight_fd(VuDev *dev, VhostUserMsg *vmsg) +{ + int fd; + void *addr; + uint64_t mmap_size; + uint16_t num_queues, queue_size; + + if (vmsg->size !=3D sizeof(vmsg->payload.inflight)) { + vu_panic(dev, "Invalid get_inflight_fd message:%d", vmsg->size); + vmsg->payload.inflight.mmap_size =3D 0; + return true; + } + + num_queues =3D vmsg->payload.inflight.num_queues; + queue_size =3D vmsg->payload.inflight.queue_size; + + DPRINT("set_inflight_fd num_queues: %"PRId16"\n", num_queues); + DPRINT("set_inflight_fd queue_size: %"PRId16"\n", queue_size); + + mmap_size =3D vu_inflight_queue_size(queue_size) * num_queues; + + addr =3D qemu_memfd_alloc("vhost-inflight", mmap_size, + F_SEAL_GROW | F_SEAL_SHRINK | F_SEAL_SEAL, + &fd, NULL); + + if (!addr) { + vu_panic(dev, "Failed to alloc vhost inflight area"); + vmsg->payload.inflight.mmap_size =3D 0; + return true; + } + + memset(addr, 0, mmap_size); + + dev->inflight_info.addr =3D addr; + dev->inflight_info.size =3D vmsg->payload.inflight.mmap_size =3D mmap_= size; + dev->inflight_info.fd =3D vmsg->fds[0] =3D fd; + vmsg->fd_num =3D 1; + vmsg->payload.inflight.mmap_offset =3D 0; + + DPRINT("send inflight mmap_size: %"PRId64"\n", + vmsg->payload.inflight.mmap_size); + DPRINT("send inflight mmap offset: %"PRId64"\n", + vmsg->payload.inflight.mmap_offset); + + return true; +} + +static bool +vu_set_inflight_fd(VuDev *dev, VhostUserMsg *vmsg) +{ + int fd, i; + uint64_t mmap_size, mmap_offset; + uint16_t num_queues, queue_size; + void *rc; + + if (vmsg->fd_num !=3D 1 || + vmsg->size !=3D sizeof(vmsg->payload.inflight)) { + vu_panic(dev, "Invalid set_inflight_fd message size:%d fds:%d", + vmsg->size, vmsg->fd_num); + return false; + } + + fd =3D vmsg->fds[0]; + mmap_size =3D vmsg->payload.inflight.mmap_size; + mmap_offset =3D vmsg->payload.inflight.mmap_offset; + num_queues =3D vmsg->payload.inflight.num_queues; + queue_size =3D vmsg->payload.inflight.queue_size; + + DPRINT("set_inflight_fd mmap_size: %"PRId64"\n", mmap_size); + DPRINT("set_inflight_fd mmap_offset: %"PRId64"\n", mmap_offset); + DPRINT("set_inflight_fd num_queues: %"PRId16"\n", num_queues); + DPRINT("set_inflight_fd queue_size: %"PRId16"\n", queue_size); + + rc =3D mmap(0, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED, + fd, mmap_offset); + + if (rc =3D=3D MAP_FAILED) { + vu_panic(dev, "set_inflight_fd mmap error: %s", strerror(errno)); + return false; + } + + if (dev->inflight_info.fd) { + close(dev->inflight_info.fd); + } + + if (dev->inflight_info.addr) { + munmap(dev->inflight_info.addr, dev->inflight_info.size); + } + + dev->inflight_info.fd =3D fd; + dev->inflight_info.addr =3D rc; + dev->inflight_info.size =3D mmap_size; + + for (i =3D 0; i < num_queues; i++) { + dev->vq[i].inflight =3D (VuVirtqInflight *)rc; + dev->vq[i].inflight->desc_num =3D queue_size; + rc =3D (void *)((char *)rc + vu_inflight_queue_size(queue_size)); + } + + return false; +} + static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { @@ -1286,6 +1484,10 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) return vu_set_postcopy_listen(dev, vmsg); case VHOST_USER_POSTCOPY_END: return vu_set_postcopy_end(dev, vmsg); + case VHOST_USER_GET_INFLIGHT_FD: + return vu_get_inflight_fd(dev, vmsg); + case VHOST_USER_SET_INFLIGHT_FD: + return vu_set_inflight_fd(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); @@ -1353,8 +1555,18 @@ vu_deinit(VuDev *dev) close(vq->err_fd); vq->err_fd =3D -1; } + vq->inflight =3D NULL; + } + + if (dev->inflight_info.addr) { + munmap(dev->inflight_info.addr, dev->inflight_info.size); + dev->inflight_info.addr =3D NULL; } =20 + if (dev->inflight_info.fd > 0) { + close(dev->inflight_info.fd); + dev->inflight_info.fd =3D -1; + } =20 vu_close_log(dev); if (dev->slave_fd !=3D -1) { @@ -1681,20 +1893,6 @@ vu_queue_empty(VuDev *dev, VuVirtq *vq) return vring_avail_idx(vq) =3D=3D vq->last_avail_idx; } =20 -static inline -bool has_feature(uint64_t features, unsigned int fbit) -{ - assert(fbit < 64); - return !!(features & (1ULL << fbit)); -} - -static inline -bool vu_has_feature(VuDev *dev, - unsigned int fbit) -{ - return has_feature(dev->features, fbit); -} - static bool vring_notify(VuDev *dev, VuVirtq *vq) { @@ -1823,12 +2021,6 @@ virtqueue_map_desc(VuDev *dev, *p_num_sg =3D num_sg; } =20 -/* Round number down to multiple */ -#define ALIGN_DOWN(n, m) ((n) / (m) * (m)) - -/* Round number up to multiple */ -#define ALIGN_UP(n, m) ALIGN_DOWN((n) + (m) - 1, (m)) - static void * virtqueue_alloc_element(size_t sz, unsigned out_num, unsigned in_num) @@ -1929,9 +2121,67 @@ vu_queue_map_desc(VuDev *dev, VuVirtq *vq, unsigned = int idx, size_t sz) return elem; } =20 +static int +vu_queue_inflight_get(VuDev *dev, VuVirtq *vq, int desc_idx) +{ + if (!has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (unlikely(!vq->inflight)) { + return -1; + } + + vq->inflight->desc[desc_idx].inflight =3D 1; + + return 0; +} + +static int +vu_queue_inflight_pre_put(VuDev *dev, VuVirtq *vq, int desc_idx) +{ + if (!has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (unlikely(!vq->inflight)) { + return -1; + } + + vq->inflight->process_head =3D desc_idx; + + return 0; +} + +static int +vu_queue_inflight_post_put(VuDev *dev, VuVirtq *vq, int desc_idx) +{ + if (!has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) { + return 0; + } + + if (unlikely(!vq->inflight)) { + return -1; + } + + barrier(); + + vq->inflight->desc[desc_idx].inflight =3D 0; + + barrier(); + + vq->inflight->used_idx =3D vq->used_idx; + + return 0; +} + void * vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) { + int i; unsigned int head; VuVirtqElement *elem; =20 @@ -1940,6 +2190,12 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) return NULL; } =20 + if (unlikely(vq->inflight_num > 0)) { + i =3D (--vq->inflight_num); + elem =3D vu_queue_map_desc(dev, vq, vq->inflight_desc[i], sz); + return elem; + } + if (vu_queue_empty(dev, vq)) { return NULL; } @@ -1970,6 +2226,8 @@ vu_queue_pop(VuDev *dev, VuVirtq *vq, size_t sz) =20 vq->inuse++; =20 + vu_queue_inflight_get(dev, vq, head); + return elem; } =20 @@ -2114,5 +2372,7 @@ vu_queue_push(VuDev *dev, VuVirtq *vq, const VuVirtqElement *elem, unsigned int len) { vu_queue_fill(dev, vq, elem, len, 0); + vu_queue_inflight_pre_put(dev, vq, elem->index); vu_queue_flush(dev, vq, 1); + vu_queue_inflight_post_put(dev, vq, elem->index); } diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 4aa55b4d2d..b1ca7fc5c1 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -53,6 +53,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_CONFIG =3D 9, VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD =3D 10, VHOST_USER_PROTOCOL_F_HOST_NOTIFIER =3D 11, + VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD =3D 12, =20 VHOST_USER_PROTOCOL_F_MAX }; @@ -91,6 +92,8 @@ typedef enum VhostUserRequest { VHOST_USER_POSTCOPY_ADVISE =3D 28, VHOST_USER_POSTCOPY_LISTEN =3D 29, VHOST_USER_POSTCOPY_END =3D 30, + VHOST_USER_GET_INFLIGHT_FD =3D 31, + VHOST_USER_SET_INFLIGHT_FD =3D 32, VHOST_USER_MAX } VhostUserRequest; =20 @@ -138,6 +141,13 @@ typedef struct VhostUserVringArea { uint64_t offset; } VhostUserVringArea; =20 +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint16_t num_queues; + uint16_t queue_size; +} VhostUserInflight; + #if defined(_WIN32) # define VU_PACKED __attribute__((gcc_struct, packed)) #else @@ -163,6 +173,7 @@ typedef struct VhostUserMsg { VhostUserLog log; VhostUserConfig config; VhostUserVringArea area; + VhostUserInflight inflight; } payload; =20 int fds[VHOST_MEMORY_MAX_NREGIONS]; @@ -234,9 +245,49 @@ typedef struct VuRing { uint32_t flags; } VuRing; =20 +typedef struct VuDescStateSplit { + /* Indicate whether this descriptor is inflight or not. + * Only available for head-descriptor. */ + uint8_t inflight; + + /* Padding */ + uint8_t padding; + + /* Link to the last processed entry */ + uint16_t next; +} VuDescStateSplit; + +typedef struct VuVirtqInflight { + /* The feature flags of this region. Now it's initialized to 0. */ + uint64_t features; + + /* The version of this region. It's 1 currently. + * Zero value indicates a vm reset happened. */ + uint16_t version; + + /* The size of VuDescStateSplit array. It's equal to the virtqueue + * size. Slave could get it from queue size field of VhostUserInflight= . */ + uint16_t desc_num; + + /* The head of processed VuDescStateSplit entry list */ + uint16_t process_head; + + /* Storing the idx value of used ring */ + uint16_t used_idx; + + /* Used to track the state of each descriptor in descriptor table */ + VuDescStateSplit desc[0]; +} VuVirtqInflight; + typedef struct VuVirtq { VuRing vring; =20 + VuVirtqInflight *inflight; + + uint16_t inflight_desc[VIRTQUEUE_MAX_SIZE]; + + uint16_t inflight_num; + /* Next head to pop */ uint16_t last_avail_idx; =20 @@ -279,11 +330,18 @@ typedef void (*vu_set_watch_cb) (VuDev *dev, int fd, = int condition, vu_watch_cb cb, void *data); typedef void (*vu_remove_watch_cb) (VuDev *dev, int fd); =20 +typedef struct VuDevInflightInfo { + int fd; + void *addr; + uint64_t size; +} VuDevInflightInfo; + struct VuDev { int sock; uint32_t nregions; VuDevRegion regions[VHOST_MEMORY_MAX_NREGIONS]; VuVirtq vq[VHOST_MAX_NR_VIRTQUEUE]; + VuDevInflightInfo inflight_info; int log_call_fd; int slave_fd; uint64_t log_size; --=20 2.17.1 From nobody Wed May 8 04:27:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1550486001535345.30495571832887; Mon, 18 Feb 2019 02:33:21 -0800 (PST) Received: from localhost ([127.0.0.1]:56045 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgEf-0003BM-Hz for importer@patchew.org; Mon, 18 Feb 2019 05:33:17 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43345) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgAM-0000AQ-5g for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gvgAI-0006K4-MT for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:48 -0500 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:33410) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gvgAH-000697-G5 for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:46 -0500 Received: by mail-pf1-x442.google.com with SMTP id c123so8357271pfb.0 for ; Mon, 18 Feb 2019 02:28:29 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id a187sm18820989pfb.61.2019.02.18.02.28.27 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Feb 2019 02:28:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=slWfpnGw8tZys3H9fBrwFQqWU/09SRe8VQgxBOnTr1M=; b=Dg9Y3tJMmp7Oq8P2bRKv2h+b/qZRcYof9N5KoC/IEQohuu9zejEdlftjme4SgJ1iHk p0ikNCI57tySttkV3B44wAEHCUkQlpZT592zOKnq4MsRiw10LyTmQPCSZktUTiU+0ZHh Wqc0uXgPhkk7u9XeZNSfc+SoeWrFgz5NUm2Z1WIxv/wFvKXgvy69+a5IssulgdfrW66U ncnr9aC4Ob2ienZNu2GM9kVrm42yvV2+Re9vCao5SZoZQfCKdW/f4cFTm6Wsq6t0GB5a ElDYeayLk04suZUiqcXYuTATNhk4Dp5ow8pIgo3SBQbbja6hS+SgEoY1PL1cTNG0hUmW 7BIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=slWfpnGw8tZys3H9fBrwFQqWU/09SRe8VQgxBOnTr1M=; b=b5iJwyp7SwMBtkzaBczEwzYllhuMkinjS6/y/G+GUxHHHsaGsDG/nfIyU+ZoBMlfZR NxZkXRmbcYdmzBz15zBMJ2BEagtyb7WTfiM9td4hh55LcVEtsNhRp2GLiGtDiZ214IqH IjbwFZ5uG7ktRFkpAww/gwyONLwcckw1WoRVi5/UQToFHpaGSHPsT7G5eTXTIWGvcSR5 rkNStPkrZ1Uoh5HOX4SwI2QO/7FFvCY8tnk1sjMEgSK4fKjkftkAlywSi3A2S9ySEbd6 KcDjJvaV4AawLxZCJP7lZeRECYE7UsvR8gGfH5ebhLCUryonxB9mvPfGk/IX/u1tVyFu JYBg== X-Gm-Message-State: AHQUAubcI4v58+JC9Hz5fbp41yN+PKXX1RD+m/uxr8wkYz6JpI0CS3Uh EfNMg24sWFWv+P0G6crUhVc= X-Google-Smtp-Source: AHgI3Ibyn55RQAMeu4JnFJLDfThrh+yWJ+Mdp8Wh3rUeuQ4ic/NdYxQhzTY6iSORKtQLWCwrsFQUJA== X-Received: by 2002:a63:4658:: with SMTP id v24mr21900915pgk.114.1550485708553; Mon, 18 Feb 2019 02:28:28 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, stefanha@gmail.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Mon, 18 Feb 2019 18:27:46 +0800 Message-Id: <20190218102748.2242-6-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190218102748.2242-1-xieyongji@baidu.com> References: <20190218102748.2242-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 Subject: [Qemu-devel] [PATCH v6 5/7] vhost-user-blk: Add support to get/set inflight buffer X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xie Yongji This patch adds support for vhost-user-blk device to get/set inflight buffer from/to backend. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- hw/block/vhost-user-blk.c | 28 ++++++++++++++++++++++++++++ include/hw/virtio/vhost-user-blk.h | 1 + 2 files changed, 29 insertions(+) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index 44ac814016..9682df1a7b 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -128,6 +128,21 @@ static void vhost_user_blk_start(VirtIODevice *vdev) } =20 s->dev.acked_features =3D vdev->guest_features; + + if (!s->inflight->addr) { + ret =3D vhost_dev_get_inflight(&s->dev, s->queue_size, s->inflight= ); + if (ret < 0) { + error_report("Error get inflight: %d", -ret); + goto err_guest_notifiers; + } + } + + ret =3D vhost_dev_set_inflight(&s->dev, s->inflight); + if (ret < 0) { + error_report("Error set inflight: %d", -ret); + goto err_guest_notifiers; + } + ret =3D vhost_dev_start(&s->dev, vdev); if (ret < 0) { error_report("Error starting vhost: %d", -ret); @@ -249,6 +264,13 @@ static void vhost_user_blk_handle_output(VirtIODevice = *vdev, VirtQueue *vq) } } =20 +static void vhost_user_blk_reset(VirtIODevice *vdev) +{ + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + + vhost_dev_free_inflight(s->inflight); +} + static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); @@ -289,6 +311,8 @@ static void vhost_user_blk_device_realize(DeviceState *= dev, Error **errp) vhost_user_blk_handle_output); } =20 + s->inflight =3D g_new0(struct vhost_inflight, 1); + s->dev.nvqs =3D s->num_queues; s->dev.vqs =3D g_new(struct vhost_virtqueue, s->dev.nvqs); s->dev.vq_index =3D 0; @@ -321,6 +345,7 @@ vhost_err: vhost_dev_cleanup(&s->dev); virtio_err: g_free(vqs); + g_free(s->inflight); virtio_cleanup(vdev); =20 vhost_user_cleanup(user); @@ -336,7 +361,9 @@ static void vhost_user_blk_device_unrealize(DeviceState= *dev, Error **errp) =20 vhost_user_blk_set_status(vdev, 0); vhost_dev_cleanup(&s->dev); + vhost_dev_free_inflight(s->inflight); g_free(vqs); + g_free(s->inflight); virtio_cleanup(vdev); =20 if (s->vhost_user) { @@ -386,6 +413,7 @@ static void vhost_user_blk_class_init(ObjectClass *klas= s, void *data) vdc->set_config =3D vhost_user_blk_set_config; vdc->get_features =3D vhost_user_blk_get_features; vdc->set_status =3D vhost_user_blk_set_status; + vdc->reset =3D vhost_user_blk_reset; } =20 static const TypeInfo vhost_user_blk_info =3D { diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-u= ser-blk.h index d52944aeeb..445516604a 100644 --- a/include/hw/virtio/vhost-user-blk.h +++ b/include/hw/virtio/vhost-user-blk.h @@ -36,6 +36,7 @@ typedef struct VHostUserBlk { uint32_t queue_size; uint32_t config_wce; struct vhost_dev dev; + struct vhost_inflight *inflight; VhostUserState *vhost_user; } VHostUserBlk; =20 --=20 2.17.1 From nobody Wed May 8 04:27:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1550485886179234.0977594058595; Mon, 18 Feb 2019 02:31:26 -0800 (PST) Received: from localhost ([127.0.0.1]:56023 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgCp-0001g7-4b for importer@patchew.org; Mon, 18 Feb 2019 05:31:23 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43435) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgAZ-0000Lj-Pb for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gvgAU-0006Qy-8S for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:00 -0500 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:34737) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gvgAQ-0006Ag-HT for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:28:56 -0500 Received: by mail-pf1-x444.google.com with SMTP id u9so1088382pfn.1 for ; Mon, 18 Feb 2019 02:28:32 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id y21sm21960204pfi.150.2019.02.18.02.28.30 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Feb 2019 02:28:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RNh/oOtv4ERgT0Ve3jIFWVWQW38wk/q784bJgu8BGrI=; b=uNbyEtOSAZbr38hc9pPhkxZGBXnUYNEi4tT5EFaf2xD3J5TJMcdNCF8OV/HZ2URWzi nnAMxOwLiFkWzYgTpFivdTEC4BQu5pxGeN7XJh9o8aoedrkE328ZP3ZZKBRi1Y2lxJ/X U9p6hgXVxrN7AQ/l+aO66DBQa/+YP3q0gBQdNLJBydORfuCzz1UoDQ9JYSRfaztWASvW 0xH4UMGHeT+hCNQyuiDZoBcew6viYthFxtpCBh12KUMV35LQ0NCsKCLRJANdy8K0/qR6 yTscF8a5UMK2G+XJJWA4UyTMEom5jmwz8XfdyaJJj+UEVLBJq9w3a4YOopr9KgSnSovy x/0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RNh/oOtv4ERgT0Ve3jIFWVWQW38wk/q784bJgu8BGrI=; b=XuGpJem/B6XVIoRGoPxT8PHsG93H6Jq2vr4f7CxrP2NDn5hsBYasR77M24QRWbVfYB mzt0jEKahpcVXVkifHWxTRRo67hW0KPkTzj0VhkeP1C+cWCD1AaySKoB+OPL3ol/CjJC DbLOYfmRXmw2nH0kuApUk3Ct7NRQnGXi44y+8DdNWOXn31FtSAbC53CY+clufq0f+ijr igHoSj6aU2EYjmsszFeM95XLuga6X/8ivEv0/GAv9IIUNWc7DN/JHuUjnmGZBhF5vXFj O/GS8HP3geTeIo9QRgH1r5N3z1wxLCUv0cUDHFEEbhATxhqwWkMNSHfchRjJUOxKHWlS 1NbQ== X-Gm-Message-State: AHQUAuZscihAft6E0RXktAok6o6KV2eLtXm5TGN8IV/cPbvnCUH1OVOW FRb+qyO6P5xAanh4QJBi/NM= X-Google-Smtp-Source: AHgI3IbtmHfSrT42WWuZBpAH9NTVC2xh0eodcUp/uBfQrV/GLbi3te0xQvuxoM9H2LABED6GBgssZw== X-Received: by 2002:a62:68c5:: with SMTP id d188mr24045994pfc.194.1550485711754; Mon, 18 Feb 2019 02:28:31 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, stefanha@gmail.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Mon, 18 Feb 2019 18:27:47 +0800 Message-Id: <20190218102748.2242-7-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190218102748.2242-1-xieyongji@baidu.com> References: <20190218102748.2242-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::444 Subject: [Qemu-devel] [PATCH v6 6/7] vhost-user-blk: Add support to reconnect backend X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xie Yongji Since we now support the message VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD. The backend is able to restart safely because it can track inflight I/O in shared memory. This patch allows qemu to reconnect the backend after connection closed. Signed-off-by: Xie Yongji Signed-off-by: Ni Xun Signed-off-by: Zhang Yu --- hw/block/vhost-user-blk.c | 205 +++++++++++++++++++++++------ include/hw/virtio/vhost-user-blk.h | 4 + 2 files changed, 167 insertions(+), 42 deletions(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index 9682df1a7b..539ea2e571 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -103,7 +103,7 @@ const VhostDevConfigOps blk_ops =3D { .vhost_dev_config_notifier =3D vhost_user_blk_handle_config_change, }; =20 -static void vhost_user_blk_start(VirtIODevice *vdev) +static int vhost_user_blk_start(VirtIODevice *vdev) { VHostUserBlk *s =3D VHOST_USER_BLK(vdev); BusState *qbus =3D BUS(qdev_get_parent_bus(DEVICE(vdev))); @@ -112,13 +112,13 @@ static void vhost_user_blk_start(VirtIODevice *vdev) =20 if (!k->set_guest_notifiers) { error_report("binding does not support guest notifiers"); - return; + return -ENOSYS; } =20 ret =3D vhost_dev_enable_notifiers(&s->dev, vdev); if (ret < 0) { error_report("Error enabling host notifiers: %d", -ret); - return; + return ret; } =20 ret =3D k->set_guest_notifiers(qbus->parent, s->dev.nvqs, true); @@ -157,12 +157,13 @@ static void vhost_user_blk_start(VirtIODevice *vdev) vhost_virtqueue_mask(&s->dev, vdev, i, false); } =20 - return; + return ret; =20 err_guest_notifiers: k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); err_host_notifiers: vhost_dev_disable_notifiers(&s->dev, vdev); + return ret; } =20 static void vhost_user_blk_stop(VirtIODevice *vdev) @@ -181,7 +182,6 @@ static void vhost_user_blk_stop(VirtIODevice *vdev) ret =3D k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); if (ret < 0) { error_report("vhost guest notifier cleanup failed: %d", ret); - return; } =20 vhost_dev_disable_notifiers(&s->dev, vdev); @@ -191,21 +191,43 @@ static void vhost_user_blk_set_status(VirtIODevice *v= dev, uint8_t status) { VHostUserBlk *s =3D VHOST_USER_BLK(vdev); bool should_start =3D status & VIRTIO_CONFIG_S_DRIVER_OK; + int ret; =20 if (!vdev->vm_running) { should_start =3D false; } =20 - if (s->dev.started =3D=3D should_start) { + if (s->should_start =3D=3D should_start) { + return; + } + + if (!s->connected || s->dev.started =3D=3D should_start) { + s->should_start =3D should_start; return; } =20 if (should_start) { - vhost_user_blk_start(vdev); + s->should_start =3D true; + /* + * make sure vhost_user_blk_handle_output() ignores fake + * guest kick by vhost_dev_enable_notifiers() + */ + barrier(); + ret =3D vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + qemu_chr_fe_disconnect(&s->chardev); + } } else { vhost_user_blk_stop(vdev); + /* + * make sure vhost_user_blk_handle_output() ignore fake + * guest kick by vhost_dev_disable_notifiers() + */ + barrier(); + s->should_start =3D false; } - } =20 static uint64_t vhost_user_blk_get_features(VirtIODevice *vdev, @@ -237,13 +259,22 @@ static uint64_t vhost_user_blk_get_features(VirtIODev= ice *vdev, static void vhost_user_blk_handle_output(VirtIODevice *vdev, VirtQueue *vq) { VHostUserBlk *s =3D VHOST_USER_BLK(vdev); - int i; + int i, ret; =20 if (!(virtio_host_has_feature(vdev, VIRTIO_F_VERSION_1) && !virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1))) { return; } =20 + if (s->should_start) { + return; + } + s->should_start =3D true; + + if (!s->connected) { + return; + } + if (s->dev.started) { return; } @@ -251,7 +282,13 @@ static void vhost_user_blk_handle_output(VirtIODevice = *vdev, VirtQueue *vq) /* Some guests kick before setting VIRTIO_CONFIG_S_DRIVER_OK so start * vhost here instead of waiting for .set_status(). */ - vhost_user_blk_start(vdev); + ret =3D vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + qemu_chr_fe_disconnect(&s->chardev); + return; + } =20 /* Kick right away to begin processing requests already in vring */ for (i =3D 0; i < s->dev.nvqs; i++) { @@ -271,13 +308,106 @@ static void vhost_user_blk_reset(VirtIODevice *vdev) vhost_dev_free_inflight(s->inflight); } =20 +static int vhost_user_blk_connect(DeviceState *dev) +{ + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + int ret =3D 0; + + if (s->connected) { + return 0; + } + s->connected =3D true; + + s->dev.nvqs =3D s->num_queues; + s->dev.vqs =3D s->vqs; + s->dev.vq_index =3D 0; + s->dev.backend_features =3D 0; + + vhost_dev_set_config_notifier(&s->dev, &blk_ops); + + ret =3D vhost_dev_init(&s->dev, s->vhost_user, VHOST_BACKEND_TYPE_USER= , 0); + if (ret < 0) { + error_report("vhost-user-blk: vhost initialization failed: %s", + strerror(-ret)); + return ret; + } + + /* restore vhost state */ + if (s->should_start) { + ret =3D vhost_user_blk_start(vdev); + if (ret < 0) { + error_report("vhost-user-blk: vhost start failed: %s", + strerror(-ret)); + return ret; + } + } + + return 0; +} + +static void vhost_user_blk_disconnect(DeviceState *dev) +{ + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + + if (!s->connected) { + return; + } + s->connected =3D false; + + if (s->dev.started) { + vhost_user_blk_stop(vdev); + } + + vhost_dev_cleanup(&s->dev); +} + +static gboolean vhost_user_blk_watch(GIOChannel *chan, GIOCondition cond, + void *opaque) +{ + DeviceState *dev =3D opaque; + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + + qemu_chr_fe_disconnect(&s->chardev); + + return true; +} + +static void vhost_user_blk_event(void *opaque, int event) +{ + DeviceState *dev =3D opaque; + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + + switch (event) { + case CHR_EVENT_OPENED: + if (vhost_user_blk_connect(dev) < 0) { + qemu_chr_fe_disconnect(&s->chardev); + return; + } + s->watch =3D qemu_chr_fe_add_watch(&s->chardev, G_IO_HUP, + vhost_user_blk_watch, dev); + break; + case CHR_EVENT_CLOSED: + vhost_user_blk_disconnect(dev); + if (s->watch) { + g_source_remove(s->watch); + s->watch =3D 0; + } + break; + } +} + + static void vhost_user_blk_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); VHostUserBlk *s =3D VHOST_USER_BLK(vdev); VhostUserState *user; - struct vhost_virtqueue *vqs =3D NULL; int i, ret; + Error *err =3D NULL; =20 if (!s->chardev.chr) { error_setg(errp, "vhost-user-blk: chardev is mandatory"); @@ -312,27 +442,28 @@ static void vhost_user_blk_device_realize(DeviceState= *dev, Error **errp) } =20 s->inflight =3D g_new0(struct vhost_inflight, 1); - - s->dev.nvqs =3D s->num_queues; - s->dev.vqs =3D g_new(struct vhost_virtqueue, s->dev.nvqs); - s->dev.vq_index =3D 0; - s->dev.backend_features =3D 0; - vqs =3D s->dev.vqs; - - vhost_dev_set_config_notifier(&s->dev, &blk_ops); - - ret =3D vhost_dev_init(&s->dev, s->vhost_user, VHOST_BACKEND_TYPE_USER= , 0); - if (ret < 0) { - error_setg(errp, "vhost-user-blk: vhost initialization failed: %s", - strerror(-ret)); - goto virtio_err; - } + s->vqs =3D g_new(struct vhost_virtqueue, s->num_queues); + s->watch =3D 0; + s->should_start =3D false; + s->connected =3D false; + + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, vhost_user_blk_even= t, + NULL, (void *)dev, NULL, true); + +reconnect: + do { + if (qemu_chr_fe_wait_connected(&s->chardev, &err) < 0) { + error_report_err(err); + err =3D NULL; + sleep(1); + } + } while (!s->connected); =20 ret =3D vhost_dev_get_config(&s->dev, (uint8_t *)&s->blkcfg, - sizeof(struct virtio_blk_config)); + sizeof(struct virtio_blk_config)); if (ret < 0) { - error_setg(errp, "vhost-user-blk: get block config failed"); - goto vhost_err; + error_report("vhost-user-blk: get block config failed"); + goto reconnect; } =20 if (s->blkcfg.num_queues !=3D s->num_queues) { @@ -340,29 +471,19 @@ static void vhost_user_blk_device_realize(DeviceState= *dev, Error **errp) } =20 return; - -vhost_err: - vhost_dev_cleanup(&s->dev); -virtio_err: - g_free(vqs); - g_free(s->inflight); - virtio_cleanup(vdev); - - vhost_user_cleanup(user); - g_free(user); - s->vhost_user =3D NULL; } =20 static void vhost_user_blk_device_unrealize(DeviceState *dev, Error **errp) { VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); VHostUserBlk *s =3D VHOST_USER_BLK(dev); - struct vhost_virtqueue *vqs =3D s->dev.vqs; =20 vhost_user_blk_set_status(vdev, 0); + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, NULL, + NULL, NULL, NULL, false); vhost_dev_cleanup(&s->dev); vhost_dev_free_inflight(s->inflight); - g_free(vqs); + g_free(s->vqs); g_free(s->inflight); virtio_cleanup(vdev); =20 diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-u= ser-blk.h index 445516604a..4849aa5eb5 100644 --- a/include/hw/virtio/vhost-user-blk.h +++ b/include/hw/virtio/vhost-user-blk.h @@ -38,6 +38,10 @@ typedef struct VHostUserBlk { struct vhost_dev dev; struct vhost_inflight *inflight; VhostUserState *vhost_user; + struct vhost_virtqueue *vqs; + guint watch; + bool should_start; + bool connected; } VHostUserBlk; =20 #endif --=20 2.17.1 From nobody Wed May 8 04:27:21 2024 Delivered-To: importer@patchew.org Received-SPF: temperror (zoho.com: Error in retrieving data from DNS) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=temperror (zoho.com: Error in retrieving data from DNS) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=gmail.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1550486169530732.4586090027772; Mon, 18 Feb 2019 02:36:09 -0800 (PST) Received: from localhost ([127.0.0.1]:56101 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgHI-0005If-Ij for importer@patchew.org; Mon, 18 Feb 2019 05:36:00 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43477) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gvgAd-0000ND-LV for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gvgAb-0006WG-UK for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:07 -0500 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]:44334) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gvgAb-0006CO-GK for qemu-devel@nongnu.org; Mon, 18 Feb 2019 05:29:05 -0500 Received: by mail-pl1-x642.google.com with SMTP id c4so3551772pls.11 for ; Mon, 18 Feb 2019 02:28:35 -0800 (PST) Received: from localhost ([116.247.112.152]) by smtp.gmail.com with ESMTPSA id h2sm4121864pfo.163.2019.02.18.02.28.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Feb 2019 02:28:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=eKXI+nTrVO5bzrZScLYB5tSJp0wzkrrEgJ96mqBPxXE=; b=G2VPoWxDBbb+BQZ80WslSpFcXmX5sWPisfqxDuI39m1a/Co7ReNuDDswshYWEHehhl 0MvNaApGWTSV7MlNAcu5umeBMQANrQQyDWEmDqA6+28+HXA8sUvWlgdlUOSX/r2K4DBY wTovG0s3L+pPlpSAMo8EMqJJnJoNflyptdZphfU8WWGPmfej2Bp3eXBpZtzbXgw7YeU2 99kpVbhqWX9cCflPDZkN3mT5U5XwIEpWVC3StjHuU4YA89RxtvX+zjDHRXr7GjsXKHd5 HQoOk48Y2p0j1utezidKjXnKx079S+Y1F85nNob+3CIuyjU6DU9YPgZhEjrFuiBu3+m+ 3kjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=eKXI+nTrVO5bzrZScLYB5tSJp0wzkrrEgJ96mqBPxXE=; b=YgpvOT2Tvohld8qJG4bfOgXQ/zoCqFu66b4X1rnkFprCTUjCORWpV6a03NudbQRblI /hcgriFQMXqyqaOqekxbg95mbnqV1xG9DBZBMGkxJlkMwY2BessRapexrj5m5wsWMrqc O93/a9Zkb2vJ9XZtN2s0Qy0RFKHRJLdBOPV03HKeqVPaWRnfxj0Y4ZLl5mX9bVTeeBP5 jSEC9+QlZO0+emLKCLReqSU1aA9dRhiTsKBaqMymTvw2927Zlwn4M5fdr936OrtAG7eM YqjiaTStbV/o1nHNahn9NiZcGpaHNrwpkidn2hzj007ubkWoD3EKboplHYK9NWRXrjo7 4Jrw== X-Gm-Message-State: AHQUAubUaUycHCjfartsmnl1f9PKoFAX2K/3KHq+57eeOfSOXtg3USr1 W4RR/IyEz5sgdUwmnnUKRvQ= X-Google-Smtp-Source: AHgI3IZCazEX3wFYIIiNpByPSvxmnudPx1EuZF6NYZAK0eVtUt2dxubSPv5AGADu88azkpD49/wDwQ== X-Received: by 2002:a17:902:e3:: with SMTP id a90mr24831825pla.82.1550485714771; Mon, 18 Feb 2019 02:28:34 -0800 (PST) From: elohimes@gmail.com X-Google-Original-From: xieyongji@baidu.com To: mst@redhat.com, stefanha@gmail.com, marcandre.lureau@redhat.com, berrange@redhat.com, jasowang@redhat.com, maxime.coquelin@redhat.com, yury-kotov@yandex-team.ru, wrfsh@yandex-team.ru Date: Mon, 18 Feb 2019 18:27:48 +0800 Message-Id: <20190218102748.2242-8-xieyongji@baidu.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190218102748.2242-1-xieyongji@baidu.com> References: <20190218102748.2242-1-xieyongji@baidu.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::642 Subject: [Qemu-devel] [PATCH v6 7/7] contrib/vhost-user-blk: enable inflight I/O tracking X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: nixun@baidu.com, qemu-devel@nongnu.org, lilin24@baidu.com, zhangyu31@baidu.com, chaiwen@baidu.com, Xie Yongji Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Xie Yongji This patch enables inflight I/O tracking for vhost-user-blk backend so that we could restart it safely. Signed-off-by: Xie Yongji Signed-off-by: Zhang Yu --- contrib/vhost-user-blk/vhost-user-blk.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/contrib/vhost-user-blk/vhost-user-blk.c b/contrib/vhost-user-b= lk/vhost-user-blk.c index 43583f2659..86a3987744 100644 --- a/contrib/vhost-user-blk/vhost-user-blk.c +++ b/contrib/vhost-user-blk/vhost-user-blk.c @@ -398,7 +398,8 @@ vub_get_features(VuDev *dev) static uint64_t vub_get_protocol_features(VuDev *dev) { - return 1ull << VHOST_USER_PROTOCOL_F_CONFIG; + return 1ull << VHOST_USER_PROTOCOL_F_CONFIG | + 1ull << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD; } =20 static int --=20 2.17.1