From nobody Mon Feb 9 11:51:26 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1648811835318572.772289599556; Fri, 1 Apr 2022 04:17:15 -0700 (PDT) Received: from localhost ([::1]:50028 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naFHF-0005TM-Uc for importer@patchew.org; Fri, 01 Apr 2022 07:17:13 -0400 Received: from eggs.gnu.org ([209.51.188.92]:51172) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naFFz-0004Aq-9V for qemu-devel@nongnu.org; Fri, 01 Apr 2022 07:15:55 -0400 Received: from smtpbg137.qq.com ([106.55.202.113]:25453 helo=smtpbg.qq.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naFFw-0001PZ-Lx for qemu-devel@nongnu.org; Fri, 01 Apr 2022 07:15:54 -0400 Received: from localhost.localdomain.localdoma ( [116.228.45.98]) by bizesmtp.qq.com (ESMTP) with id ; Fri, 01 Apr 2022 19:06:31 +0800 (CST) X-QQ-mid: bizesmtp66t1648811193tlidghw2 X-QQ-SSF: 01000000002000803000C00A0000000 X-QQ-FEAT: SIq+DGERJtzzFDgZO+cJW86DuahPtpRbbS1bSpmPzG1D4HP24R9yiij22pfiu s9afFfScbps5niCoD6BmdUjcxa1udQPQ8b6GMKt+pQgJJVDxjYRCOfjJWVvi9tugbr0MgBR sOhk6z+anJze1JhmD3Zc9qvd050UzEVZDJvmuXBxtPuzeOWQ2KqMzRngXj2VSWGtT17DEiZ yEQqzbaOgUHaIaAsGeiB/XVNn+Xl2fjBYa4kjHKhVPSk55UZF6IwHXVwdFPdqloRsKLSjLN U8reP0+HecdwiuO/5ZLyiPPI2tOrAwCyygMIsu9x5RzOnEmNJ6tcqxLJKniayiTMcsHQgs5 OCIfWscnoPJlAOoVSNHsCJOJC3i7qm+BlXauZVu X-QQ-GoodBg: 0 From: Michael Qiu To: jasowang@redhat.com, mst@redhat.com, si-wei.liu@oracle.com Subject: [PATCH 3/3 v5] vdpa: reset the backend device in the end of vhost_net_stop() Date: Fri, 1 Apr 2022 07:06:13 -0400 Message-Id: <1648811173-15293-4-git-send-email-qiudayu@archeros.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648811173-15293-1-git-send-email-qiudayu@archeros.com> References: <1648776683-23739-1-git-send-email-qiudayu@archeros.com> <1648811173-15293-1-git-send-email-qiudayu@archeros.com> X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:archeros.com:qybgspam:qybgspam9 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: none client-ip=106.55.202.113; envelope-from=qiudayu@archeros.com; helo=smtpbg.qq.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.187, SPF_HELO_NONE=0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Qiu , eperezma@redhat.com, lingshan.zhu@intel.com, qemu-devel@nongnu.org, lulu@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1648811838889100001 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Currently, when VM poweroff, it will trigger vdpa device(such as mlx bluefield2 VF) reset many times(with 1 datapath queue pair and one control queue, triggered 3 times), this leads to below issue: vhost VQ 2 ring restore failed: -22: Invalid argument (22) This because in vhost_net_stop(), it will stop all vhost device bind to this virtio device, and in vhost_dev_stop(), qemu tries to stop the device , then stop the queue: vhost_virtqueue_stop(). In vhost_dev_stop(), it resets the device, which clear some flags in low level driver, and in next loop(stop other vhost backends), qemu try to stop the queue corresponding to the vhost backend, the driver finds that the VQ is invalied, this is the root cause. To solve the issue, vdpa should set vring unready, and remove reset ops in device stop: vhost_dev_start(hdev, false). and implement a new function vhost_dev_reset, only reset backend device after all vhost(per-queue) stopped. Signed-off-by: Michael Qiu Acked-by: Jason Wang --- v5 --> v4: move vhost_dev_reset() call after set_guest_notifiers remove implementation of vhost_dev_reset() remove backend check for VHOST_BACKEND_TYPE_VDPA =20 v4 --> v3: Nothing changed, becasue of issue with mimecast, when the From: tag is different from the sender, the some mail client will take the patch as an attachment, RESEND v3 does not work, So resend the patch as v4 v3 --> v2: Call vhost_dev_reset() at the end of vhost_net_stop(). Since the vDPA device need re-add the status bit=20 VIRTIO_CONFIG_S_ACKNOWLEDGE and VIRTIO_CONFIG_S_DRIVER, simply, add them inside vhost_vdpa_reset_device, and the only way calling vhost_vdpa_reset_device is in vhost_net_stop(), so it keeps the same behavior as before. v2 --> v1: Implement a new function vhost_dev_reset, reset the backend kernel device at last. --- hw/net/vhost_net.c | 22 +++++++++++++++++++--- hw/virtio/vhost-vdpa.c | 15 +++++++++------ 2 files changed, 28 insertions(+), 9 deletions(-) diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index 30379d2..30c76ca 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -325,7 +325,7 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *= ncs, int total_notifiers =3D data_queue_pairs * 2 + cvq; VirtIONet *n =3D VIRTIO_NET(dev); int nvhosts =3D data_queue_pairs + cvq; - struct vhost_net *net; + struct vhost_net *net =3D NULL; int r, e, i, index_end =3D data_queue_pairs * 2; NetClientState *peer; =20 @@ -391,13 +391,21 @@ int vhost_net_start(VirtIODevice *dev, NetClientState= *ncs, err_start: while (--i >=3D 0) { peer =3D qemu_get_peer(ncs , i); - vhost_net_stop_one(get_vhost_net(peer), dev); + + net =3D get_vhost_net(peer); + + vhost_net_stop_one(net, dev); } + e =3D k->set_guest_notifiers(qbus->parent, total_notifiers, false); if (e < 0) { fprintf(stderr, "vhost guest notifier cleanup failed: %d\n", e); fflush(stderr); } + + if (net) { + vhost_dev_reset(&net->dev); + } err: return r; } @@ -410,6 +418,7 @@ void vhost_net_stop(VirtIODevice *dev, NetClientState *= ncs, VirtioBusClass *k =3D VIRTIO_BUS_GET_CLASS(vbus); VirtIONet *n =3D VIRTIO_NET(dev); NetClientState *peer; + struct vhost_net *net =3D NULL; int total_notifiers =3D data_queue_pairs * 2 + cvq; int nvhosts =3D data_queue_pairs + cvq; int i, r; @@ -420,7 +429,10 @@ void vhost_net_stop(VirtIODevice *dev, NetClientState = *ncs, } else { peer =3D qemu_get_peer(ncs, n->max_queue_pairs); } - vhost_net_stop_one(get_vhost_net(peer), dev); + + net =3D get_vhost_net(peer); + + vhost_net_stop_one(net, dev); } =20 r =3D k->set_guest_notifiers(qbus->parent, total_notifiers, false); @@ -429,6 +441,10 @@ void vhost_net_stop(VirtIODevice *dev, NetClientState = *ncs, fflush(stderr); } assert(r >=3D 0); + + if (net) { + vhost_dev_reset(&net->dev); + } } =20 void vhost_net_cleanup(struct vhost_net *net) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index c5ed7a3..3ef0199 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -708,6 +708,11 @@ static int vhost_vdpa_reset_device(struct vhost_dev *d= ev) =20 ret =3D vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &status); trace_vhost_vdpa_reset_device(dev, status); + + /* Add back this status, so that the device could work next time*/ + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | + VIRTIO_CONFIG_S_DRIVER); + return ret; } =20 @@ -719,14 +724,14 @@ static int vhost_vdpa_get_vq_index(struct vhost_dev *= dev, int idx) return idx; } =20 -static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev) +static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev, unsigned int = ready) { int i; trace_vhost_vdpa_set_vring_ready(dev); for (i =3D 0; i < dev->nvqs; ++i) { struct vhost_vring_state state =3D { .index =3D dev->vq_index + i, - .num =3D 1, + .num =3D ready, }; vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state); } @@ -1088,8 +1093,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev= , bool started) if (unlikely(!ok)) { return -1; } - vhost_vdpa_set_vring_ready(dev); + vhost_vdpa_set_vring_ready(dev, 1); } else { + vhost_vdpa_set_vring_ready(dev, 0); ok =3D vhost_vdpa_svqs_stop(dev); if (unlikely(!ok)) { return -1; @@ -1105,9 +1111,6 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev= , bool started) memory_listener_register(&v->listener, &address_space_memory); return vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK); } else { - vhost_vdpa_reset_device(dev); - vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | - VIRTIO_CONFIG_S_DRIVER); memory_listener_unregister(&v->listener); =20 return 0; --=20 1.8.3.1