From nobody Sun May 4 01:19:46 2025
Delivered-To: importer@patchew.org
Authentication-Results: mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass(p=none dis=none) header.from=redhat.com
ARC-Seal: i=1; a=rsa-sha256; t=1689190684; cv=none;
d=zohomail.com; s=zohoarc;
b=nDaoaVi/gEdv8mtE0b/Or0hkhZF4voHPtS092YNwbYLfxnbM5zsEWHzC30cJ69ghUOQVjEKuHa+k2vfnb8AECg/QRlMeFwXzunG0Pls5YPnqq+znKr5N457VODkuNnig6C+9PxhqzBB5oy5LdFnKDyQl93lSob9IupM3blNsaJo=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
s=zohoarc;
t=1689190684;
h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To;
bh=Y4ySrccUg5zcQvoQQ7hsJ4pytFTcCgR4GbSUc4Abkuw=;
b=jt6p8WwcoOuMnqvOKN9D6IBo5ocH85P4RFaptpFeLweL3Ge3lFMpcWLEg5TbdotQFZZU/Xko3l4jE08/H4jU6IWYzGkb1o6sh7cVsPIRiZMl0cxZyBKb2fgp+gk+Qw36FloKlRPzQX0TMjOyK+2cFBU5RQKbb2yqynzbxIqOqdA=
ARC-Authentication-Results: i=1; mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass header.from=<stefanha@redhat.com> (p=none dis=none)
Return-Path: <qemu-devel-bounces+importer=patchew.org@nongnu.org>
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by
mx.zohomail.com
with SMTPS id 1689190684281567.6335971787095;
Wed, 12 Jul 2023 12:38:04 -0700 (PDT)
Received: from localhost ([::1] helo=lists1p.gnu.org)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <qemu-devel-bounces@nongnu.org>)
id 1qJfeF-0004Ps-8K; Wed, 12 Jul 2023 15:37:15 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <stefanha@redhat.com>)
id 1qJfe9-0004M8-Jn
for qemu-devel@nongnu.org; Wed, 12 Jul 2023 15:37:10 -0400
Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <stefanha@redhat.com>)
id 1qJfe7-0005Vw-Jg
for qemu-devel@nongnu.org; Wed, 12 Jul 2023 15:37:09 -0400
Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by
relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
us-mta-25-qS6CUCToOUWHHdon4Ik0Lg-1; Wed, 12 Jul 2023 15:37:03 -0400
Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com
[10.11.54.3])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C4D541C06914;
Wed, 12 Jul 2023 19:37:02 +0000 (UTC)
Received: from localhost (unknown [10.39.192.18])
by smtp.corp.redhat.com (Postfix) with ESMTP id 28DC3111E3EC;
Wed, 12 Jul 2023 19:37:01 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
s=mimecast20190719; t=1689190626;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding:
in-reply-to:in-reply-to:references:references;
bh=Y4ySrccUg5zcQvoQQ7hsJ4pytFTcCgR4GbSUc4Abkuw=;
b=APjP49JnBZUE8n7RecfhlhC86Sm7/OIdVQ3WU9FK4Hg2UfQinqU2LFIsppZlahbxVCfM/k
QHAlz2100akfq8y6/njpM5LrLgNI6IJyoZOHWwjGXHBqXzm7K/O7VcRD1t5K9smZfuvRXF
JM+fLkDE59QgInlh9W41EUoIZ8eXAGA=
X-MC-Unique: qS6CUCToOUWHHdon4Ik0Lg-1
From: Stefan Hajnoczi <stefanha@redhat.com>
To: qemu-devel@nongnu.org
Cc: Kevin Wolf <kwolf@redhat.com>, qemu-block@nongnu.org,
Hanna Reitz <hreitz@redhat.com>, Stefan Hajnoczi <stefanha@redhat.com>,
Richard Henderson <rth@twiddle.net>,
=?UTF-8?q?Luk=C3=A1=C5=A1=20Doktor?= <ldoktor@redhat.com>
Subject: [PULL 1/1] virtio-blk: fix host notifier issues during dataplane
start/stop
Date: Wed, 12 Jul 2023 15:36:58 -0400
Message-Id: <20230712193658.255676-2-stefanha@redhat.com>
In-Reply-To: <20230712193658.255676-1-stefanha@redhat.com>
References: <20230712193658.255676-1-stefanha@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3
Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17
as permitted sender) client-ip=209.51.188.17;
envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org;
helo=lists.gnu.org;
Received-SPF: pass client-ip=170.10.129.124;
envelope-from=stefanha@redhat.com;
helo=us-smtp-delivery-124.mimecast.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001,
DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,
SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org
Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org
X-ZohoMail-DKIM: pass (identity @redhat.com)
X-ZM-MESSAGEID: 1689190686025100003
The main loop thread can consume 100% CPU when using --device
virtio-blk-pci,iothread=3D<iothread>. ppoll() constantly returns but
reading virtqueue host notifiers fails with EAGAIN. The file descriptors
are stale and remain registered with the AioContext because of bugs in
the virtio-blk dataplane start/stop code.
The problem is that the dataplane start/stop code involves drain
operations, which call virtio_blk_drained_begin() and
virtio_blk_drained_end() at points where the host notifier is not
operational:
- In virtio_blk_data_plane_start(), blk_set_aio_context() drains after
vblk->dataplane_started has been set to true but the host notifier has
not been attached yet.
- In virtio_blk_data_plane_stop(), blk_drain() and blk_set_aio_context()
drain after the host notifier has already been detached but with
vblk->dataplane_started still set to true.
I would like to simplify ->ioeventfd_start/stop() to avoid interactions
with drain entirely, but couldn't find a way to do that. Instead, this
patch accepts the fragile nature of the code and reorders it so that
vblk->dataplane_started is false during drain operations. This way the
virtio_blk_drained_begin() and virtio_blk_drained_end() calls don't
touch the host notifier. The result is that
virtio_blk_data_plane_start() and virtio_blk_data_plane_stop() have
complete control over the host notifier and stale file descriptors are
no longer left in the AioContext.
This patch fixes the 100% CPU consumption in the main loop thread and
correctly moves host notifier processing to the IOThread.
Fixes: 1665d9326fd2 ("virtio-blk: implement BlockDevOps->drained_begin()")
Reported-by: Luk=C3=A1=C5=A1 Doktor <ldoktor@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Tested-by: Lukas Doktor <ldoktor@redhat.com>
Message-id: 20230704151527.193586-1-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
hw/block/dataplane/virtio-blk.c | 67 +++++++++++++++++++--------------
1 file changed, 38 insertions(+), 29 deletions(-)
diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-bl=
k.c
index c227b39408..da36fcfd0b 100644
--- a/hw/block/dataplane/virtio-blk.c
+++ b/hw/block/dataplane/virtio-blk.c
@@ -219,13 +219,6 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev)
=20
memory_region_transaction_commit();
=20
- /*
- * These fields are visible to the IOThread so we rely on implicit bar=
riers
- * in aio_context_acquire() on the write side and aio_notify_accept() =
on
- * the read side.
- */
- s->starting =3D false;
- vblk->dataplane_started =3D true;
trace_virtio_blk_data_plane_start(s);
=20
old_context =3D blk_get_aio_context(s->conf->conf.blk);
@@ -244,6 +237,18 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev)
event_notifier_set(virtio_queue_get_host_notifier(vq));
}
=20
+ /*
+ * These fields must be visible to the IOThread when it processes the
+ * virtqueue, otherwise it will think dataplane has not started yet.
+ *
+ * Make sure ->dataplane_started is false when blk_set_aio_context() is
+ * called above so that draining does not cause the host notifier to be
+ * detached/attached prematurely.
+ */
+ s->starting =3D false;
+ vblk->dataplane_started =3D true;
+ smp_wmb(); /* paired with aio_notify_accept() on the read side */
+
/* Get this show started by hooking up our callbacks */
if (!blk_in_drain(s->conf->conf.blk)) {
aio_context_acquire(s->ctx);
@@ -273,7 +278,6 @@ int virtio_blk_data_plane_start(VirtIODevice *vdev)
fail_guest_notifiers:
vblk->dataplane_disabled =3D true;
s->starting =3D false;
- vblk->dataplane_started =3D true;
return -ENOSYS;
}
=20
@@ -327,6 +331,32 @@ void virtio_blk_data_plane_stop(VirtIODevice *vdev)
aio_wait_bh_oneshot(s->ctx, virtio_blk_data_plane_stop_bh, s);
}
=20
+ /*
+ * Batch all the host notifiers in a single transaction to avoid
+ * quadratic time complexity in address_space_update_ioeventfds().
+ */
+ memory_region_transaction_begin();
+
+ for (i =3D 0; i < nvqs; i++) {
+ virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false);
+ }
+
+ /*
+ * The transaction expects the ioeventfds to be open when it
+ * commits. Do it now, before the cleanup loop.
+ */
+ memory_region_transaction_commit();
+
+ for (i =3D 0; i < nvqs; i++) {
+ virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i);
+ }
+
+ /*
+ * Set ->dataplane_started to false before draining so that host notif=
iers
+ * are not detached/attached anymore.
+ */
+ vblk->dataplane_started =3D false;
+
aio_context_acquire(s->ctx);
=20
/* Wait for virtio_blk_dma_restart_bh() and in flight I/O to complete =
*/
@@ -340,32 +370,11 @@ void virtio_blk_data_plane_stop(VirtIODevice *vdev)
=20
aio_context_release(s->ctx);
=20
- /*
- * Batch all the host notifiers in a single transaction to avoid
- * quadratic time complexity in address_space_update_ioeventfds().
- */
- memory_region_transaction_begin();
-
- for (i =3D 0; i < nvqs; i++) {
- virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), i, false);
- }
-
- /*
- * The transaction expects the ioeventfds to be open when it
- * commits. Do it now, before the cleanup loop.
- */
- memory_region_transaction_commit();
-
- for (i =3D 0; i < nvqs; i++) {
- virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), i);
- }
-
qemu_bh_cancel(s->bh);
notify_guest_bh(s); /* final chance to notify guest */
=20
/* Clean up guest notifier (irq) */
k->set_guest_notifiers(qbus->parent, nvqs, false);
=20
- vblk->dataplane_started =3D false;
s->stopping =3D false;
}
--=20
2.40.1