From nobody Wed Nov 5 10:07:48 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 153363392882392.25565113990933; Tue, 7 Aug 2018 02:25:28 -0700 (PDT) Received: from localhost ([::1]:38089 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmyF1-0005Gr-ML for importer@patchew.org; Tue, 07 Aug 2018 05:25:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49716) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmy7O-0007In-Pp for qemu-devel@nongnu.org; Tue, 07 Aug 2018 05:17:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmy7N-0007EY-ES for qemu-devel@nongnu.org; Tue, 07 Aug 2018 05:17:30 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51388 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fmy7I-0007BS-KL; Tue, 07 Aug 2018 05:17:24 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4108E7C6CA; Tue, 7 Aug 2018 09:17:24 +0000 (UTC) Received: from lemon.usersys.redhat.com (ovpn-12-63.pek2.redhat.com [10.72.12.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA94B1C640; Tue, 7 Aug 2018 09:17:19 +0000 (UTC) From: Fam Zheng To: qemu-devel@nongnu.org Date: Tue, 7 Aug 2018 17:16:55 +0800 Message-Id: <20180807091655.32472-3-famz@redhat.com> In-Reply-To: <20180807091655.32472-1-famz@redhat.com> References: <20180807091655.32472-1-famz@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Tue, 07 Aug 2018 09:17:24 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Tue, 07 Aug 2018 09:17:24 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'famz@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v2 2/2] aio: Do aio_notify_accept only during blocking aio_poll X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: pbonzini@redhat.com, Fam Zheng , qemu-block@nongnu.org, Stefan Hajnoczi , lersek@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" An aio_notify() pairs with an aio_notify_accept(). The former should happen in the main thread or a vCPU thread, and the latter should be done in the IOThread. There is one rare case that the main thread or vCPU thread may "steal" the aio_notify() event just raised by itself, in bdrv_set_aio_context() [1]. The sequence is like this: main thread IO Thread =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D bdrv_drained_begin() aio_disable_external(ctx) aio_poll(ctx, true) ctx->notify_me +=3D 2 ppoll() /* blocked */ ... bdrv_drained_end() ... aio_notify() ... bdrv_set_aio_context() aio_poll(ctx, false) [1] aio_notify_accept(ctx) /* Hang! */ [1] is problematic. It will clear the ctx->notifier event so that the blocked ppoll() will not return. (For the curious, this bug was noticed when booting a number of VMs simultaneously in RHV. One or two of the VMs will hit this race condition, making the VIRTIO device unresponsive to I/O commands. When it hangs, Seabios is busy waiting for a read request to complete (read MBR), right after initializing the virtio-blk-pci device, using 100% guest CPU. See also https://bugzilla.redhat.com/show_bug.cgi?id=3D1562750 for the original bug analysis.) aio_notify() only injects an event when ctx->notify_me is set, correspondingly aio_notify_accept() is only useful when ctx->notify_me _was_ set. Move the call to it into the "blocking" branch. This will effectively skip [1] and fix the hang. Furthermore, blocking aio_poll is only allowed on home thread (in_aio_context_home_thread), because otherwise two blocking aio_poll()'s can steal each other's ctx->notifier event and cause hanging just like described above. Suggested-by: Paolo Bonzini Signed-off-by: Fam Zheng --- util/aio-posix.c | 4 ++-- util/aio-win32.c | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/util/aio-posix.c b/util/aio-posix.c index b5c7f463aa..b5c609b68b 100644 --- a/util/aio-posix.c +++ b/util/aio-posix.c @@ -591,6 +591,7 @@ bool aio_poll(AioContext *ctx, bool blocking) * so disable the optimization now. */ if (blocking) { + assert(in_aio_context_home_thread(ctx)); atomic_add(&ctx->notify_me, 2); } =20 @@ -633,6 +634,7 @@ bool aio_poll(AioContext *ctx, bool blocking) =20 if (blocking) { atomic_sub(&ctx->notify_me, 2); + aio_notify_accept(ctx); } =20 /* Adjust polling time */ @@ -676,8 +678,6 @@ bool aio_poll(AioContext *ctx, bool blocking) } } =20 - aio_notify_accept(ctx); - /* if we have any readable fds, dispatch event */ if (ret > 0) { for (i =3D 0; i < npfd; i++) { diff --git a/util/aio-win32.c b/util/aio-win32.c index e676a8d9b2..c58957cc4b 100644 --- a/util/aio-win32.c +++ b/util/aio-win32.c @@ -373,11 +373,12 @@ bool aio_poll(AioContext *ctx, bool blocking) ret =3D WaitForMultipleObjects(count, events, FALSE, timeout); if (blocking) { assert(first); + assert(in_aio_context_home_thread(ctx)); atomic_sub(&ctx->notify_me, 2); + aio_notify_accept(ctx); } =20 if (first) { - aio_notify_accept(ctx); progress |=3D aio_bh_poll(ctx); first =3D false; } --=20 2.17.1