From nobody Thu May 7 07:02:48 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1656424336919229.028525187706; Tue, 28 Jun 2022 06:52:16 -0700 (PDT) Received: from localhost ([::1]:51064 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o6BdX-00036r-SL for importer@patchew.org; Tue, 28 Jun 2022 09:52:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54576) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o6BbJ-0001DV-Gd for qemu-devel@nongnu.org; Tue, 28 Jun 2022 09:49:57 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:50553) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o6BbF-00078D-Ay for qemu-devel@nongnu.org; Tue, 28 Jun 2022 09:49:55 -0400 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-502-dqn1VH3TMB2Ft8pFet_ptA-1; Tue, 28 Jun 2022 09:49:49 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C1413384F800; Tue, 28 Jun 2022 13:49:48 +0000 (UTC) Received: from localhost (unknown [10.39.208.43]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7A754141510C; Tue, 28 Jun 2022 13:49:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1656424192; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vKr15rhBWWTXIMixtzUVs/TQGOHpDegIlCXWx5otV7M=; b=Z8XdIizUtSJZT76F4NoyXHhx8xl1h8EY+VCmiUFVy/DPXCQKuvrpvoMrmMlE8qoF8eAvrF HjfF25XMRZiHRMwCzvP5iIG5Anj/4KgX4ojshQWVNQrj5MUmdcRr16yvoAgPXmAt6edQo6 1hcDYrfRT/WQL5rNjmQr+1vQDWeC8ts= X-MC-Unique: dqn1VH3TMB2Ft8pFet_ptA-1 From: marcandre.lureau@redhat.com To: qemu-devel@nongnu.org Cc: John Snow , Richard Henderson , Beraldo Leal , Cleber Rosa , =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= Subject: [PATCH 1/2] python/qemu/machine: replace subprocess.Popen with asyncio Date: Tue, 28 Jun 2022 17:49:38 +0400 Message-Id: <20220628134939.680174-2-marcandre.lureau@redhat.com> In-Reply-To: <20220628134939.680174-1-marcandre.lureau@redhat.com> References: <20220628134939.680174-1-marcandre.lureau@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=marcandre.lureau@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1656424338856100002 From: Marc-Andr=C3=A9 Lureau The following patch is going to wait for both subprocess and accept tasks concurrently. Switch to using asyncio for subprocess handling. Signed-off-by: Marc-Andr=C3=A9 Lureau --- python/qemu/machine/machine.py | 47 ++++++++++++++++++++++------------ 1 file changed, 31 insertions(+), 16 deletions(-) diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py index 37191f433b2d..55c45f4b1205 100644 --- a/python/qemu/machine/machine.py +++ b/python/qemu/machine/machine.py @@ -17,6 +17,7 @@ # Based on qmp.py. # =20 +import asyncio import errno from itertools import chain import locale @@ -30,6 +31,7 @@ from types import TracebackType from typing import ( Any, + Awaitable, BinaryIO, Dict, List, @@ -180,7 +182,7 @@ def __init__(self, # Runstate self._qemu_log_path: Optional[str] =3D None self._qemu_log_file: Optional[BinaryIO] =3D None - self._popen: Optional['subprocess.Popen[bytes]'] =3D None + self._subproc: Optional['asyncio.subprocess.Process'] =3D None self._events: List[QMPMessage] =3D [] self._iolog: Optional[str] =3D None self._qmp_set =3D True # Enable QMP monitor by default. @@ -198,6 +200,7 @@ def __init__(self, self._remove_files: List[str] =3D [] self._user_killed =3D False self._quit_issued =3D False + self._aloop =3D asyncio.get_event_loop() =20 def __enter__(self: _T) -> _T: return self @@ -269,19 +272,19 @@ def _remove_if_exists(path: str) -> None: =20 def is_running(self) -> bool: """Returns true if the VM is running.""" - return self._popen is not None and self._popen.poll() is None + return self._subproc is not None and self._subproc.returncode is N= one =20 @property - def _subp(self) -> 'subprocess.Popen[bytes]': - if self._popen is None: + def _subp(self) -> 'asyncio.subprocess.Process': + if self._subproc is None: raise QEMUMachineError('Subprocess pipe not present') - return self._popen + return self._subproc =20 def exitcode(self) -> Optional[int]: """Returns the exit code if possible, or None.""" - if self._popen is None: + if self._subproc is None: return None - return self._popen.poll() + return self._subproc.returncode =20 def get_pid(self) -> Optional[int]: """Returns the PID of the running process, or None.""" @@ -443,6 +446,13 @@ def launch(self) -> None: # that exception. However, we still want to clean up. raise =20 + def _sync( + self, future: Awaitable[_T], timeout: Optional[float] =3D None + ) -> _T: + return self._aloop.run_until_complete( + asyncio.wait_for(future, timeout=3Dtimeout) + ) + def _launch(self) -> None: """ Launch the VM and establish a QMP connection @@ -452,12 +462,13 @@ def _launch(self) -> None: =20 # Cleaning up of this subprocess is guaranteed by _do_shutdown. # pylint: disable=3Dconsider-using-with - self._popen =3D subprocess.Popen(self._qemu_full_args, - stdin=3Dsubprocess.DEVNULL, - stdout=3Dself._qemu_log_file, - stderr=3Dsubprocess.STDOUT, - shell=3DFalse, - close_fds=3DFalse) + self._subproc =3D self._sync( + asyncio.create_subprocess_exec(*self._qemu_full_args, + stdin=3Dasyncio.subprocess.DEVN= ULL, + stdout=3Dself._qemu_log_file, + stderr=3Dasyncio.subprocess.STD= OUT, + close_fds=3DFalse) + ) self._launched =3D True self._post_launch() =20 @@ -508,8 +519,10 @@ def _hard_shutdown(self) -> None: waiting for the QEMU process to terminate. """ self._early_cleanup() - self._subp.kill() - self._subp.wait(timeout=3D60) + self._sync( + self._subp.kill(), + asyncio.wait_for(self._subp.wait(), timeout=3D60) + ) =20 def _soft_shutdown(self, timeout: Optional[int]) -> None: """ @@ -536,7 +549,9 @@ def _soft_shutdown(self, timeout: Optional[int]) -> Non= e: self._close_qmp_connection() =20 # May raise subprocess.TimeoutExpired - self._subp.wait(timeout=3Dtimeout) + self._sync( + asyncio.wait_for(self._subp.wait(), timeout=3Dtimeout) + ) =20 def _do_shutdown(self, timeout: Optional[int]) -> None: """ --=20 2.37.0.rc0 From nobody Thu May 7 07:02:48 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1656424477353457.83641434128253; Tue, 28 Jun 2022 06:54:37 -0700 (PDT) Received: from localhost ([::1]:55768 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o6Bfo-0006wW-BB for importer@patchew.org; Tue, 28 Jun 2022 09:54:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54612) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o6BbK-0001DX-6u for qemu-devel@nongnu.org; Tue, 28 Jun 2022 09:49:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:40132) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o6BbH-00078Y-IK for qemu-devel@nongnu.org; Tue, 28 Jun 2022 09:49:56 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-261-1a5CsgCCNmW8UgZOdo0glg-1; Tue, 28 Jun 2022 09:49:53 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0C5058001EA; Tue, 28 Jun 2022 13:49:53 +0000 (UTC) Received: from localhost (unknown [10.39.208.43]) by smtp.corp.redhat.com (Postfix) with ESMTP id 01E95404E4C8; Tue, 28 Jun 2022 13:49:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1656424194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G6zoiS/09yWGU2Jxrv4ABxlm5cp7QXkcgeXPJwJsvF8=; b=F2ovjwHf+mnDvYwSU8ECZfdIi4ui4m36FDUiiR2FIsETl/kYXrB1X9Z/+t8bhXNUtLq6OF EccDoab5D/gyEQJXHxo65CPMiLMuF51SIabB57Fz1yC28r6dFLN/P6m+7NBvmpfm55OfM0 kGllA+AI35wKWNX/Ybv5CBefNNjU3M0= X-MC-Unique: 1a5CsgCCNmW8UgZOdo0glg-1 From: marcandre.lureau@redhat.com To: qemu-devel@nongnu.org Cc: John Snow , Richard Henderson , Beraldo Leal , Cleber Rosa , =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= Subject: [PATCH 2/2] python/qemu/machine: accept QMP connection asynchronously Date: Tue, 28 Jun 2022 17:49:39 +0400 Message-Id: <20220628134939.680174-3-marcandre.lureau@redhat.com> In-Reply-To: <20220628134939.680174-1-marcandre.lureau@redhat.com> References: <20220628134939.680174-1-marcandre.lureau@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=marcandre.lureau@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1656424477899100001 From: Marc-Andr=C3=A9 Lureau QMP accept is currently synchronous. If qemu dies before the connection is established, it will wait there. Instead turn the code to do concurrently accept() and wait(). Returns when the first task is completed to determine whether a connection was established. Signed-off-by: Marc-Andr=C3=A9 Lureau --- python/qemu/machine/machine.py | 11 ++++++++++- python/qemu/qmp/legacy.py | 10 ++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py index 55c45f4b1205..5e2df7dc5055 100644 --- a/python/qemu/machine/machine.py +++ b/python/qemu/machine/machine.py @@ -362,9 +362,18 @@ def _pre_launch(self) -> None: self._args )) =20 + async def _async_accept(self) -> bool: + accept =3D asyncio.create_task(self._qmp.async_accept()) + wait =3D asyncio.create_task(self._subproc.wait()) + done, pending =3D await asyncio.wait([accept, wait], + return_when=3Dasyncio.FIRST_COM= PLETED) + return accept in done + def _post_launch(self) -> None: if self._qmp_connection: - self._qmp.accept(self._qmp_timer) + accepted =3D self._sync(self._async_accept()) + if not accepted: + raise QEMUMachineError('VM returned before QMP accept') =20 def _close_qemu_log_file(self) -> None: if self._qemu_log_file is not None: diff --git a/python/qemu/qmp/legacy.py b/python/qemu/qmp/legacy.py index 03b5574618fa..88bdbfb6e350 100644 --- a/python/qemu/qmp/legacy.py +++ b/python/qemu/qmp/legacy.py @@ -167,6 +167,16 @@ def accept(self, timeout: Optional[float] =3D 15.0) ->= QMPMessage: assert ret is not None return ret =20 + async def async_accept(self) -> QMPMessage: + self._qmp.await_greeting =3D True + self._qmp.negotiate =3D True + + await self._qmp.accept() + + ret =3D self._get_greeting() + assert ret is not None + return ret + def cmd_obj(self, qmp_cmd: QMPMessage) -> QMPMessage: """ Send a QMP command to the QMP Monitor. --=20 2.37.0.rc0